The google-pubsub() source has the following options.
Required parameters
-
credentials()
-
project()
-
subscription()
Optional parameters
-
ack-tracker-batch-size()
-
ack-tracker-timeout()
-
log-fetch-limit()
-
prefix()
-
time-reopen()
-
workers()
The google-pubsub() source options, in more detail:
ack-tracker-batch-size()
| Type: |
string |
| Default: |
100 |
| Required: |
no |
Description: Optional parameter.
The syslog-ng PE application retains acknowledgements on the source side and either acknowledges an ack-tracker-batch-size() number of messages in a batch, or sends acknowledgements after the ack-tracker-timeout() expires. If the value of your ack-tracker-timeout() is larger than the value of your Acknowledgement deadline, it may result in message duplication.
ack-tracker-timeout()
| Type: |
time [milliseconds] |
| Default: |
3000 |
| Required: |
no |
Description: Optional parameter.
The syslog-ng PE application retains acknowledgements on the source side and either acknowledges an ack-tracker-batch-size() number of messages in a batch, or sends acknowledgements after the ack-tracker-timeout() expires. If the value of your ack-tracker-timeout() is larger than the value of your Acknowledgement deadline, it may result in message duplication.
credentials()
| Type: |
string |
| Default: |
n/a |
| Required: |
yes |
Description: Required parameter.
The credentials of your Google Pub/Sub project.
log-fetch-limit()
| Type: |
number |
| Default: |
100 |
| Required: |
no |
Description: Optional parameter.
The maximum number of messages fetched from a source during a single poll loop.
prefix()
| Type: |
string |
| Default: |
.pubsub. |
| Required: |
no |
Description: Optional parameter.
This prefix will be added to the name of the macros created from the message attributes of the Google Pub/Sub message.
project()
| Type: |
string |
| Default: |
n/a |
| Required: |
yes |
Description: Required parameter.
The ID of your Google Pub/Sub project.
subscription()
| Type: |
string |
| Default: |
n/a |
| Required: |
yes |
Description: Required parameter.
The ID of your Google Pub/Sub subscription.
time-reopen()
| Type: |
number (seconds) |
| Default: |
60 |
| Required: |
no |
Description: Optional parameter.
The time to wait in seconds before a broken connection is reestablished.
workers()
| Type: |
integer |
| Default: |
1 |
| Required: |
no |
Description: Optional parameter.
Specifies the number of worker threads (at least 1) that syslog-ng PE uses to receive messages from the Google Pub/Sub messaging service. Increasing the number of worker threads can drastically improve the performance of the destination.
The google-pubsub() source utilizes the At-Least-Once delivery behavior of Google Pub/Sub. This behavior is intentional, and its purpose is to avoid potential log loss. However, in certain cases, the At-Least-Once delivery behavior results in message duplication on the syslog-ng PE side.
Issue
On the Google Cloud Platform side, you can set the value of the Acknowledgment deadline (the default value is 10 seconds) when creating your Google Pub/Sub Subscription.
For more information, see Set up your Google Cloud project and Pub/Sub topic and subscriptions.
The syslog-ng PE application has to acknowledge log-fetch-limit() number of messages within the Acknowledgement deadline time limit. If syslog-ng PE does not acknowledge Google Pub/Sub messages no later than the time limit specified in the Acknowledgment deadline, the Google Pub/Sub service will attempt to redeliver the message to syslog-ng PE.
As a result, any acknowledgement sent later than the Acknowledgment deadline will result in message duplication on the syslog-ng PE side. This issue occurs most often if you have flow-control turned on, and your syslog-ng PE destinations are slow.
Workaround
To avoid message duplication, you can use one of these methods:
-
Using the disk-buffer option if flow control is on
When using the disk-buffer option, syslog-ng PE acknowledges Pub/Sub messages as soon as they are sent to the output queue or overflow queue, instead of acknowledging them when the destination sends or rewrites them.
-
Adjusting the value of your ack-tracker-timeout() to the Acknowledgment deadline, and the value of your ack-tracker-batch-size() to your log-fetch-limit()
The syslog-ng PE application acknowledges messages in batches. You can set the size (ack-tracker-batch-size(), the default value is 100), and timeout (ack-tracker-timeout(), the default value is 3000 milliseconds, or 3 seconds) of these batches.
To avoid message duplication, set your ack-tracker-timeout() to a value not larger than the value of your Acknowledgment deadline, and your ack-tracker-batch-size() to a value not larger than your log-fetch-limit().
Common error messages with workaround solutions
The following table describes the possible error messages that you may encounter while using the google-pubsub() source.
DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. |
Use the credentials() option to specify a key file.
For more information, see Create service account credentials. |
404 Resource not found (resource=...). |
The specified subscription is not valid. Check the value of the subscription() option. Make sure you specify the Subscription ID, not the name of the subscription.
For more information, see Set up your Google Cloud project and Pub/Sub topic and subscriptions. |
400 Invalid resource name given (name=...). |
The specified subscription is not valid. Check the value of the subscription() option. Make sure you specify the Subscription ID, not the name of the subscription.
For more information, see Set up your Google Cloud project and Pub/Sub topic and subscriptions. |
404 Requested project not found or user does not have access to it (project=nonproj). Make sure to specify the unique project identifier and not the Google Cloud Console display name. |
Make sure to specify the unique project identifier and not the Google Cloud Console display name. |
PubSubSource init, failed to load credentials file, path=... |
The file specified in the credentials() option does not exist, or it is not accessible. Check the path and its permissions. |
Error messages that require assistance from our Support Team
In some cases, you may encounter different error messages that require assistance from our Support Team. If you encounter similar error messages as those listed in the following table, One Identity recommends that you contact our Support Team for assistance.
PubSubSource error while pulling messages, error: ... |
A non-retryable error occurred.
If you encounter an error message other than those listed in the previous table, contact our Support Team with the complete error message. |
PubSubSource error while acking messages, error: ... |
A non-retryable error occurred.
If you encounter an error message other than those listed in the previous table, contact our Support Team with the complete error message. |
The wildcard-file() source collects log messages from multiple plain-text files from multiple directories. The wildcard-file() source is available in syslog-ng PE version 3.107.0.3 and later.
The syslog-ng PE application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When syslog-ng PE is restarted, it records the position of the last sent log message in the /opt/syslog-ng/var/syslog-ng.persist file, and continues to send messages from this position after the restart.
Declaration
wildcard-file(
base-dir("<pathname>")
filename-pattern("<filename>")
);
Note the following important points:
-
You can use the * and ? wildcard characters in the filename (the filename-pattern() option), but not in the path (the base-dir() option).
-
When using the wildcard-file() source, always set how often syslog-ng PE should check the files for new messages using the follow-freq() parameter.
-
If you use multiple wildcard-file() sources in your configuration, make sure that the files and folders that match the wildcards do not overlap. That is, every file and folder should belong to only one file source. Monitoring a file from multiple wildcard sources can lead to data loss.
-
When using wildcards, syslog-ng PE monitors every matching file (up to the limit set in the max-files() option), and can receive new log messages from any of the files. However, monitoring (polling) many files (that is, more than ten) has a significant overhead and may affect performance. On Linux this overhead is not so significant, because syslog-ng PE uses the inotify feature of the kernel. Set the max-files() option at least to the number of files you want to monitor. If the wildcard-file source matches more files than the value of the max-files() option, it is random which files will syslog-ng PE actually monitor. The default value of max-files() is 100.
-
If the message does not have a proper syslog header, syslog-ng PE treats messages received from files as sent by the kern facility. Use the default-facility() and default-priority() options in the source definition to assign a different facility if needed.
Required parameters: base-dir(), filename-pattern(). For the list of available optional parameters, see wildcard-file() source options.
Example: Using the wildcard-file() driver
The following example monitors every file with the .log extension in the /var/log directory for log messages.
