Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.24 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng PE The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files google-pubsub: collecting messages from the Google Pub/Sub messaging service wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mssql, oracle, sql: collecting messages from an SQL database network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files google_pubsub: Sending logs to the Google Cloud Pub/Sub messaging service hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel: Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql: Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

google-pubsub() source options

The google-pubsub() source has the following options.

Required parameters
  • credentials()

  • project()

  • subscription()

Optional parameters
  • ack-tracker-batch-size()

  • ack-tracker-timeout()

  • log-fetch-limit()

  • prefix()

  • time-reopen()

  • workers()

The google-pubsub() source options, in more detail:

ack-tracker-batch-size()
Type: string
Default: 100
Required: no

Description: Optional parameter.

The syslog-ng PE application retains acknowledgements on the source side and either acknowledges an ack-tracker-batch-size() number of messages in a batch, or sends acknowledgements after the ack-tracker-timeout() expires. If the value of your ack-tracker-timeout() is larger than the value of your Acknowledgement deadline, it may result in message duplication.

ack-tracker-timeout()
Type: time [milliseconds]
Default: 3000
Required: no

Description: Optional parameter.

The syslog-ng PE application retains acknowledgements on the source side and either acknowledges an ack-tracker-batch-size() number of messages in a batch, or sends acknowledgements after the ack-tracker-timeout() expires. If the value of your ack-tracker-timeout() is larger than the value of your Acknowledgement deadline, it may result in message duplication.

credentials()
Type: string
Default: n/a
Required: yes

Description: Required parameter.

The credentials of your Google Pub/Sub project.

log-fetch-limit()
Type: number
Default: 100
Required: no

Description: Optional parameter.

The maximum number of messages fetched from a source during a single poll loop.

prefix()
Type: string
Default: .pubsub.
Required: no

Description: Optional parameter.

This prefix will be added to the name of the macros created from the message attributes of the Google Pub/Sub message.

project()
Type: string
Default: n/a
Required: yes

Description: Required parameter.

The ID of your Google Pub/Sub project.

subscription()
Type: string
Default: n/a
Required: yes

Description: Required parameter.

The ID of your Google Pub/Sub subscription.

time-reopen()
Type: number (seconds)
Default: 60
Required: no

Description: Optional parameter.

The time to wait in seconds before a broken connection is reestablished.

workers()
Type: integer
Default: 1
Required: no

Description: Optional parameter.

Specifies the number of worker threads (at least 1) that syslog-ng PE uses to receive messages from the Google Pub/Sub messaging service. Increasing the number of worker threads can drastically improve the performance of the destination.

Preventing message duplication resulting from the At-Least-Once delivery behavior

The google-pubsub() source utilizes the At-Least-Once delivery behavior of Google Pub/Sub. This behavior is intentional, and its purpose is to avoid potential log loss. However, in certain cases, the At-Least-Once delivery behavior results in message duplication on the syslog-ng PE side.

Issue

On the Google Cloud Platform side, you can set the value of the Acknowledgment deadline (the default value is 10 seconds) when creating your Google Pub/Sub Subscription.

For more information, see Set up your Google Cloud project and Pub/Sub topic and subscriptions.

The syslog-ng PE application has to acknowledge log-fetch-limit() number of messages within the Acknowledgement deadline time limit. If syslog-ng PE does not acknowledge Google Pub/Sub messages no later than the time limit specified in the Acknowledgment deadline, the Google Pub/Sub service will attempt to redeliver the message to syslog-ng PE.

As a result, any acknowledgement sent later than the Acknowledgment deadline will result in message duplication on the syslog-ng PE side. This issue occurs most often if you have flow-control turned on, and your syslog-ng PE destinations are slow.

Workaround

To avoid message duplication, you can use one of these methods:

  • Using the disk-buffer option if flow control is on

    When using the disk-buffer option, syslog-ng PE acknowledges Pub/Sub messages as soon as they are sent to the output queue or overflow queue, instead of acknowledging them when the destination sends or rewrites them.

  • Adjusting the value of your ack-tracker-timeout() to the Acknowledgment deadline, and the value of your ack-tracker-batch-size() to your log-fetch-limit()

    The syslog-ng PE application acknowledges messages in batches. You can set the size (ack-tracker-batch-size(), the default value is 100), and timeout (ack-tracker-timeout(), the default value is 3000 milliseconds, or 3 seconds) of these batches.

    To avoid message duplication, set your ack-tracker-timeout() to a value not larger than the value of your Acknowledgment deadline, and your ack-tracker-batch-size() to a value not larger than your log-fetch-limit().

Error messages you may encounter while using the google-pubsub() source

Common error messages with workaround solutions

The following table describes the possible error messages that you may encounter while using the google-pubsub() source.

Error message Possible reason(s)/resolution(s)
DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application.
Use the credentials() option to specify a key file.

For more information, see Create service account credentials.

404 Resource not found (resource=...).

The specified subscription is not valid. Check the value of the subscription() option. Make sure you specify the Subscription ID, not the name of the subscription.

For more information, see Set up your Google Cloud project and Pub/Sub topic and subscriptions.

400 Invalid resource name given (name=...).

The specified subscription is not valid. Check the value of the subscription() option. Make sure you specify the Subscription ID, not the name of the subscription.

For more information, see Set up your Google Cloud project and Pub/Sub topic and subscriptions.

404 Requested project not found or user does not have access to it (project=nonproj). Make sure to specify the unique project identifier and not the Google Cloud Console display name.
Make sure to specify the unique project identifier and not the Google Cloud Console display name.
PubSubSource init, failed to load credentials file, path=...

The file specified in the credentials() option does not exist, or it is not accessible. Check the path and its permissions.

Error messages that require assistance from our Support Team

In some cases, you may encounter different error messages that require assistance from our Support Team. If you encounter similar error messages as those listed in the following table, One Identity recommends that you contact our Support Team for assistance.

Error message Possible reason(s)/resolution(s)
PubSubSource error while pulling messages, error: ...

A non-retryable error occurred.

If you encounter an error message other than those listed in the previous table, contact our Support Team with the complete error message.

PubSubSource error while acking messages, error: ...

A non-retryable error occurred.

If you encounter an error message other than those listed in the previous table, contact our Support Team with the complete error message.

wildcard-file: Collecting messages from multiple text files

The wildcard-file() source collects log messages from multiple plain-text files from multiple directories. The wildcard-file() source is available in syslog-ng PE version 7.0.3 and later.

The syslog-ng PE application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When syslog-ng PE is restarted, it records the position of the last sent log message in the /opt/syslog-ng/var/syslog-ng.persist file, and continues to send messages from this position after the restart.

Declaration
wildcard-file(
    base-dir("<pathname>")
    filename-pattern("<filename>")
);

Note the following important points:

  • You can use the * and ? wildcard characters in the filename (the filename-pattern() option), but not in the path (the base-dir() option).

  • When using the wildcard-file() source, always set how often syslog-ng PE should check the files for new messages using the follow-freq() parameter.

  • If you use multiple wildcard-file() sources in your configuration, make sure that the files and folders that match the wildcards do not overlap. That is, every file and folder should belong to only one file source. Monitoring a file from multiple wildcard sources can lead to data loss.

  • When using wildcards, syslog-ng PE monitors every matching file (up to the limit set in the max-files() option), and can receive new log messages from any of the files. However, monitoring (polling) many files (that is, more than ten) has a significant overhead and may affect performance. On Linux this overhead is not so significant, because syslog-ng PE uses the inotify feature of the kernel. Set the max-files() option at least to the number of files you want to monitor. If the wildcard-file source matches more files than the value of the max-files() option, it is random which files will syslog-ng PE actually monitor. The default value of max-files() is 100.

  • If the message does not have a proper syslog header, syslog-ng PE treats messages received from files as sent by the kern facility. Use the default-facility() and default-priority() options in the source definition to assign a different facility if needed.

Required parameters: base-dir(), filename-pattern(). For the list of available optional parameters, see wildcard-file() source options.

Example: Using the wildcard-file() driver

The following example monitors every file with the .log extension in the /var/log directory for log messages.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating