Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.20 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel: Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql: Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

Summary of changes

This section lists the changes of The syslog-ng Premium Edition(syslog-ng PE) Administration Guide.

Version 7.0.19 - 7.0.20

Version 7.0.20 of syslog-ng PE includes the following main features.

  • Red Hat EL 8 support

    Starting with version 7.0.20, syslog-ng PE supports Red Hat EL 8 (kernel version 4.18).

    For more information, see Supported platforms and udp-balancer: Receiving UDP messages at very high rate.

  • SNMP destination

    The snmp() destination of syslog-ng PE can send log messages to an SNMP destination.

    For more information, see snmp: Sending SNMP traps.

  • The syslog-ng PE application uses its own Python interpreter

    Starting from version 7.0.20, the syslog-ng PE application uses its own Python interpreter (shipped with the default syslog-ng PE installation) instead of the system's Python interpreter.

    NOTE: The syslog-ng PE application's built-in Python interpreter only supports Python 3. As a result, any custom Python code used in syslog-ng PE must be compatible with Python 3.

  • Disabling and forcing TLSv1.3 in TLS contexts
    • The tlsv1_3 value has been added to the available values of ssl-version() for drivers based on the http() destination.

    • The no-tlsv13 value has been added to the available values of ssl-options() in TLS contexts.

  • New statistics item: the connections counter

    The connections counter of syslog-ng PE displays the number of active connections for network-based sources in the syslog-ng-ctl stats statistics information.

    For more information, see connections statistics counter.

Version 7.0.18 - 7.0.19

Version 7.0.19 of syslog-ng PE includes the following main features.

  • Sending logs to Microsoft Azure Sentinel

    Version 7.0.19 of syslog-ng PE can directly post log messages to the Microsoft Azure Sentinel cloud using Microsoft Azure Sentinel's public HTTP Data Collector API interface.

    For more information, see sentinel: Sending logs to the Microsoft Azure Sentinel cloud.

  • Persist name assigned to Python sources and destinations

    Starting with 7.0.19, syslog-ng PE assigns a persist name to Python sources and destinations. The persist name is generated from the class name. If you want to use the same Python class multiple times in your syslog-ng PE configuration, add a unique persist-name() to each source or destination, otherwise syslog-ng PE will not start. For example:

    log {
        source { python(class(PyNetworkSource) options("port" "8080") persist-name("<unique-string>); };
        source { python(class(PyNetworkSource) options("port" "8081")); };

    Alternatively, you can include the following line in the Python package: @staticmethod generate_persist_name. For example:

    from syslogng import LogSource
      class PyNetworSource(LogSource):
        def generate_persist_name(options):
            return options["port"]
        def run(self):
        def request_exit(self):
  • OpenSSL version 1.1.1d

    From version 7.0.19, syslog-ng PE supports OpenSSL 1.1.1d for Linux glibc 2.11 and Ubuntu Bionic.

    For more information, see Prerequisites to installing syslog-ng PE.

  • Changes in documentation
Version 7.0.14 - 7.0.16
Changes in product
Changes in documentation
Version 7.0.14 - 7.0.16
Changes in product
Changes in documentation
Version 7.0.13 - 7.0.14
Changes in product
Version 7.0.12 - 7.0.13
Changes in product
  • The Linux Audit Parser can parse the log messages of the Linux Audit subsystem (auditd). The syslog-ng PE application can separate these log messages to name-value pairs. For details, see "Linux audit parser" in the Administration Guide.

  • The windowsevent() source can now automatically process XML arrays, making the array elements available as name-value pairs. For details, see "windowsevent: Collecting Windows event logs" in the Administration Guide.

  • Installing the syslog-ng Premium Edition application in Docker containers is now officially supported on CentOS 7, Red Hat EL 7.5, and Ubuntu 18.04 (Bionic Beaver) platforms. For details, see "Installing syslog-ng in Docker" in the Administration Guide.

  • The persist-tool utility is now part of the syslog-ng PE package. For details, see the persist-tool manual page.

  • Since ElasticSearch version 1.x has reached its end of life, its support has been removed from syslog-ng PE. Use the elasticsearch2 destination instead.

Version 7.0.11 - 7.0.12
Changes in product
  • Version 7.0.12 of syslog-ng PE can directly post log messages to a Splunk deployment using the HTTP Event Collector (HEC) over the HTTP and Secure HTTP (HTTPS) protocols. The solution is optimized for performance, and supports sending messages in batch mode, multithreaded message sending, and load-balancing to multiple Splunk indexer nodes.

    HTTPS connection, as well as password- and certificate-based authentication is supported. The content of the events is sent in JSON format.

    For details, see "splunk-hec: Sending messages to Splunk HTTP Event Collector" in the Administration Guide.

  • The http() destination now supports load balancing, so a single syslog-ng PE instance can feed log data to multiple HTTP servers, for example, multiple ingestion nodes of an Elasticsearch cluster. For details, see "Batch mode and load balancing" in the Administration Guide.

    HTTP and HTTPS redirections now also handled automatically.

  • The syslog() and network() drivers now support the so-reuseport() option that allows multiple sockets on the same host to bind to the same port, improving the performance of multithreaded network server applications running on top of multicore systems.

  • The Cisco parser now supports Cisco Catalyst formatted triplets.

  • Version 7.0.12 of syslog-ng PE is now available on the Ubuntu 18.04 platform. Note that the Java-based drivers of syslog-ng PE (used for Apache Kafka, Elasticsearch, HDFS) require Java 8, Java 10 is not supported.

  • The allow-compress() option of the ALTP communication has been renamed to allow-plain-compress().

Version 7.0.10 - 7.0.11
Changes in product
Changes in documentation

Extending syslog-ng PE in Python has been supported for several releases, but so far this feature was mostly undocumented. Now you can find more details about this feature in "python: writing custom Python destinations" in the Administration Guide.

Version 7.0.9 - 7.0.10
Changes in product
  • It is now possible to not only receive but also send messages using ALTP to hosts that are running version 6 or at least version 7.0.9 of syslog-ng PE or the syslog-ng Agent for Windows application. For details, see Advanced Log Transfer Protocol .
  • The global flush-lines() option does not affect the batch size of ALTP any more. Instead, a new ALTP has been introduced, called batch-size(). For details, see ALTP options.
  • The failover() option allows you to specify what happens after syslog-ng PE fails over to a secondary server. Additionally, the failover-servers() option has been deprecated and removed from the document. For more information about the failover() option, see Client-side failover.

  • You can now refer to any additional parameters at the end of the argument in a block by adding three dots to it (). It tells syslog-ng PE that this macro accepts `__VARARGS__`, therefore any name-value pair can be passed without validation. For details, see Passing arguments to configuration blocks.

  • You can now make parameters mandatory in block definitions by defining them with empty brackets (). For details, see Mandatory parameters.

Changes in documentation
  • A note about JVM still running after deleting all Java destinations and reloading syslog-ng has been added to the description of Java destinations.

  • The default value of the --skip-tokens parameter of the loggen application has been changed to 0. For details, see The loggen manual page.

Version 7.0.8 - 7.0.9
Changes in product
Version 7.0.7 - 7.0.8
Changes in product
  • A new log path flag, drop-unmatched, has been added. The new flag causes messages to be dropped along a log path when they do not match a filter or are discarded by a parser. For details, see "Log path flags" in the Administration Guide.

  • Support for Elasticsearch's Shield has been removed.

  • Support for POSIX regular expressions has been removed.

    All "posix" regular expressions are automatically switched to "pcre". In case you have POSIX regular expressions configured, ensure that they work with PCRE, and also specify type("pcre") explicitly.

Version 7.0.6 - 7.0.7
Changes in product
Changes in documentation
Version 7.0.5 - 7.0.6
Changes in product
Changes in documentation
Version 7.0.4 - 7.0.5
Changes in product
Changes in documentation
Version 7.0.3 - 7.0.4
Changes in product
Changes in documentation
  • Several corrections and editorial changes.

Version 7.0.2 - 7.0.3
Changes in product
Changes in documentation
Version 7.0 - 7.0.2
Changes in product
Changes in documentation
  • Several corrections and editorial changes.

Version 6 LTS - 7.0
Changes in product

Several features that are available in syslog-ng Premium Edition 6 LTS are not yet available in syslog-ng PE 7. For a list of features that are available in syslog-ng PE 6 LTS but not in 7 see "Differences in features between syslog-ng PE 6 LTS and 7" in the Release Notes.

Changes in documentation


One Identity would like to express its gratitude to the syslog-ng users and the syslog-ng community for their invaluable help and support.

Introduction to syslog-ng

This chapter introduces the syslog-ng Premium Edition application in a non-technical manner, discussing how and why is it useful, and the benefits it offers to an existing IT infrastructure.

What syslog-ng is

The syslog-ng Premium Edition (syslog-ng PE) application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions. Among others, syslog-ng PE allows you the following.

Secure and reliable log transfer

The syslog-ng PE application enables you to send the log messages of your hosts to remote servers using the latest protocol standards. You can collect and store your log data centrally on dedicated log servers. Transfer log messages using the ALTP protocol ensures that no messages are lost.

The disk-buffer option for messages

To minimize the risk of losing important log messages, the syslog-ng PE application can store messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished, in the same order the messages were received. The disk-buffer option is persistent – no messages are lost even if syslog-ng is restarted.

Secure logging using TLS

Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng PE supports the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows you to authenticate your clients and the logserver using X.509 certificates.

Flexible data extraction and processing

Most log messages are inherently unstructured, which makes them difficult to process. To overcome this problem, syslog-ng PE comes with a set of built-in parsers, which you can combine to build very complex things.

Filter and classify

The syslog-ng PE application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. You can create directories, files, and database tables dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.

Parse and rewrite

The syslog-ng PE application can segment log messages to named fields or columns, and also modify the values of these fields. You can process JSON messages, key-value pairs, and more.

To get the most information out of your log data, syslog-ng PE allows you to correlate log messages and aggregate the extracted information into a single message. You can also use external information to enrich your log data.

Big data clusters

The log data that your organization has to process, store, and review increases daily, so many organizations use big data solutions for their logs. To accomodate this huge amount of data, syslog-ng PE natively supports storing log messages in HDFS files and Elasticsearch clusters.

Message queue support

Large organizations increasingly rely on queuing infrastructure to transfer their data. For that purpose, syslog-ng PE supports Apache Kafka.

SQL, NoSQL, and monitoring

Storing your log messages in a database allows you to easily search and query the messages and interoperate with log analyzing applications. The syslog-ng application supports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, and SQLite.

Wide protocol and platform support
syslog protocol standards

syslog-ng not only supports legacy BSD syslog (RFC3164) and the enhanced RFC5424 protocols, but also JavaScript Object Notation (JSON) and journald message formats.

Heterogeneous environments

The syslog-ng PE application is the ideal choice to collect logs in massively heterogeneous environments using several different operating systems and hardware platforms, including Linux, Unix, BSD, Sun Solaris, HP-UX, and AIX.

IPv4 and IPv6 support

The syslog-ng application can operate in both IPv4 and IPv6 network environments, and can receive and send messages to both types of networks.

Encrypted and timestamped log storage

The syslog-ng PE application can store log messages securely in encrypted, compressed, and timestamped binary files. Timestamps can be requested from an external Timestamping Authority (TSA).

Excellent performance

Depending on the exact syslog-ng PE configuration, environment, and other parameters, syslog-ng PE is capable of processing:

  • Over 635,000 messages per second (over 235 MB of data per second) when receiving messages from multiple connections and storing them in text files.

  • Over 615,000 messages per second (over 230 MB of data per second) when receiving messages from multiple secure (TLS-encrypted) connections and storing them in text files.

Related Documents