syslog-ng Open Source Edition 3.22 - Administration Guide

Version 3.21 - 3.22

Version 3.22 of syslog-ng Open Source Edition includes the following main features.

Version 3.20 - 3.21

Version 3.19 - 3.20

Version 3.18 - 3.19

Version 3.17 - 3.18

Version 3.16 - 3.17

Changes in product:

• A new destination driver, telegram(), has been added. The telegram() destination sends log messages to Telegram, which is a secure, cloud-based mobile and desktop messaging app. For more information, see Telegram: Sending messages to Telegram.

• A new template function, urlencode, has been added. You can use the urlencode template function together with the telegram() destination to send syslog messages to Telegram. For more information, see Template functions of syslog-ng OSE.

• To ensure that a module is loaded, you can use the @requires statement. For more information, see Loading modules.
• The add-contextual-data() has been extended with the ignore-case() option. For more information, see Options add-contextual-data().
• The hook-commands() has been added, which makes it possible to execute external programs when they are initialized or torn down. The hook-commands() can be used for both source and destination drivers. For more information, see hook-commands().

Changes in product:

• It is now possible to use if {}, elif {}, and else {} blocks to configure conditional expressions. For details, see if-else-elif: Conditional expressions.

• A new log path flag, drop-unmatched, has been added. The new flag causes messages to be dropped along a log path when they do not match a filter or are discarded by a parser. For details, see Log path flags.

• Support for Elasticsearch's Shield has been removed.

• Support for POSIX regular expressions has been removed.

Changes in product:

• You can use password-protected private keys in the network() and syslog() source and destination drivers. For details, see Password-protected keys.

• To better control to which log messages you add contextual data, you can use filters as selectors. In this case, the first column of the CSV database file must contain the name of a filter. For each message, syslog-ng OSE evaluates the filters in the order they appear in the database file. If a filter matches the message, syslog-ng OSE adds the name-value pair related to the filter. For details, see Using filters as selector.

Changes in product:

• A new source driver, stdin(), has been added. The stdin() driver collects messages from the standard input stream. For more information, see stdin: Collecting messages from the standard input stream.

• A new destination, Sending logs to Graylog, and a template to send syslog messages to Graylog, format-gelf, has been added.

• A new template function, getent, has been added. You can use the getent template function to look up entries from the Name Service Switch libraries. For more information, see getent.

• The default values of the --enable-json, --enable-mongodb, and --with-libmongo-client compile parameters have changed. For more information, see Compiling options of syslog-ng OSE.

• A new compile option, --with-module-path, has been added. The new option specifies syslog-ng OSE's module installation directory. For more information, see Compiling options of syslog-ng OSE.

• A new destination driver, osquery(), has been added. The new driver sends log messages to osquery's syslog table. For more information, see osquery: Sending log messages to osquery's syslog table.

• It is now possible to specify TLS options in a tls() block. For more information, see:

  • amqp() destination options

  • HTTP destination options

  • riemann() destination options

• Support for microseconds in Riemann destinations has been introduced. For more information, see event-time().

• Module auto-loading now also works for the system() source. For more information, see --default-modules .

Changes in documentation:

• A new section describing common error messages has been added to the document. For more information, see Error messages .

• Several corrections and editorial changes.

Changes in product:

• A new systemd-journal() source option, called read-old-records(), has been added. For more information, see read-old-records().

• An option called jvm-options() has been added, which allows you to fine-tune Java Virtual Machine settings when configuring Elasticsearch, HDFS, and Apache Kafka destinations, or web services to which you send log messages via the HTTP protocol. For details, see:

  • Elasticsearch2 destination options (DEPRECATED)

  • HDFS destination options

  • HTTP destination options

  • Kafka destination options

  • Global options

• A new HDFS destination option, called hdfs-append-enabled() has been added. For further information, see hdfs-append-enabled().

• Macros are now supported in the hdfs-file() option. For details, see hdfs-file().

• The following new TLS options have been added:

  • dhparam-file()

  • ecdh-curve-list()

  • pkcs12-file().

• A new parser, capable of processing input in XML format, has been added. For more information, see XML parser.

Changes in documentation:

• Added section about commercial version of syslog-ng. For more information, see Commercial version of syslog-ng.

• Added warning about the requirement to delete the persist file once the dir() option of disk-buffer() has been modified or a new one has been added. For more information, see destination: Forward, send, and store log messages.

• Clarified information about the Python parser's deinit() method. It runs not only at a syslog-ng graceful stop, but at a reload too. For details, see Methods of the python() parser.

• Several corrections and editorial changes.

Changes in product:

• Looking up GeoIP2 data from IP addresses has been added to the document.

• http: Posting messages over HTTP without Java has been upgraded with new improvements.

• The geoip() parser is now deprecated. Looking up GeoIP data from IP addresses (DEPRECATED).

• The template() option has been added to the Apache access log parser. For details, see: Apache access log parser.

• SSL-related options have been added to amqp() destination. For details, see: amqp() destination options.

• The prefix() option has been added to the Cisco parser. For details, see: Cisco parser.

• The drop-unmatched() option has been added to the db-parser() statement. For details, see: Using pattern databases.

• The event-time() option has been added to the Riemann destination. For details, see: riemann: Monitoring your data with Riemann.

Changes in documentation:

• A new example has been added to the osquery() source. For details, see: osquery: Collect and parse osquery result logs.

• Several corrections and editorial changes.

Changes in product:

• wildcard-file: Collecting messages from multiple text files has been added to the document.

• snmptrap: Read Net-SNMP traps has been added to the document.

• osquery: Collect and parse osquery result logs has been added to the document.

• The elasticsearch2() destination now supports HTTPS mode, including encryption, and also password- and certificate-based authentication. For details, see elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED).

• The http() destination now supports encryption, and also password- and certificate-based authentication. For details, see HTTP destination options.

• The hdfs() destination now supports Kerberos authentication. For details, see Kerberos authentication with syslog-ng hdfs() destination.

• Python parser has been added to the document.

• Cisco parser has been added to the document.

• map-value-pairs: Rename value-pairs to normalize logs has been added to the document.

• The list-* template functions allow you to manipulate comma-separated lists. For details, see List manipulation.

• The new basename() and dirname() template functions allow you to easily separate the path and filenames. For details, see Template functions of syslog-ng OSE.

• stardate has been added to the document.

• create-statement-append() has been added to the document.

• The default value of the log-msg-size() option has been increased to 64k. That way syslog-ng OSE will not truncate long log messages, which are getting increasingly common.

Changes in documentation:

• Splunk: Sending log messages to Splunk has been added to the document.

• About disk queue files has been added to the document.

• An example failure script has been added to Running a failure script.

• Several corrections and editorial changes.

Changes in product:

• When using TLS-transport, you can now use certain fields of the X.509 certificates as macros. For details, see .TLS.X509.

• The elastic2() destination driver now supports Search Guard, an alternative security solution for Elasticsearch. For details, see Search Guard and syslog-ng OSE.

• .TLS.X509 has been added to the document.

• Unsetting message fields has been updated with groupunset().

Changes in documentation:

• Corrections and editorial changes.

Changes in product:

• Enriching log messages with external data has been added to the document.

• Correlating log messages has been added to the document.

• elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) has been added to the document.

• http: Posting messages over HTTP without Java has been added to the document.

• logmatic: Using Logmatic.io has been added to the document.

• loggly: Using Loggly has been added to the document.

• Disk-based buffering has been added to syslog-ng OSE. For details, see Using disk-based and memory buffering.

• What's new in the syslog-ng pattern database format V5, , has been added to Element: create-context has been added to db-parser: Process message content with a pattern database (patterndb).

• Parsing dates and timestamps has been added to parser: Parse and segment structured messages.

• Apache access log parser has been added to parser: Parse and segment structured messages.

• New options of the set() rewrite operator have been added to Setting message fields to specific values.

• A rewrite operator to unset fields has been added to Unsetting message fields.

• A template function that formats name-value pairs as ArcSight Common Event Format extension has been added to format-cef-extension.

• Numerical template functions that work on numerical values of a correlation context have been added to Numerical operations.

• The inherit-environment() option has been added to program: Receiving messages from external applications and program: Sending messages to external applications.

• @NLSTRING@ has been added to Using pattern parsers.

Changes in documentation:

• Looking up GeoIP data from IP addresses (DEPRECATED) has been moved to Enriching log messages with external data.

• Several corrections and editorial changes.

Changes in product:

• mbox: Converting local e-mail messages to log messages has been added to the document.

• The keep-alive() option has been added to the program() destination.

• Linux audit parser has been added to parser: Parse and segment structured messages.

• python has been added to Template functions of syslog-ng OSE.

• Posting messages over HTTP has been added to the document.

• Write your own custom destination in Java or Python has been added to the document.

• Looking up GeoIP data from IP addresses (DEPRECATED) has been added to the document.

• kafka: Publishing messages to Apache Kafka (Java implementation) has been added to the document.

• hdfs: Storing messages on the Hadoop Distributed File System (HDFS) has been added to the document.

• Parsing key=value pairs has been added to the document.

• format-cim has been added to the document.

• Simple templates can be defined without braces. Templates can also reference other templates. For details, see Templates and macros.

• Custom template functions can be defined in the syslog-ng OSE configuration. For details, see Using template functions.

• CSV-parsers can use strings as delimiters. For details, see delimiters().

• IPv6 addresses can be filtered using a new filter. For details, see netmask6().

• The loggen utility can send messages indefinitely using the --permanent option.

• The ssl-options() option has beed added to TLS options.

• TLS-support has been added to riemann() destination options.

• The extract-solaris-msgid() parser has beed added to sun-streams: Collecting messages on Sun Solaris.

• The context option of inherit-properties has beed added to Actions and message correlation.

• riemann() destination options has been added to the document.

• The sanitize-utf8 flag has been added to the list of source flags.

• The format-welf function has been added to Template functions of syslog-ng OSE.

• The pass-unix-credentials() option has been added to Global options of syslog-ng OSE.

• The use-uniqid() option has been added to Global options of syslog-ng OSE.

• The UNIQID macro has been added to Macros of syslog-ng OSE.

• The JSON-parser now handles special characters in object names. For details, see extract-prefix().

• The syslog-debun tool used to generate syslog-ng OSE debug bundles has been documented. For details, see The syslog-ng-debun manual page.

• The --control option has been added to the The syslog-ng manual page manual page.

• Version 3.7 and newer automatically includes the plugin.conf files from the <directory-where-syslog-ng-is-installed>/scl/*/ directories, making it easier to use and distribute configuration blocks.

• The --enable-all-modules compiler option has beed added to Compiling options of syslog-ng OSE.

• The create-dirs() option has been added to unix-stream() and unix-dgram() destination options.

Changes in documentation:

• Generating configuration blocks from a script has been added to the document.

• Example: Sending alert when a client disappears has been added to the document.

• The tcp(), tcp6(), udp(), udp6() source and destination drivers have been deprecated, as all of their functionality can be achieved with the network() driver. For help on migrating to the network() driver, see Change an old source driver to the network() driver and Change an old destination driver to the network() driver.

• The beginning of Troubleshooting syslog-ng has been extended with basic troubleshooting information.

• The description of the chain-hostnames() global option has been clarified and extended. For details, see chain-hostnames().

• Other editorial corrections.

Changes in product:

Changes in documentation:

• riemann: Monitoring your data with Riemann has been added to the document.

• nodejs: Receiving JSON messages from nodejs applications has been added to the document.

• systemd-journal: Collecting messages from the systemd-journal system log storage has been added to the document.

• systemd-syslog: Collecting systemd messages using a socket has been added to the document.

• use-rcptid() has been added to the document.

• Setting multiple message fields to specific values has been added to the document.

• The retries and throttle options are available for the SMTP, MongoDB, AMQP, and Redis destinations.

• The description of the multi-line-mode option has been updated.

• UNIX credentials and other metadata has been added to the document.

• RUNID has been added to Macros of syslog-ng OSE.

• The extract-prefix option has been added to JSON parser.

• The graphite-output, or and padding template functions have been added to Template functions of syslog-ng OSE.

• PCRE is now a required dependency of syslog-ng OSE, and by default, syslog-ng OSE uses PCRE-style regular expressions. Therefore, the --enable-pcre compliation option has been removed.

• graphite: Sending metrics to Graphite has been added to the document.

• pseudofile() has been added to the document.

• The custom-domain() and stats-lifetime() options have been added to Global options.

• The retry_sql_inserts option has been renamed to retries to increase consistency.

• on-error() can be set locally for MongoDB destinations as well. Also, MongoDB destinations support the username and password options, and connecting to the server using UNIX domain sockets. For details, see mongodb: Storing messages in a MongoDB database.

• How syslog-ng OSE connects the MongoDB server has been added to the document.

• Several typos and syntax errors in examples have been corrected.

Secure and reliable log transfer

The syslog-ng OSE application enables you to send the log messages of your hosts to remote servers using the latest protocol standards. You can collect and store your log data centrally on dedicated log servers. Transfer log messages using the TCP protocol ensures that no messages are lost.

Disk-based message buffering

To minimize the risk of losing important log messages, the syslog-ng OSE application can store messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished, in the same order the messages were received. The disk buffer is persistent – no messages are lost even if syslog-ng is restarted.

Secure logging using TLS

Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng OSE supports the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows you to authenticate your clients and the logserver using X.509 certificates.

Flexible data extraction and processing

Most log messages are inherently unstructured, which makes them difficult to process. To overcome this problem, syslog-ng OSE comes with a set of built-in parsers, which you can combine to build very complex things.

Filter and classify

The syslog-ng OSE application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. You can create directories, files, and database tables dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.

Parse and rewrite

The syslog-ng OSE application can segment log messages to named fields or columns, and also modify the values of these fields. You can process JSON messages, key-value pairs, and more.

To get the most information out of your log data, syslog-ng OSE allows you to correlate log messages and aggregate the extracted information into a single message. You can also use external information to enrich your log data.

Big data clusters

The log data that your organization has to process, store, and review increases daily, so many organizations use big data solutions for their logs. To accomodate this huge amount of data, syslog-ng OSE natively supports storing log messages in HDFS files and Elasticsearch clusters.

Message queue support

Large organizations increasingly rely on queuing infrastructure to transfer their data. For that purpose, syslog-ng OSE supports Apache Kafka, the Advanced Message Queuing Protocol (AMQP), and the Simple Text Oriented Messaging Protocol (STOMP).

SQL, NoSQL, and monitoring

Storing your log messages in a database allows you to easily search and query the messages and interoperate with log analyzing applications. The syslog-ng application supports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, and SQLite.

syslog-ng OSE also allows you to extract the information you need from your log data, and directly send it to your Graphite, Redis, or Riemann monitoring system.

syslog protocol standards

syslog-ng not only supports legacy BSD syslog (RFC3164) and the enhanced RFC5424 protocols but also JavaScript Object Notation (JSON) and journald message formats.

Heterogeneous environments

The syslog-ng OSE application is the ideal choice to collect logs in massively heterogeneous environments using several different operating systems and hardware platforms, including Linux, Unix, BSD, Sun Solaris, HP-UX, Tru64, and AIX.

IPv4 and IPv6 support

The syslog-ng application can operate in both IPv4 and IPv6 network environments, and can receive and send messages to both types of networks.