This guide is a work-in-progress document with new versions appearing periodically.
The latest version of this document can be downloaded from the syslog-ng Documentation page.
Changes in product:
The syslog-ng PE application can now separate a message consisting of whitespace or comma-separated key=value pairs (for example, Postfix log messages) into name-value pairs. You can also specify other separator character instead of the equal sign, for example, colon (:) to parse MySQL log messages. For details, see the section called “Parsing key=value pairs”.
The OpenSSL application used in syslog-ng PE has been updated to version 1.0.2, so you can use new, stronger ciphers to protect the communication between your syslog-ng PE clients and servers. For an updated list of supported ciphers, see the section called “cipher-suite()”.
You can now specify the curves that are permitted in the connection using the curve-list() option, and also import Diffie-Hellman parameters from a file using the dhparam-file() option. For details, see the section called “TLS options”.
A new file source driver option, force-directory-polling(), has been introduced, which can be used when the filename specified as the source file contains a wildcard to force the polling matching logifles. For details, see the section called “Collecting messages from text files” and the section called “force-directory-polling()”.
Changes in documentation:
Extended the list of internal() source options with options host-override(), log-iw-size(), normalize-hostnames(), program-override(), and use-fqdn(). For details, see the section called “host-override()”, the section called “log-iw-size()”, the section called “normalize-hostnames()”, the section called “program-override()”, and the section called “use-fqdn()”.
Added information about max-uid-query().
Added warning about the requirement to delete the persist file once the dir() option of disk-buffer() has been modified or a new one has been added. For more information, see Chapter 7, Sending and storing log messages — destinations and destination drivers.
Clarified information on the usage of the cert-subject() TLS option in the section called “cert-subject()”.
Reworked the section called “Optimizing multithreaded performance” to make information more accessible.
Editorial corrections.
Changes in product:
Elasticsearch 2.x destination and Shield support is now available in the product. For details, see the section called “Sending messages directly to Elasticsearch version 2.0 or higher”.
A new template function, format-cef-extension is available to format name-value pairs as ArcSight Common Event Format extensions. For details, see the section called “format-cef-extension”.
The Debian 6 (squeeze) platform is no longer supported.
You can now specify the location where syslog-ng PE stores the disk-buffer files using the dir() option of disk-buffer().
Value-pairs now have a new option to select every value-pair that has a name beginning with a specified prefix, but remove the prefix when formatting the message. For details, see the section called “value-pairs()”.
Changes in documentation:
Editorial corrections.
Changes in product:
the section called “Storing messages on the Hadoop Distributed File System (HDFS)” has been added to the document.
CSV-parsers can use strings as delimiters. For details, see the section called “delimiters()”.
Changes in documentation:
The tcp(), tcp6(), udp(), udp6() source and destination drivers have been deprecated, as all of their functionality can be achieved with the network() driver. For help on migrating to the network() driver, see Procedure 6.1, “Change an old source driver to the network() driver” and Procedure 7.8, “Change an old destination driver to the network() driver”.
The beginning of Chapter 19, Troubleshooting syslog-ng has been extended with basic troubleshooting information.
the section called “Supported platforms” has been updated.
Other editorial corrections.
Changes in product:
the section called “netmask6()” has been added to the document.
the section called “Security-enhanced Linux: grsecurity, SELinux” has been added to the document.
New examples have been added to the section called “hash”.
The use-rcptid() global option has been deprecated. the section called “use-uniqid()” has been added to the document.
The assume-utf8 source flag has been documented.
Multiple message fields can be modified using the groupset rewrite rule. For details, see the section called “Setting multiple message fields to specific values”.
Platforms CentOS 7, Ubuntu 14.04 LTS (Trusty Tahr), and Red Hat ES 7 are now supported.
the section called “Generating SMTP messages (e-mail) from logs” has been added to the document.
the section called “Collecting messages from the systemd-journal system log storage” has been added to the document.
Log rotation with syslog-ng PE has been described in the section called “Configuring log rotation”.
The inlist() filter has been added to the section called “Filter functions”.
The option time-sleep() is now deprecated.
Changes in product:
the section called “Storing messages in a MongoDB database” has been added to the document.
A new parser is available to parse JSON-formatted messages. For details, see the section called “The JSON parser”.
the section called “format-json” has been added to the document.
Changes in documentation:
The retry_sql_inserts option has been renamed to retries to increase consistency.
the section called “on-error()” can be set locally for MongoDB destinations as well.
the section called “timestamp-freq()” has been added to Chapter 9, Global options of syslog-ng PE.
The FILE_NAME and SOURCE macros have been added to the section called “Macros of syslog-ng PE”.
the section called “Structuring macros, metadata, and other value-pairs” has been added to the document.
the section called “on-error()” has been added to the document.
the section called “Supported platforms” has been updated.
Chapter 12, Reliable Log Transfer Protocol™ has been expanded, including several clarifications and improvements.
Option api() has been renamed to event-api() in the section called “eventlog() source options”.
Any feedback is greatly appreciated, especially on what else this document should cover. General comments, errors found in the text, and any suggestions about how to improve the documentation is welcome at documentation@balabit.com.
This chapter introduces the syslog-ng Premium Edition application in a non-technical manner, discussing how and why is it useful, and the benefits it offers to an existing IT infrastructure.
The syslog-ng Premium Edition (syslog-ng PE) application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions. Among others, syslog-ng PE allows you the following.
The syslog-ng PE application enables you to send the log messages of your hosts to remote servers using the latest protocol standards. You can collect and store your log data centrally on dedicated log servers. Transfer log messages using the RLTP™ protocol ensures that no messages are lost.
Disk-based message buffering. To minimize the risk of losing important log messages, the syslog-ng PE application can store messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished, in the same order the messages were received. The disk buffer is persistent – no messages are lost even if syslog-ng is restarted.
Secure logging using TLS. Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng PE supports the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows you to authenticate your clients and the logserver using X.509 certificates.
Most log messages are inherently unstructured, which makes them difficult to process. To overcome this problem, syslog-ng PE comes with a set of built-in parsers, which you can combine to build very complex things.
Filter and classify. The syslog-ng PE application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. You can create directories, files, and database tables dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.
Parse and rewrite. The syslog-ng PE application can segment log messages to named fields or columns, and also modify the values of these fields. You can process JSON messages, key-value pairs, and more.
To get the most information out of your log data, syslog-ng PE allows you to correlate log messages and aggregate the extracted information into a single message. You can also use external information to enrich your log data.
The log data that your organization has to process, store, and review increases daily, so many organizations use big data solutions for their logs. To accomodate this huge amount of data, syslog-ng PE natively supports storing log messages in HDFS files and Elasticsearch clusters.
Large organizations increasingly rely on queuing infrastructure to transfer their data. syslog-ng PE supports Apache Kafka, and the Simple Text Oriented Messaging Protocol (STOMP).
Storing your log messages in a database allows you to easily search and query the messages and interoperate with log analyzing applications. The syslog-ng application supports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, and SQLite.
syslog protocol standards. syslog-ng not only supports legacy BSD syslog (RFC3164) and the enhanced RFC5424 protocols, but also JavaScript Object Notation (JSON) and journald message formats.
Heterogeneous environments. The syslog-ng PE application is the ideal choice to collect logs in massively heterogeneous environments using several different operating systems and hardware platforms, including Linux, Unix, BSD, Sun Solaris, HP-UX, Tru64, and AIX.
IPv4 and IPv6 support. The syslog-ng application can operate in both IPv4 and IPv6 network environments, and can receive and send messages to both types of networks.
When transferring messages to a remote server, the syslog-ng PE clients can be configured to send the log messages to secondary servers if the primary server becomes unaccessible.
The syslog-ng PE application can store log messages securely in encrypted, compressed, and timestamped binary files. Timestamps can be requested from an external Timestamping Authority (TSA).
Depending on the exact syslog-ng PE configuration, environment, and other parameters, syslog-ng PE is capable of processing:
Over 590,000 messages per second (over 220 MB of data per second) when receiving messages from multiple connections and storing them in text files.
Over 560,000 messages per second (over 210 MB of data per second) when receiving messages from multiple connections and storing them in logstore files (that is, encrypted files).
Over 565,000 messages per second (over 210 MB of data per second) when receiving messages from multiple secure (TLS-encrypted) connections and storing them in text files.
The syslog-ng application is not log analysis software. It can filter log messages and select only the ones matching certain criteria. It can even convert the messages and restructure them to a predefined format, or parse the messages and segment them into different fields. But syslog-ng cannot interpret and analyze the meaning behind the messages, or recognize patterns in the occurrence of different messages.
Log messages contain information about the events happening on the hosts. Monitoring system events is essential for security and system health monitoring reasons.
The original syslog protocol separates messages based on the priority of the message and the facility sending the message. These two parameters alone are often inadequate to consistently classify messages, as many applications might use the same facility — and the facility itself is not even included in the log message. To make things worse, many log messages contain unimportant information. The syslog-ng application helps you to select only the really interesting messages, and forward them to a central server.
Company policies or other regulations often require log messages to be archived. Storing the important messages in a central location greatly simplifies this process.
For details on how can you use syslog-ng PE to comply with various regulations, see the Regulatory compliance and system logging whitepaper.
© 2022 One Identity LLC. ALL RIGHTS RESERVED. Feedback Terms of Use Privacy
