This chapter introduces the syslog-ng Premium Edition application in a non-technical manner, discussing how and why is it useful, and the benefits it offers to an existing IT infrastructure.
The syslog-ng Premium Edition (syslog-ng PE) application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions. Among others, syslog-ng PE allows you the following.
Secure and reliable log transfer
The syslog-ng PE application enables you to send the log messages of your hosts to remote servers using the latest protocol standards. You can collect and store your log data centrally on dedicated log servers. Transfer log messages using the RLTP™ protocol ensures that no messages are lost.
Disk-based message buffering. To minimize the risk of losing important log messages, the syslog-ng PE application can store messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished, in the same order the messages were received. The disk buffer is persistent – no messages are lost even if syslog-ng is restarted.
Secure logging using TLS. Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng PE supports the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows you to authenticate your clients and the logserver using X.509 certificates.
Flexible data extraction and processing
Most log messages are inherently unstructured, which makes them difficult to process. To overcome this problem, syslog-ng PE comes with a set of built-in parsers, which you can combine to build very complex things.
Filter and classify. The syslog-ng PE application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. You can create directories, files, and database tables dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.
Parse and rewrite. The syslog-ng PE application can segment log messages to named fields or columns, and also modify the values of these fields. You can process JSON messages, key-value pairs, and more.
To get the most information out of your log data, syslog-ng PE allows you to correlate log messages and aggregate the extracted information into a single message. You can also use external information to enrich your log data.
Wide protocol and platform support
syslog protocol standards. syslog-ng not only supports legacy BSD syslog (RFC3164) and the enhanced RFC5424 protocols, but also JavaScript Object Notation (JSON) and journald message formats.
IPv4 and IPv6 support. The syslog-ng application can operate in both IPv4 and IPv6 network environments, and can receive and send messages to both types of networks.
When transferring messages to a remote server, the syslog-ng PE clients can be configured to send the log messages to secondary servers if the primary server becomes unaccessible.
Encrypted and timestamped log storage
The syslog-ng PE application can store log messages securely in encrypted, compressed, and timestamped binary files. Timestamps can be requested from an external Timestamping Authority (TSA).
Depending on the exact syslog-ng PE configuration, environment, and other parameters, syslog-ng PE is capable of processing:
-
Over 590,000 messages per second (over 220 MB of data per second) when receiving messages from multiple connections and storing them in text files.
-
Over 560,000 messages per second (over 210 MB of data per second) when receiving messages from multiple connections and storing them in logstore files (that is, encrypted files).
-
Over 565,000 messages per second (over 210 MB of data per second) when receiving messages from multiple secure (TLS-encrypted) connections and storing them in text files.
