This section lists the changes of The syslog-ng Premium Edition Administrator Guide.
Version
HTTPS connection, as well as password- and certificate-based authentication is supported. The content of the events is sent in JSON format.
For details, see "splunk-hec: Sending messages to Splunk HTTP Event Collector" in the Administration Guide.
The http() destination now supports load balancing, so a single syslog-ng PE instance can feed log data to multiple HTTP servers, for example, multiple ingestion nodes of an Elasticsearch cluster. For details, see "Batch mode and load balancing" in the Administration Guide.
HTTP and HTTPS redirections now also handled automatically.
The syslog() and network() drivers now support the so-reuseport() option that allows multiple sockets on the same host to bind to the same port, improving the performance of multithreaded network server applications running on top of multicore systems.
The Cisco parser now supports Cisco Catalyst formatted triplets.
Version 7.0.12 of syslog-ng PE is now available on the Ubuntu 18.04 platform. Note that the Java-based drivers of syslog-ng PE (used for Apache Kafka, Elasticsearch, HDFS) require Java 8, Java 10 is not supported.
The allow-compress() option of the ALTP communication has been renamed to allow-plain-compress().
Starting with syslog-ng PE version
You can now configure syslog-ng PE to reset the counter that stores the list of known hosts. That way, you can make syslog-ng PE forget old clients that do not exist anymore, and otherwise would be counted against the license limit. This is especially useful in large datacenters or cloud environments where the client hosts are deployed and removed frequently.
For details, see the "Global options" in the Administration Guide.
When hdfs-append-enabled is set to true, syslog-ng PE will append new data to the end of an already existing HDFS file. Note that in this case, archiving is automatically disabled, and syslog-ng PE will ignore the hdfs-archive-dir option.
New template functions are available: url-decode(), url-encode() and base64-encode(). For details, see "Template functions of syslog-ng PE" in the Administration Guide.
The syslog-ng-ctl config command can display the contents of the configuration file that syslog-ng PE is currently running.
Extending syslog-ng PE in Python has been supported for several releases, but so far this feature was mostly undocumented. Now you can find more details about this feature in "python: writing custom Python destinations" in the Administration Guide.
The failover() option allows you to specify what happens after syslog-ng PE fails over to a secondary server. Additionally, the failover-servers() option has been deprecated and removed from the document. For more information about the failover() option, see Client-side failover.
You can now refer to any additional parameters at the end of the argument in a block by adding three dots to it (…). It tells syslog-ng PE that this macro accepts `__VARARGS__`, therefore any name-value pair can be passed without validation. For details, see Passing arguments to configuration blocks.
You can now make parameters mandatory in block definitions by defining them with empty brackets (). For details, see Mandatory parameters.
A note about JVM still running after deleting all Java destinations and reloading syslog-ng has been added to the description of Java destinations.
The default value of the --skip-tokens parameter of the loggen application has been changed to 0. For details, see The loggen manual page.
Advanced Log Transfer Protocol has been added to the document.
default-network-drivers: Receive and parse common syslog messages has been added to the document.
if-else-elif: Conditional expressions has been added to the document.
Enterprise-wide message model (EWMM), format-ewmm, Parsing enterprise-wide message model (EWMM) messages, and syslog-ng(): Forward logs to another syslog-ng node have been added to the document.
The iptables parser and The sudo parser have been added to the document.
The RAWMSG macro has been added to Macros of syslog-ng PE.
The store-raw-message flag has been added to flags().
The ignore-tns-ora() option has been added to the document. For details, see Using the sql() driver with an Oracle database.
A new log path flag, drop-unmatched, has been added. The new flag causes messages to be dropped along a log path when they do not match a filter or are discarded by a parser. For details, see "Log path flags" in the Administration Guide.
Support for Elasticsearch's Shield has been removed.
Support for POSIX regular expressions has been removed.
All "posix" regular expressions are automatically switched to "pcre". In case you have POSIX regular expressions configured, ensure that they work with PCRE, and also specify type("pcre") explicitly.
The logstore() destination that was available only in syslog-ng PE version 6 is now available in version 7.0.7, allowing you to store messages in encrypted files. For more details, see "logstore: Storing messages in encrypted files" in the Administration Guide.
You can use password-protected private keys in the network() and syslog() source and destination drivers. For details, see "Password-protected keys" in the Administration Guide.
A new section describing common error messages has been added to the document. For more information, see "Error messages" in the Administration Guide.
Several corrections and editorial changes.
A new source called windowsevent() has been added. The windowsevent() source receives Windows event logs from the Windows Event Collector tool, which collects event logs from Windows hosts. For more information, see "windowsevent: Collecting Windows event logs" in the Administration Guide.
Added a section about how to unset message fields and groups of fields. For more information, see "Unsetting message fields" in the Administration Guide.
A new section describing common error messages has been added to the document. For more information, see "Error messages" in the Administration Guide.
Several corrections and editorial changes.
syslog-ng PE now supports Oracle Linux 6 (x84_64). For details, see "Supported platforms" in the Administration Guide.
A new systemd-journal() source option, called read-old-records(), has been added. For more information, see Administration Guide.
An option called jvm-options() has been added, which allows you to fine-tune Java Virtual Machine settings when configuring Elasticsearch, HDFS, and Apache Kafka destinations. For details, see:
A new HDFS destination option, called hdfs-append-enabled() has been added. For further information, see Administration Guide.
Macros are now supported in the hdfs-file() option. For details, see Administration Guide.
The following new TLS options have been added:
A new parser, capable of processing input in XML format, has been added. For more information, see "The XML parser" in the Administration Guide.
A new parser, capable of parsing the log messages of various Cisco devices, has been added. For details, see "The Cisco Parser" in the Administration Guide.
Added a section on upgrading from syslog-ng OSE to syslog-ng-PE. For more information, see "Upgrade from syslog-ng OSE to syslog-ng PE" in the Administration Guide.
Added warning about the requirement to delete the persist file once the dir() option of disk-buffer() has been modified or a new one has been added. For more information, see "Sending and storing log messages — destinations and destination drivers" in the Administration Guide.
Clarified information about the Python parser's deinit() method. It runs not only at a syslog-ng graceful stop, but at a reload too. For details, see Administration Guide.
Reworked Optimizing multithreaded performance to make information more accessible.
Several corrections and editorial changes.
"Looking up GeoIP2 data from IP addresses" in the Administration Guide has been added to the document.
"http: Posting messages over HTTP without Java" in the Administration Guide has been added to the document.
"osquery: Collect and parse osquery result logs" in the Administration Guide has been added to the document.
RHEL6 has been added to the supported platforms in "Supported platforms" in the Administration Guide.
The geoip() parser has been removed from the document.
Several corrections and editorial changes.
"wildcard-file: Collecting messages from multiple text files" in the Administration Guide has been added to the document.
"snmptrap: Read Net-SNMP traps" in the Administration Guide has been added to the document.
New counters and metrics have been added to "Metrics and counters of syslog-ng PE" in the Administration Guide.
The default value of the log-msg-size() option has been increased to 64k. That way syslog-ng PE will not truncate long log messages, which are getting increasingly common.
The hdfs() destination now supports Kerberos authentication. For details, see "Kerberos authentication with syslog-ng hdfs() destination" in the Administration Guide.
The new basename() and dirname() template functions allow you to easily separate the path and filenames. For details, see "Template functions of syslog-ng PE" in the Administration Guide.
"About disk queue files" in the Administration Guide has been added to the document.
An example failure script has been added to "Running a failure script" in the Administration Guide.
Several corrections and editorial changes.
"The Python Parser" in the Administration Guide has been added to the document.
Administration Guide has been added to the document.
The new monitoring() and monitoring-welf() sources have been added to "The monitoring() source" in the Administration Guide.
The syslog-ng-query application is available in syslog-ng PE 7.0.2, as part of the syslog-ng-ctl utility. For details, see Administration Guide.
Several corrections and editorial changes.
Several features that are available in syslog-ng Premium Edition 6 LTS are not yet available in syslog-ng PE 7. For a list of features that are available in syslog-ng PE 6 LTS but not in 7 see "Features available only in syslog-ng PE 6 LTS" in the Release Notes.
Supported platforms has been updated.
Enriching log messages with external data has been added to the document.
Correlating log messages has been added to the document.
What's new in the syslog-ng pattern database format V5, Element: create-context, has been added to Processing message content with a pattern database.
dbd-option() has been added to sql: Storing messages in an SQL database.
Defining configuration objects inline has been added to The syslog-ng PE configuration file.
Using channels in configuration objects has been added to The syslog-ng PE configuration file.
Anonymizing credit card numbers has been added to Manipulating messages.
Parsing syslog messages has been added to parser: Parse and segment structured messages.
Parsing dates and timestamps has been added to parser: Parse and segment structured messages.
A template function that formats name-value pairs as ArcSight Common Event Format extension has been added to format-cef-extension.
Numerical template functions that work on numerical values of a correlation context have been added to Numerical operations.
The inherit-environment() option has been added to program: Receiving messages from external applications and program: Sending messages to external applications.
The multi-line-mode() option has been added to file() source options.
New parsers have been added to Using pattern parsers.
You can use human-readable units when setting configuration options, for example, log-fifo-size(2Mb). For details, see Notes about the configuration syntax.
Extended the list of internal() source options with options host-override(), log-iw-size(), normalize-hostnames(), program-override(), and use-fqdn(). For details, see host-override(), log-iw-size(), normalize-hostnames(), program-override(), and use-fqdn().
Several corrections and editorial changes.
One Identity would like to express its gratitude to the syslog-ng users and the syslog-ng community for their invaluable help and support.
This chapter introduces the syslog-ng Premium Edition application in a non-technical manner, discussing how and why is it useful, and the benefits it offers to an existing IT infrastructure.
The syslog-ng Premium Edition (syslog-ng PE) application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions. Among others, syslog-ng PE allows you the following.
The syslog-ng PE application enables you to send the log messages of your hosts to remote servers using the latest protocol standards. You can collect and store your log data centrally on dedicated log servers. Transfer log messages using the
To minimize the risk of losing important log messages, the syslog-ng PE application can store messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished, in the same order the messages were received. The disk buffer is persistent – no messages are lost even if syslog-ng is restarted.
Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng PE supports the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows you to authenticate your clients and the logserver using X.509 certificates.
Most log messages are inherently unstructured, which makes them difficult to process. To overcome this problem, syslog-ng PE comes with a set of built-in parsers, which you can combine to build very complex things.
The syslog-ng PE application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. You can create directories, files, and database tables dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.
The syslog-ng PE application can segment log messages to named fields or columns, and also modify the values of these fields. You can process JSON messages, key-value pairs, and more.
To get the most information out of your log data, syslog-ng PE allows you to correlate log messages and aggregate the extracted information into a single message. You can also use external information to enrich your log data.
The log data that your organization has to process, store, and review increases daily, so many organizations use big data solutions for their logs. To accomodate this huge amount of data, syslog-ng PE natively supports storing log messages in HDFS files and Elasticsearch clusters.
Large organizations increasingly rely on queuing infrastructure to transfer their data. For that purpose, syslog-ng PE supports Apache Kafka.
Storing your log messages in a database allows you to easily search and query the messages and interoperate with log analyzing applications. The syslog-ng application supports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, and SQLite.
syslog-ng not only supports legacy BSD syslog (RFC3164) and the enhanced RFC5424 protocols, but also JavaScript Object Notation (JSON) and journald message formats.
The syslog-ng PE application is the ideal choice to collect logs in massively heterogeneous environments using several different operating systems and hardware platforms, including Linux, Unix, BSD, Sun Solaris, HP-UX, Tru64, and AIX.
The syslog-ng application can operate in both IPv4 and IPv6 network environments, and can receive and send messages to both types of networks.
The syslog-ng PE application can store log messages securely in encrypted, compressed, and timestamped binary files. Timestamps can be requested from an external Timestamping Authority (TSA).
Depending on the exact syslog-ng PE configuration, environment, and other parameters, syslog-ng PE is capable of processing:
Over 635,000 messages per second (over 235 MB of data per second) when receiving messages from multiple connections and storing them in text files.
Over 615,000 messages per second (over 230 MB of data per second) when receiving messages from multiple secure (TLS-encrypted) connections and storing them in text files.
© 2020 One Identity LLC. ALL RIGHTS RESERVED. Feedback Terms of Use Privacy
