This section lists the changes of The syslog-ng Open Source Edition Administrator Guide.
Version 3.25 of syslog-ng Open Source Edition includes the following main features.
You can use the syslog-ng-cfg-db.py utility to list the available options of configuration objects. For example, you can list all the options that can be set in the file source, and so on. For details, see "Listing configuration options" in the Administration Guide.
Starting with syslog-ng OSE
The default-network-drivers() source now supports the max-connections() option.
Starting with
log { source { python(class(PyNetworkSource) options("port" "8080") persist-name("<unique-string>); }; source { python(class(PyNetworkSource) options("port" "8081")); }; };
Alternatively, you can include the following line in the Python package: @staticmethod generate_persist_name. For example:
from syslogng import LogSource class PyNetworSource(LogSource): @staticmethod def generate_persist_name(options): return options["port"] def run(self): pass def request_exit(self): pass
The type() and flags() options of regular expressions were separated in the documentation. For further details, see Options of regular expressions.
Version 3.24 of syslog-ng Open Source Edition includes the following main features.
The format-flat-json template function is identical to the format-json template function, but nested JSON objects are flattened in the output. For details, see "Template functions of syslog-ng OSE" in the Administration Guide.
You can set, fix, or guess the timezone of messages using timezone-specific rewrite rules. For details, see "Rewrite the timezone of a message" in the Administration Guide.
You can now use shell-style globbing (wildcards) in the selector of add-contextual-data() databases. For details, see "Shell-style globbing in the selector" in the Administration Guide.
You can now send Server Name Identification (SNI) information when using transport(tls) by enabling the sni(yes) option. For details, see "TLS options" in the Administration Guide.
You can use ISO 8601 compliant week numbering using the ${ISOWEEK} macro and its variants: S_ISOWEEK, R_ISOWEEK and C_ISOWEEK. For details, see "Macros of syslog-ng OSE" in the Administration Guide.
The ${LOGHOST} macro now honours the use-fqdn() option.
You can now reference the location of your configuration file in the configuration file using the `syslog-ng-sysconfdir` variable.
You can now assign disk-queue files to entries in the persist file. For details, see dqtool --help.
Version 3.23 of syslog-ng Open Source Edition includes the following main features.
In syslog-ng OSE version 3.23 and later, you can specify a comma-separated list of formats to parse multiple date formats with a single parser. For example:
date-parser(format( "%FT%T.%f", "%F %T,%f", "%F %T" ));
For details, see "Options of date-parser() parsers" in the Administration Guide.
You can now display the preprocessed configuration of syslog-ng OSE on the standard output using the syslog-ng --preprocess-into=/dev/stdout command.
You can now move or rename disk-buffer files using the dqtool relocate command.
You can now set the time-reap() option separately for each destination. Note that how this option works has also changed. For details, see "Global options" in the Administration Guide.
Version 3.22 of syslog-ng Open Source Edition includes the following main features.
Starting with version
As a result of these changes the log-fifo-size() option only affects log paths that are not flow-controlled. It is expected that after configuring the dynamic message window, you can decrease the value of log-fifo-size(). For details, see .
You can now send SNMP traps directly from syslog-ng OSE using the snmp() destination driver. For details, see .
A new template function called template can resolve static and dynamic templates in template functions. For example, the name of the template to be invoked can be extracted from the message, or from a name-value pair set using the add-contextual-data() feature. For details, see .
Numerical template functions can now handle floating-point numbers. For details, see the ceil, floor, numerical operations, and round template functions.
HTTP-based destinations can now accept multiple URLs in various formats.
The message rate of the loggen command can be changed while loggen is running. Send SIGUSR1 to double the message rate, or SIGUSR2 to halve it, for example: kill -USR1 <loggen-pid>
The Check Point Log Exporter parser can now parse Check Point log messages in the Splunk format. For details, see .
New constants are available in the fetch method of the Python source. For details, see .
Global option can be defined in reusable blocks. For details, see .
The date-parser() now supports microseconds (%f). For details, see .
The value of add-contextual-data() selectors can be a template or a template function, not only a string. For details, see .
New template functions are available: $(explode) and $(implode). For more information, see "Template functions of syslog-ng OSE" in the Administration Guide.
You can use a relay for many different use cases. For more information, see Example relay use cases.
The Websense parser can parse the log messages of Websense Content Gateway (Raytheon|Websense, now Forcepoint). These messages do not completely comply with the syslog RFCs, making them difficult to parse. The websense-parser() of syslog-ng OSE solves this problem, and can separate these log messages to name-value pairs. For details, see "Websense parser" in the Administration Guide.
The Netskope parser can parse Netskope log messages. These messages do not completely comply with the syslog RFCs, making them difficult to parse. The netskope-parser() of syslog-ng OSE solves this problem, and can separate these log messages to name-value pairs. For details, see "Netskope parser" in the Administration Guide.
The persist-tool utility is now part of the syslog-ng OSE package. For details, see the persist-tool manual page.
Since ElasticSearch version 1.x has reached its end of life, its support has been removed from syslog-ng OSE. Use the elasticsearch2 destination instead.
The http() destination now supports load balancing, so a single syslog-ng OSE instance can feed log data to multiple HTTP servers, for example, multiple ingestion nodes of an Elasticsearch cluster. For details, see "Batch mode and load balancing" in the Administration Guide.
HTTP and HTTPS redirections now also handled automatically.
The use-system-cert-store() allows you to use the certificate store of the system for verifying HTTPS certificates. For details, see the curl documentation.
The slack() destination driver sends messages to a Slack channel using the Slack Web API. For the list of available optional parameters, see Slack destination options. This destination is available in version
The syslog() and network() drivers now support the so-reuseport() option that allows multiple sockets on the same host to bind to the same port, improving the performance of multithreaded network server applications running on top of multicore systems.
The allow-compress() option is now available for TLS connections.
The loaders() option is available for python destinations.
The exclude-kmsg() option of the internal() and linux-audit() source is not supported anymore.
The Cisco parser now supports Cisco Catalyst formatted triplets.
The flush-bytes(), flush-lines(), and flush-timeout() options have been renamed to batch-bytes(), batch-lines(), and batch-timeout().
Starting with syslog-ng OSE version
Starting with syslog-ng OSE version
The http() destination can now send a batch of log messages in a single HTTP request, greatly improving the performance. In addition, this feature also allows you to post proper JSON-encoded arrays as POST payloads, which is required by several REST APIs. For details, see "http: Posting messages over HTTP without Java" in the Administration Guide.
When hdfs-append-enabled is set to true, syslog-ng OSE will append new data to the end of an already existing HDFS file. Note that in this case, archiving is automatically disabled, and syslog-ng OSE will ignore the hdfs-archive-dir option.
The hdfs destination now supports the time-reap() option. For details, see "HDFS destination options" in the Administration Guide.
New template functions are available: url-decode() and base64-encode(). For details, see "Template functions of syslog-ng OSE" in the Administration Guide.
The syslog-ng-ctl config command can display the contents of the configuration file that syslog-ng OSE is currently running.
The rekey option of value-pairs() now supports a new transformation: shift-levels. It cuts dot-delimited "levels" in the name (including the initial dot). For example, --shift-levels 2 deletes the prefix up to the second dot in the name of the key: .iptables.SRC becomes SRC
For details, see "value-pairs()" in the Administration Guide.
The value-pairs() option now has a new scope: none. This scope resets previously added scopes, making it possible to get remove automatically added name-value pairs from the scope.
For details, see "value-pairs()" in the Administration Guide.
The max-channel and frame-size options have been added to the amqp() destination.
Extending syslog-ng OSE in Python has been supported for several releases, but so far this feature was mostly undocumented. Now you can find more details about this feature in "python: writing custom Python destinations" in the Administration Guide.
A new source driver, linux-audit(), has been added. The linux-audit() source reads and automatically parses the Linux audit logs. For details, see linux-audit: Collecting messages from Linux audit logs.
A new system source option, exclude-kmsg() makes it possible to avoid duplicate collection of kernel logs or errors in kernel log collection (for example, in scenarios where the log management on the host system and the containerized solution are collecting the kernel logs simultaneously). When set to yes, syslog-ng OSE will omit kernel logs on platforms where they are available separately.
You can now refer to any additional parameters at the end of the argument in a block by adding three dots to it (…). It tells syslog-ng OSE that this macro accepts `__VARARGS__`, therefore any name-value pair can be passed without validation. For details, see Passing arguments to configuration blocks.
You can now make parameters mandatory in block definitions by defining them with empty brackets (). For details, see Mandatory parameters.
The failover() option allows you to specify what happens after syslog-ng OSE fails over to a secondary server. Additionally, the failover-servers() option has been deprecated and removed from the document. For more information about the failover() option, see Client-side failover.
Added support for the timestamp format used by Cisco Unified Call Manager in the Cisco parser. For details, see the source code of this parser on GitHub.
A note about JVM still running after deleting all Java destinations and reloading syslog-ng has been added to the description of Java destinations.
The default value of the --skip-tokens parameter of the loggen application has been changed to 0. For details, see The loggen manual page.
A new destination driver, telegram(), has been added. The telegram() destination sends log messages to Telegram, which is a secure, cloud-based mobile and desktop messaging app. For more information, see Telegram: Sending messages to Telegram.
A new template function, urlencode, has been added. You can use the urlencode template function together with the telegram() destination to send syslog messages to Telegram. For more information, see Template functions of syslog-ng OSE.
It is now possible to use if {}, elif {}, and else {} blocks to configure conditional expressions. For details, see if-else-elif: Conditional expressions.
A new log path flag, drop-unmatched, has been added. The new flag causes messages to be dropped along a log path when they do not match a filter or are discarded by a parser. For details, see Log path flags.
Support for Elasticsearch's Shield has been removed.
Support for POSIX regular expressions has been removed.
You can use password-protected private keys in the network() and syslog() source and destination drivers. For details, see Password-protected keys.
To better control to which log messages you add contextual data, you can use filters as selectors. In this case, the first column of the CSV database file must contain the name of a filter. For each message, syslog-ng OSE evaluates the filters in the order they appear in the database file. If a filter matches the message, syslog-ng OSE adds the name-value pair related to the filter. For details, see Using filters as selector.
A new source driver, stdin(), has been added. The stdin() driver collects messages from the standard input stream. For more information, see stdin: Collecting messages from the standard input stream.
A new destination, Sending logs to Graylog, and a template to send syslog messages to Graylog, format-gelf, has been added.
A new template function, getent, has been added. You can use the getent template function to look up entries from the Name Service Switch libraries. For more information, see getent.
The default values of the --enable-json, --enable-mongodb, and --with-libmongo-client compile parameters have changed. For more information, see Compiling options of syslog-ng OSE.
A new compile option, --with-module-path, has been added. The new option specifies syslog-ng OSE's module installation directory. For more information, see Compiling options of syslog-ng OSE.
A new destination driver, osquery(), has been added. The new driver sends log messages to osquery's syslog table. For more information, see osquery: Sending log messages to osquery's syslog table.
It is now possible to specify TLS options in a tls() block. For more information, see:
Support for microseconds in Riemann destinations has been introduced. For more information, see event-time().
Module auto-loading now also works for the system() source. For more information, see --default-modules.
A new section describing common error messages has been added to the document. For more information, see Error messages .
Several corrections and editorial changes.
A new systemd-journal() source option, called read-old-records(), has been added. For more information, see read-old-records().
An option called jvm-options() has been added, which allows you to fine-tune Java Virtual Machine settings when configuring Elasticsearch, HDFS, and Apache Kafka destinations, or web services to which you send log messages via the HTTP protocol. For details, see:
A new HDFS destination option, called hdfs-append-enabled() has been added. For further information, see hdfs-append-enabled().
Macros are now supported in the hdfs-file() option. For details, see hdfs-file().
The following new TLS options have been added:
A new parser, capable of processing input in XML format, has been added. For more information, see XML parser.
Added section about commercial version of syslog-ng. For more information, see Commercial version of syslog-ng.
Added warning about the requirement to delete the persist file once the dir() option of disk-buffer() has been modified or a new one has been added. For more information, see destination: Forward, send, and store log messages.
Clarified information about the Python parser's deinit() method. It runs not only at a syslog-ng graceful stop, but at a reload too. For details, see Methods of the python() parser.
Several corrections and editorial changes.
Looking up GeoIP2 data from IP addresses has been added to the document.
http: Posting messages over HTTP without Java has been upgraded with new improvements.
The geoip() parser is now deprecated. Looking up GeoIP data from IP addresses (DEPRECATED).
The template() option has been added to the Apache access log parser. For details, see: Apache access log parser.
SSL-related options have been added to amqp() destination. For details, see: amqp() destination options.
The prefix() option has been added to the Cisco parser. For details, see: Cisco parser.
The drop-unmatched() option has been added to the db-parser() statement. For details, see: Using pattern databases.
The event-time() option has been added to the Riemann destination. For details, see: riemann: Monitoring your data with Riemann.
A new example has been added to the osquery() source. For details, see: osquery: Collect and parse osquery result logs.
Several corrections and editorial changes.
wildcard-file: Collecting messages from multiple text files has been added to the document.
snmptrap: Read Net-SNMP traps has been added to the document.
osquery: Collect and parse osquery result logs has been added to the document.
The elasticsearch2() destination now supports HTTPS mode, including encryption, and also password- and certificate-based authentication. For details, see elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED).
The http() destination now supports encryption, and also password- and certificate-based authentication. For details, see HTTP destination options.
The hdfs() destination now supports Kerberos authentication. For details, see Kerberos authentication with syslog-ng hdfs() destination.
Python parser has been added to the document.
Cisco parser has been added to the document.
map-value-pairs: Rename value-pairs to normalize logs has been added to the document.
The list-* template functions allow you to manipulate comma-separated lists. For details, see List manipulation.
The new basename() and dirname() template functions allow you to easily separate the path and filenames. For details, see Template functions of syslog-ng OSE.
stardate has been added to the document.
create-statement-append() has been added to the document.
The default value of the log-msg-size() option has been increased to 64k. That way syslog-ng OSE will not truncate long log messages, which are getting increasingly common.
Splunk: Sending log messages to Splunk has been added to the document.
About disk queue files has been added to the document.
An example failure script has been added to Running a failure script.
Several corrections and editorial changes.
When using TLS-transport, you can now use certain fields of the X.509 certificates as macros. For details, see .TLS.X509.
The elastic2() destination driver now supports Search Guard, an alternative security solution for Elasticsearch. For details, see Search Guard and syslog-ng OSE.
.TLS.X509 has been added to the document.
Unsetting message fields has been updated with groupunset().
Corrections and editorial changes.
Enriching log messages with external data has been added to the document.
Correlating log messages has been added to the document.
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) has been added to the document.
http: Posting messages over HTTP without Java has been added to the document.
logmatic: Using Logmatic.io has been added to the document.
loggly: Using Loggly has been added to the document.
Disk-based buffering has been added to syslog-ng OSE. For details, see Using disk-based and memory buffering.
What's new in the syslog-ng pattern database format V5, , has been added to Element: create-context has been added to db-parser: Process message content with a pattern database (patterndb).
Parsing dates and timestamps has been added to parser: Parse and segment structured messages.
Apache access log parser has been added to parser: Parse and segment structured messages.
New options of the set() rewrite operator have been added to Setting message fields to specific values.
A rewrite operator to unset fields has been added to Unsetting message fields.
A template function that formats name-value pairs as ArcSight Common Event Format extension has been added to format-cef-extension.
Numerical template functions that work on numerical values of a correlation context have been added to Numerical operations.
The inherit-environment() option has been added to program: Receiving messages from external applications and program: Sending messages to external applications.
@NLSTRING@ has been added to Using pattern parsers.
Looking up GeoIP data from IP addresses (DEPRECATED) has been moved to Enriching log messages with external data.
Several corrections and editorial changes.
mbox: Converting local email messages to log messages has been added to the document.
The keep-alive() option has been added to the program() destination.
Linux audit parser has been added to parser: Parse and segment structured messages.
python has been added to Template functions of syslog-ng OSE.
Posting messages over HTTP has been added to the document.
Write your own custom destination in Java or Python has been added to the document.
Looking up GeoIP data from IP addresses (DEPRECATED) has been added to the document.
kafka: Publishing messages to Apache Kafka (Java implementation) has been added to the document.
hdfs: Storing messages on the Hadoop Distributed File System (HDFS) has been added to the document.
Parsing key=value pairs has been added to the document.
format-cim has been added to the document.
Simple templates can be defined without braces. Templates can also reference other templates. For details, see Templates and macros.
Custom template functions can be defined in the syslog-ng OSE configuration. For details, see Using template functions.
CSV-parsers can use strings as delimiters. For details, see delimiters().
IPv6 addresses can be filtered using a new filter. For details, see netmask6().
The loggen utility can send messages indefinitely using the --permanent option.
The ssl-options() option has beed added to TLS options.
TLS-support has been added to riemann() destination options.
The extract-solaris-msgid() parser has beed added to sun-streams: Collecting messages on Sun Solaris.
The context option of inherit-properties has beed added to Actions and message correlation.
riemann() destination options has been added to the document.
The sanitize-utf8 flag has been added to the list of source flags.
The format-welf function has been added to Template functions of syslog-ng OSE.
The pass-unix-credentials() option has been added to Global options of syslog-ng OSE.
The use-uniqid() option has been added to Global options of syslog-ng OSE.
The UNIQID macro has been added to Macros of syslog-ng OSE.
The JSON-parser now handles special characters in object names. For details, see extract-prefix().
The syslog-debun tool used to generate syslog-ng OSE debug bundles has been documented. For details, see The syslog-ng-debun manual page.
The --control option has been added to the The syslog-ng manual page manual page.
Version
The --enable-all-modules compiler option has beed added to Compiling options of syslog-ng OSE.
The create-dirs() option has been added to unix-stream() and unix-dgram() destination options.
Generating configuration blocks from a script has been added to the document.
Example: Sending alert when a client disappears has been added to the document.
The tcp(), tcp6(), udp(), udp6() source and destination drivers have been deprecated, as all of their functionality can be achieved with the network() driver. For help on migrating to the network() driver, see Change an old source driver to the network() driver and Change an old destination driver to the network() driver.
The beginning of Troubleshooting syslog-ng has been extended with basic troubleshooting information.
The description of the chain-hostnames() global option has been clarified and extended. For details, see chain-hostnames().
Other editorial corrections.
riemann: Monitoring your data with Riemann has been added to the document.
nodejs: Receiving JSON messages from nodejs applications has been added to the document.
systemd-journal: Collecting messages from the systemd-journal system log storage has been added to the document.
systemd-syslog: Collecting systemd messages using a socket has been added to the document.
use-rcptid() has been added to the document.
Setting multiple message fields to specific values has been added to the document.
The retries and throttle options are available for the SMTP, MongoDB, AMQP, and Redis destinations.
The description of the multi-line-mode option has been updated.
UNIX credentials and other metadata has been added to the document.
RUNID has been added to Macros of syslog-ng OSE.
The extract-prefix option has been added to JSON parser.
The graphite-output, or and padding template functions have been added to Template functions of syslog-ng OSE.
PCRE is now a required dependency of syslog-ng OSE, and by default, syslog-ng OSE uses PCRE-style regular expressions. Therefore, the --enable-pcre compliation option has been removed.
graphite: Sending metrics to Graphite has been added to the document.
pseudofile() has been added to the document.
The custom-domain() and stats-lifetime() options have been added to Global options.
The retry_sql_inserts option has been renamed to retries to increase consistency.
on-error() can be set locally for MongoDB destinations as well. Also, MongoDB destinations support the username and password options, and connecting to the server using UNIX domain sockets. For details, see mongodb: Storing messages in a MongoDB database.
How syslog-ng OSE connects the MongoDB server has been added to the document.
Several typos and syntax errors in examples have been corrected.
One Identity would like to express its gratitude to the syslog-ng users and the syslog-ng community for their invaluable help and support.
This chapter introduces the syslog-ng Open Source Edition application in a non-technical manner, discussing how and why is it useful, and the benefits it offers to an existing IT infrastructure.
The syslog-ng Open Source Edition (syslog-ng OSE) application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions. Among others, syslog-ng OSE allows you the following.
The syslog-ng OSE application enables you to send the log messages of your hosts to remote servers using the latest protocol standards. You can collect and store your log data centrally on dedicated log servers. Transfer log messages using the
To minimize the risk of losing important log messages, the syslog-ng OSE application can store messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished, in the same order the messages were received. The disk buffer is persistent – no messages are lost even if syslog-ng is restarted.
Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng OSE supports the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows you to authenticate your clients and the logserver using X.509 certificates.
Most log messages are inherently unstructured, which makes them difficult to process. To overcome this problem, syslog-ng OSE comes with a set of built-in parsers, which you can combine to build very complex things.
The syslog-ng OSE application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. You can create directories, files, and database tables dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.
The syslog-ng OSE application can segment log messages to named fields or columns, and also modify the values of these fields. You can process JSON messages, key-value pairs, and more.
To get the most information out of your log data, syslog-ng OSE allows you to correlate log messages and aggregate the extracted information into a single message. You can also use external information to enrich your log data.
The log data that your organization has to process, store, and review increases daily, so many organizations use big data solutions for their logs. To accomodate this huge amount of data, syslog-ng OSE natively supports storing log messages in HDFS files and Elasticsearch clusters.
Large organizations increasingly rely on queuing infrastructure to transfer their data. For that purpose, syslog-ng OSE supports Apache Kafka
Storing your log messages in a database allows you to easily search and query the messages and interoperate with log analyzing applications. The syslog-ng application supports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, and SQLite.
syslog-ng OSE also allows you to extract the information you need from your log data, and directly send it to your Graphite, Redis, or Riemann monitoring system.
syslog-ng not only supports legacy BSD syslog (RFC3164) and the enhanced RFC5424 protocols but also JavaScript Object Notation (JSON) and journald message formats.
The syslog-ng OSE application is the ideal choice to collect logs in massively heterogeneous environments using several different operating systems and hardware platforms, including Linux, Unix, BSD, Sun Solaris, HP-UX, and AIX.
The syslog-ng application can operate in both IPv4 and IPv6 network environments, and can receive and send messages to both types of networks.
© 2024 One Identity LLC. ALL RIGHTS RESERVED. Feedback Termini di utilizzo Privacy Cookie Preference Center