Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.9 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files network: Collecting messages using the RFC3164 protocol (network() driver) osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch: Sending messages directly to Elasticsearch version 1.x elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher file: Storing messages in plain-text files hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications smtp: Generating SMTP messages (e-mail) from logs Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng: Forwarding messages and tags to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Reliable Log Transfer Protocol Manipulating messages Parsers and segmenting structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages About us

Specifying data types in value-pairs

By default, syslog-ng PE handles every data as strings. However, certain destinations and data formats (for example, SQL, MongoDB, JSON) support other types of data as well, for example, numbers or dates. The syslog-ng PE application allows you to specify the data type in templates (this is also called type-hinting). If the destination driver supports data types, it converts the incoming data to the specified data type. For example, this allows you to store integer numbers as numbers in MongoDB, instead of strings.

Caution:

Hazard of data loss! If syslog-ng PE cannot convert the data into the specified type, an error occurs, and syslog-ng PE drops the message by default. To change how syslog-ng PE handles data-conversion errors, see on-error().

To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>"), for example: int("$PID").

Currently the mongodb() destination and the format-json template function supports data types.

Example: Using type-hinting

The following example stores the MESSAGE, PID, DATE, and PROGRAM fields of a log message in a MongoDB database. The DATE and PID parts are stored as numbers instead of strings.

mongodb(
    value-pairs(pair("date", datetime("$UNIXTIME"))
            pair("pid", int64("$PID"))
            pair("program", "$PROGRAM"))
            pair("message", "$MESSAGE"))
             )
);

The following example formats the same fields into JSON.

$(format-json date=datetime("$UNIXTIME") pid=int64("$PID") program="$PROGRAM" message="$MESSAGE")

The syslog-ng PE application currently supports the following data-types.

  • boolean: Converts the data to a boolean value. Anything that begins with a t or 1 is converted to true, anything that begins with an f or 0 is converted to false.

  • datetime: Use it only with UNIX timestamps, anything else will likely result in an error. This means that currently you can use only the $UNIXTIME macro for this purpose.

  • double: A floating-point number.

  • literal: The data as a literal string, without adding any quotes or escape characters.

  • int or int32: 32-bit integer.

  • int64: 64-bit integer.

  • string: The data as a string.

value-pairs()

Type: parameter list of the value-pairs() option
Default:
empty string

Description: The value-pairs() option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a mongodb() destination.

Example: Using the value-pairs() option

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message.

value-pairs(
    scope(nv_pairs core syslog all_macros selected_macros everything)
    exclude("R_*")
    exclude("S_*")
    key(".SDATA.meta.sequenceId")
    pair("MSGHDR" "$PROGRAM[$PID]: ")
)

The following example selects the same information as the previous example, but converts it into JSON format.

$(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \
  --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
  --pair MSGHDR="$PROGRAM[$PID]: ")

NOTE:

Every macro is included in the selection only once, but redundant information may appear if multiple macros include the same information (for example, including several date-related macros in the selection).

The value-pairs() option has the following parameters. The parameters are evaluated in the following order:

  1. scope()

  2. exclude()

  3. key()

  4. pair()

exclude()

 

Type: Space-separated list of macros to remove from the selection created using the scope() option.
Default: empty string

Description: This option removes the specified macros from the selection. Use it to remove unneeded macros selected using the scope() parameter.

For example, the following example removes the SDATA macros from the selection.

value-pairs(
   scope(rfc5424 selected_macros)
   exclude(".SDATA*")
)

The name of the macro to remove can include wildcards (*, ?). Regular expressions are not supported.

key()

 

Type: Space-separated list of macros to be included in selection
Default: empty string

Description: This option selects the specified macros. The selected macros will be included as MACRONAME = MACROVALUE, that is using key("HOST") will result in HOST = $HOST. You can use wildcards (*, ?) to select multiple macros. For example:

value-pairs(
        scope(rfc3164)
        key("HOST"))
value-pairs(
        scope(rfc3164)
        key("HOST", "PROGRAM"))

 

pair()

 

Type: name value pairs in "<NAME>" "<VALUE>" format
Default: empty string

Description: This option defines a new name-value pair to be included in the message. The value part can include macros, templates, and template functions as well. For example:

 

The following sample selects every value-pair that begins with .cee., deletes this prefix by cutting 4 characters from the names, and adds a new prefix (events.).

value-pairs(
  key(".cee.*"
    rekey(
      shift(4)
      add-prefix("events.")
	)
    )
)

The rekey() option can be used with the format-json template-function as well, using the following syntax:

$(format-json --rekey .cee.* --add-prefix events.)

 

rekey()

 

Type: <pattern-to-select-names>, <list of transformations>
Default: empty string

Description: This option allows you to manipulate and modify the name of the value-pairs. You can define transformations, which are are applied to the selected name-value pairs. The first parameter of the rekey() option is a glob pattern that selects the name-value pairs to modify. If you omit the pattern, the transformations are applied to every key of the scope. For details on globs, see glob.

  • If rekey() is used within a key() option, the name-value pairs specified in the glob of the key() option are transformed.

  • If rekey() is used outside the key() option, every name-value pair of the scope() is transformed.

The following transformations are available:

  • add-prefix("<my-prefix>")
  • Adds the specified prefix to every name. For example, rekey( add-prefix("my-prefix."))

  • replace-prefix("<prefix-to-replace>", "<new-prefix>")
  • Replaces a substring at the beginning of the key with another string. Only prefixes can be replaced. For example, replace-prefix(".class", ",patterndb") changes the beginning tag .class to .patterndb

  • shift("<number>")
  • Cuts the specified number of characters from the beginning of the name.

Example:
Table 6: Using the rekey() option
scope()

Type: space-separated list of macro groups to include in selection
Default: empty string

Description: This option selects predefined groups of macros. The following groups are available:

  • nv-pairs: Every soft macro (name-value pair) associated with the message, except the ones that start with a dot (.) character. Macros starting with a dot character are generated within syslog-ng PE and are not originally part of the message, therefore are not included in this group.

  • dot-nv-pairs: Every soft macro (name-value pair) associated with the message which starts with a dot (.) character. For example, .classifier.rule_id and .sdata.*. Macros starting with a dot character are generated within syslog-ng PE and are not originally part of the message.

  • all-nv-pairs: Include every soft macro (name-value pair). Equivalent to using both nv-pairs and dot-nv-pairs.

  • rfc3164: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, and $DATE.

  • rfc5424: The macros that correspond to the RFC5424 (IETF-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, $MSGID, $R_DATE, and the metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA..

    The rfc5424 group also has the following alias: syslog-proto. Note that the value of $R_DATE will be listed under the DATE key.

    The rfc5424 group does not contain any metadata about the message, only information that was present in the original message. To include the most commonly used metadata (for example, the $SOURCEIP macro), use the selected-macros group instead.

  • all-macros: Include every hard macro. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

  • selected-macros: Include the macros of the rfc3164 groups, and the most commonly used metadata about the log message: the $TAGS, $SOURCEIP, and $SEQNUM macros.

  • sdata: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA.

  • everything: Include every hard and soft macros. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

For example:

value-pairs(
        scope(rfc3164 selected-macros))

Things to consider when forwarding messages between syslog-ng PE hosts

When you send your log messages from a syslog-ng PE client through the network to a syslog-ng PE server, you can use different protocols and options. Every combination has its advantages and disadvantages. The most important thing is to use matching protocols and options, so the server handles the incoming log messages properly.

In syslog-ng PE you can change many aspects of the network communication. First of all, there is the structure of the messages itself. Currently, syslog-ng PE supports two standard syslog protocols: the BSD (RFC3164) and the syslog (RFC5424) message format.

These RFCs describe the format and the structure of the log message, and add a (lightweight) framing around the messages. You can set this framing/structure by selecting the appropriate driver in syslog-ng PE. There are two drivers you can use: the network() driver and the syslog() driver. The syslog() driver is for the syslog (RFC5424) protocol and the network() driver is for the BSD (RFC3164) protocol.

The tcp() and udp() drivers are now deprecated, they are essentially equivalent with the network(transport(tcp)) and network(transport(udp)) drivers.

In addition to selecting the driver to use, both drivers allow you to use different transport-layer protocols: TCP and UDP, and optionally also higher-level transport protocols: TLS (over TCP, and ALTP (optionally using TLS). To complicate things a bit more, you can configure the network() driver (corresponding to the BSD (RFC3164) protocol) to send the messages in the syslog (RFC5424) format (but without the framing used in RFC5424) using the flag(syslog-protocol) option.

Because some combination of drivers and options are invalid, you can use the following drivers and options as sources and as destinations:

  1. syslog(transport(tcp))

  2. syslog(transport(udp))

  3. syslog(transport(altp))

  4. syslog(transport(tls))

  5. syslog(transport(altp(tls-required(yes)))

  6. network(transport(tcp))

  7. network(transport(udp))

  8. network(transport(altp))

  9. network(transport(tls))

  10. network(transport(altp(tls-required(yes)))

  11. network(transport(tcp) flag(syslog-protocol))

  12. network(transport(udp) flag(syslog-protocol))

  13. network(transport(altp)flag(syslog-protocol))

  14. network(transport(tls) flag(syslog-protocol))

  15. network(transport(altp(tls-required(yes)) flag(syslog-protocol))

If you use the same driver and options in the destination of your syslog-ng PE client and the source of your syslog-ng PE server, everything should work as expected. Unfortunately there are some other combinations, that seem to work, but result in losing parts of the messages. The following table show the combinations:

Table 7: Source-destination driver combinations
Source \ Destination syslog/tcp syslog/udp syslog/tls network/tcp network/udp network/tls network/tcp/flag network/udp/flag network/tls/flag
syslog/tcp - - - - ! - - - - ! - - - -
syslog/udp - - - - - ! - - - - ! - - -
syslog/tls - - - - - - - ! - - - - ! -
network/tcp - - - - - - - - - ✔? - - - -
network/udp - ✔? - - - - - - - - ✔? - - -
network/tls - - - - - - - - - - - - ✔? -
network/tcp/flag ! - - - - ! - - - - - - - -
network/udp/flag - ! - - - - ! - - - - - - -
network/tls/flag - - - ! - - - - ! - - - - -
  • - This method does not work. The logs will not get to the server.

  • ✔ This method works.

  • ! This method has some visible drawbacks. The logs go through, but some of the values are missing/misplaced/and so on.

  • ✔? This method seems to work, but it is not recommended because this can change in a future release.

NFS file system for log files

Using the NFS network file system can lead to problems if NFS connection is not stable, therefore One Identity does neither recommend nor officially support such scenarios. If you can avoid it, do not store log files on NFS. If the NFS connection is stable and reliable, syslog-ng PE can read and write files on mounted NFS partitions as a normal file source or destination. Read this section carefully before using syslog-ng PE and NFS-mounted log files.

Risks

If there is any issue with the NFS connection (for example, connection loss, the NFS server stops), syslog-ng PE can stop working. These NFS issues can be related to the operating system, and can also vary depending on its patch level and kernel version. The possible effects include the following:

  • syslog-ng PE freezes, does not respond, does not process logs, is unable to stop or reload, and you can stop it only using the kill -9 command

  • syslog-ng PE is not able to start, and hangs during startup

  • Message loss or message duplication

  • Message becomes corrupt (it is not lost, but the message or some parts of it contain garbage)

  • When using the logstore() destination, the logstore file becomes corrupt

  • On some RHEL-based systems (possibly depending on the kernel version too), NFS returns NULL characters when reading a file that another process is writing at the very same moment.

Limitations of using syslog-ng PE with NFS
  • Do not use the logstore() destination to store files on an NFS-mounted partition

  • To use wildcards in the file source, set the force-directory_polling() option to yes to detect newly created files. Note that this option is available only in syslog-ng PE version 6.0.3 and newer versions of the 6.x branch, and is not yet available in syslog-ng PE version 7.

  • Since One Identity does not officially support scenarios where you use syslog-ng PE together with NFS, One Identity will handle support requests and bugs related to such scenarios only if you can reproduce the issue independently from NFS.

Recommendations for using NFS with syslog-ng PE

If you cannot avoid using NFS with syslog-ng PE note the following points.

  • USE at least NFS v4 (or newer if available)

  • USE the soft mount option (-o soft) to mount the partition

  • USE the TCP mount option (-o tcp) to mount the partition

  • DO NOT install syslog-ng PE on an NFS-mounted partition

  • DO NOT store the runtime files (for example, the configuration or the persist file) of syslog-ng PE on an NFS-mounted partition

  • DO NOT use logstore on an NFS-mounted partition, it can easily become corrupted

Installing syslog-ng

This chapter explains how to install syslog-ng Premium Edition on the supported platforms using the precompiled binary files.

  • The syslog-ng PE application features a unified installer package with identical look on every supported Linux and UNIX platforms. The generic installer, as well as installing platform-specific (for example, RPM) is described in the following sections.

The syslog-ng PE binaries include all required libraries and dependencies of syslog-ng PE, only the ncurses library is required as an external dependency (syslog-ng PE itself does not use the ncurses library, it is required only during the installation). The components are installed into the /opt/syslog-ng directory. It can automatically re-use existing configuration and license files, and also generate a simple configuration automatically into the /opt/syslog-ng/etc/syslog-ng.conf file.

NOTE:

There are two versions of every binary release. The one with the compact suffix does not include SQL support. If you are installing syslog-ng PE in client or relay mode, or you do not use the sql() source or destination, use the compact binaries. That way no unnecessary components are installed to your system.

The syslog-ng PE application can be installed interactively following the on-screen instructions as described in Installing syslog-ng using the .run installer, and also without user interaction using the silent installation option — see Installing syslog-ng PE without user-interaction.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating