syslog-ng Open Source Edition 3.25 - Administration Guide

BSD-syslog or legacy-syslog messages

This section describes the format of a syslog message, according to the legacy-syslog or BSD-syslog protocol. A syslog message consists of the following parts:

• PRI

• HEADER

• MSG

The total message cannot be longer than 1024 bytes.

The following is a sample syslog message:

<133>Feb 25 14:09:07 webserver syslogd: restart

The message corresponds to the following format:

<priority>timestamp hostname application: message

The different parts of the message are explained in the following sections.

IETF-syslog messages

This section describes the format of a syslog message, according to the IETF-syslog protocol. A syslog message consists of the following parts:

• HEADER (includes the PRI as well)

• STRUCTURED-DATA

• MSG

The following is a sample syslog message (source: https://tools.ietf.org/html/rfc5424):

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

The message corresponds to the following format:

<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

• Facility is 4, severity is 2, so PRI is 34.

• The VERSION is 1.

• The message was created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second.

• The message originated from a host that identifies itself as "mymachine.example.com".

• The APP-NAME is "su" and the PROCID is unknown.

• The MSGID is "ID47".

• The MSG is "'su root' failed for lonvick...", encoded in UTF-8.

• In this example, the encoding is defined by the BOM:

  The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

• There is no STRUCTURED-DATA present in the message, this is indicated by "-" in the STRUCTURED-DATA field.

The HEADER part of the message must be in plain ASCII format, the parameter values of the STRUCTURED-DATA part must be in UTF-8, while the MSG part should be in UTF-8. The different parts of the message are explained in the following sections.

[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]

The Enterprise-wide message model

The following section describes the structure of log messages using the Enterprise-wide message model or EWMM message format.

The Enterprise-wide message model or EWMM allows you to deliver structured messages from the initial receiving syslog-ng component right up to the central log server, through any number of hops. It does not matter if you parse the messages on the client, on a relay, or on the central server, their structured results will be available where you store the messages. Optionally, you can also forward the original raw message as the first syslog-ng component in your infrastructure has received it, which is important if you want to forward a message for example, to a SIEM system. To make use of the enterprise-wide message model, you have to use the syslog-ng() destination on the sender side, and the default-network-drivers() source on the receiver side.

The following is a sample log message in EWMM format.

<13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
{"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
/dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}

The message has the following parts:

• The header of the complies with the RFC5424 message format, where the PROGRAM field is set to @syslog-ng, and the SDATA field is empty.

• The MESSAGE part is in JSON format, and contains the actual message, as well as any name-value pairs that syslog-ng OSE has attached to or extracted from the message. The ${._TAGS} field contains the identifier of the syslog-ng source that has originally received the message on the first syslog-ng node.

To send a message in EWMM format, you can use the syslog-ng() destination driver, or the format-ewmm() template function.

To receive a message in EWMM format, you can use the default-destination-drivers() source driver, or the ewmm-parser() parser.

Message representation in syslog-ng OSE

When the syslog-ng OSE application receives a message, it automatically parses the message. The syslog-ng OSE application can automatically parse log messages that conform to the RFC3164 (BSD or legacy-syslog) or the RFC5424 (IETF-syslog) message formats. If syslog-ng OSE cannot parse a message, it results in an error.

A parsed syslog message has the following parts:

• Timestamps

  Two timestamps are associated with every message: one is the timestamp contained within the message (that is, when the sender sent the message), the other is the time when syslog-ng OSE has actually received the message.

• Severity

  The severity of the message.

• Facility

  The facility that sent the message.

• Tags

  Custom text labels added to the message that are mainly used for filtering. None of the current message transport protocols adds tags to the log messages. Tags can be added to the log message only within syslog-ng OSE. The syslog-ng OSE application automatically adds the id of the source as a tag to the incoming messages. Other tags can be added to the message by the pattern database, or using the tags() option of the source.

• IP address of the sender

  The IP address of the host that sent the message. Note that the IP address of the sender is a hard macro and cannot be modified within syslog-ng OSE but the associated hostname can be modified, for example, using rewrite rules.

• Hard macros

  Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. The most important consideration with hard macros is that they are read-only, meaning they cannot be modified using rewrite rules or other means.

• Soft macros

  Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the syslog-ng pattern database or a CSV-parser. The SDATA fields of RFC5424-formatted log messages become soft macros as well. In contrast with hard macros, soft macros are writable and can be modified within syslog-ng OSE, for example, using rewrite rules.

  NOTE:

  It is also possible to set the value of built-in soft macros using parsers, for example, to set the ${HOST} macro from the message using a column of a CSV-parser.

  The data extracted from the log messages using named pattern parsers in the pattern database are also soft macros.

  TIP:

  For the list of hard and soft macros, see Hard versus soft macros.