syslog-ng Open Source Edition 3.16 - Administration Guide

The structure of a log message

The following sections describe the structure of log messages. Currently there are two standard syslog message formats:

• The old standard described in RFC 3164 (also called the BSD-syslog or the legacy-syslog protocol): see BSD-syslog or legacy-syslog messages

• The new standard described in RFC 5424 (also called the IETF-syslog protocol): see IETF-syslog messages

• The Enterprise-wide message model or EWMM allows you to deliver structured messages between syslog-ng nodes: see Enterprise-wide message model (EWMM)

• How messages are represented in syslog-ng OSE: see Message representation in syslog-ng OSE.

BSD-syslog or legacy-syslog messages

This section describes the format of a syslog message, according to the legacy-syslog or BSD-syslog protocol. A syslog message consists of the following parts:

• PRI

• HEADER

• MSG

The total message cannot be longer than 1024 bytes.

The following is a sample syslog message:

<133>Feb 25 14:09:07 webserver syslogd: restart

The message corresponds to the following format:

<priority>timestamp hostname application: message

The different parts of the message are explained in the following sections.

The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.

The following table lists the severity values.

The HEADER message part

The HEADER part contains a timestamp and the hostname (without the domain name) or the IP address of the device. The timestamp field is the local time in the Mmm dd hh:mm:ss format, where:

• Mmm is the English abbreviation of the month: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec.

• dd is the day of the month on two digits. If the day of the month is less than 10, the first digit is replaced with a space. (For example Aug 7.)

• hh:mm:ss is the local time. The hour (hh) is represented in a 24-hour format. Valid entries are between 00 and 23, inclusive. The minute (mm) and second (ss) entries are between 00 and 59 inclusive.

The MSG message part

The MSG part contains the name of the program or process that generated the message, and the text of the message itself. The MSG part is usually in the following format: program[pid]: message text.

IETF-syslog messages

This section describes the format of a syslog message, according to the IETF-syslog protocol. A syslog message consists of the following parts:

• HEADER (includes the PRI as well)

• STRUCTURED-DATA

• MSG

The following is a sample syslog message (source: https://tools.ietf.org/html/rfc5424):

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

The message corresponds to the following format:

<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

In this example, the Facility has the value of 4, severity is 2, so PRI is 34. The VERSION is 1. The message was created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second. The message originated from a host that identifies itself as "mymachine.example.com". The APP-NAME is "su" and the PROCID is unknown. The MSGID is "ID47". The MSG is "'su root' failed for lonvick...", encoded in UTF-8. The encoding is defined by the BOM:

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

There is no STRUCTURED-DATA present in the message, this is indicated by "-" in the STRUCTURED-DATA field. The MSG is "'su root' failed for lonvick...".

The HEADER part of the message must be in plain ASCII format, the parameter values of the STRUCTURED-DATA part must be in UTF-8, while the MSG part should be in UTF-8. The different parts of the message are explained in the following sections.

The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.

The following table lists the severity values.

The HEADER message part

The HEADER part contains the following elements:

• VERSION: Version number of the syslog protocol standard. Currently this can only be 1.

• ISOTIMESTAMP: The time when the message was generated in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00.

• HOSTNAME: The machine that originally sent the message.

• APPLICATION: The device or application that generated the message

• PID: The process name or process ID of the syslog application that sent the message. It is not necessarily the process ID of the application that generated the message.

• MESSAGEID: The ID number of the message.

The syslog-ng OSE application will truncate the following fields:

• If APP-NAME is longer than 48 characters it will be truncated to 48 characters.

• If PROC-ID is longer than 128 characters it will be truncated to 128 characters.

• If MSGID is longer than 32 characters it will be truncated to 32 characters.

• If HOSTNAME is longer than 255 characters it will be truncated to 255 characters.

The STRUCTURED-DATA message part

The STRUCTURED-DATA message part may contain meta- information about the syslog message, or application-specific information such as traffic counters or IP addresses. STRUCTURED-DATA consists of data blocks enclosed in brackets ([]). Every block includes the ID of the block, and one or more name=value pairs. The syslog-ng application automatically parses the STRUCTURED-DATA part of syslog messages, which can be referenced in macros (for details, see Macros of syslog-ng OSE). An example STRUCTURED-DATA block looks like:

[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]

The MSG message part

The MSG part contains the text of the message itself. The encoding of the text must be UTF-8 if theBOMThe byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.1character is present in the message. If the message does not contain the BOM character, the encoding is treated as unknown. Usually messages arriving from legacy sources do not include the BOM character. CRLF characters will not be removed from the message.

The Enterprise-wide message model

The following section describes the structure of log messages using the Enterprise-wide message model or EWMM message format.

The Enterprise-wide message model or EWMM allows you to deliver structured messages from the initial receiving syslog-ng component right up to the central log server, through any number of hops. It does not matter if you parse the messages on the client, on a relay, or on the central server, their structured results will be available where you store the messages. Optionally, you can also forward the original raw message as the first syslog-ng component in your infrastructure has received it, which is important if you want to forward a message for example to a SIEM system. To make use of the enterprise-wide message model, you have to use the syslog-ng() destination on the sender side, and the default-network-drivers() source on the receiver side.

The follwing is a sample log message in EWMM format.

<13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
{"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
/dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}

The message has the following parts.

• The header of the complies with the RFC5424 message format, where the PROGRAM field is set to @syslog-ng, and the SDATA field is empty.

• The MESSAGE part is in JSON format, and contains the actual message, as well as any name-value pairs that syslog-ng OSE has attached to or extracted from the message. The ${._TAGS} field contains the identifier of the syslog-ng source that has originally received the message on the first syslog-ng node.

To send a message in EWMM format, you can use the syslog-ng() destination driver, or the format-ewmm() template function.

To receive a message in EWMM format, you can use the default-destination-drivers() source driver, or the ewmm-parser() parser.