The following sections describe the structure of log messages. Currently there are two standard syslog message formats:
The old standard described in RFC 3164 (also called the BSD-syslog or the legacy-syslog protocol): see the section called “BSD-syslog or legacy-syslog messages”
The new standard described in RFC 5424 (also called the IETF-syslog protocol): see the section called “IETF-syslog messages”
How messages are represented in syslog-ng PE: see the section called “Message representation in syslog-ng PE”.
This section describes the format of a syslog message, according to the legacy-syslog or BSD-syslog protocol. A syslog message consists of the following parts:
The total message cannot be longer than 1024 bytes.
The following is a sample syslog message: <133>Feb 25 14:09:07 webserver syslogd: restart. The message corresponds to the following format: <priority>timestamp hostname application: message. The different parts of the message are explained in the following sections.
|
NOTE:
The syslog-ng application supports longer messages as well. For details, see the |
The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.
|
NOTE:
Facility codes may slightly vary between different platforms. The syslog-ng application accepts facility codes as numerical values as well. |
Table 2.2. syslog Message Facilities
Numerical Code | Facility |
---|---|
0 | kernel messages |
1 | user-level messages |
2 | mail system |
3 | system daemons |
4 | security/authorization messages |
5 | messages generated internally by syslogd |
6 | line printer subsystem |
7 | network news subsystem |
8 | UUCP subsystem |
9 | clock daemon |
10 | security/authorization messages |
11 | FTP daemon |
12 | NTP subsystem |
13 | log audit |
14 | log alert |
15 | clock daemon |
16-23 | locally used facilities (local0-local7) |
The following table lists the severity values.
Table 2.3. syslog Message Severities
Numerical Code | Severity |
---|---|
0 | Emergency: system is unusable |
1 | Alert: action must be taken immediately |
2 | Critical: critical conditions |
3 | Error: error conditions |
4 | Warning: warning conditions |
5 | Notice: normal but significant condition |
6 | Informational: informational messages |
7 | Debug: debug-level messages |
The HEADER part contains a timestamp and the hostname (without the domain name) or the IP address of the device. The timestamp field is the local time in the Mmm dd hh:mm:ss format, where:
Mmm is the English abbreviation of the month: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec.
dd is the day of the month on two digits. If the day of the month is less than 10, the first digit is replaced with a space. (For example Aug 7.)
hh:mm:ss is the local time. The hour (hh) is represented in a 24-hour format. Valid entries are between 00 and 23, inclusive. The minute (mm) and second (ss) entries are between 00 and 59 inclusive.
|
NOTE:
The syslog-ng application supports other timestamp formats as well, like ISO, or the PIX extended format. For details, see the |
This section describes the format of a syslog message, according to the IETF-syslog protocol. A syslog message consists of the following parts:
The following is a sample syslog message:[1]
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8
The message corresponds to the following format:
<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG
In this example, the Facility has the value of 4, severity is 2, so PRI is 34. The VERSION is 1. The message was created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second. The message originated from a host that identifies itself as "mymachine.example.com". The APP-NAME is "su" and the PROCID is unknown. The MSGID is "ID47". The MSG is "'su root' failed for lonvick...", encoded in UTF-8. The encoding is defined by theBOM[2]. There is no STRUCTURED-DATA present in the message, this is indicated by "-" in the STRUCTURED-DATA field. The MSG is "'su root' failed for lonvick...".
The HEADER part of the message must be in plain ASCII format, the parameter values of the STRUCTURED-DATA part must be in UTF-8, while the MSG part should be in UTF-8. The different parts of the message are explained in the following sections.
The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.
|
NOTE:
Facility codes may slightly vary between different platforms. The syslog-ng application accepts facility codes as numerical values as well. |
Table 2.4. syslog Message Facilities
Numerical Code | Facility |
---|---|
0 | kernel messages |
1 | user-level messages |
2 | mail system |
3 | system daemons |
4 | security/authorization messages |
5 | messages generated internally by syslogd |
6 | line printer subsystem |
7 | network news subsystem |
8 | UUCP subsystem |
9 | clock daemon |
10 | security/authorization messages |
11 | FTP daemon |
12 | NTP subsystem |
13 | log audit |
14 | log alert |
15 | clock daemon |
16-23 | locally used facilities (local0-local7) |
The following table lists the severity values.
Table 2.5. syslog Message Severities
Numerical Code | Severity |
---|---|
0 | Emergency: system is unusable |
1 | Alert: action must be taken immediately |
2 | Critical: critical conditions |
3 | Error: error conditions |
4 | Warning: warning conditions |
5 | Notice: normal but significant condition |
6 | Informational: informational messages |
7 | Debug: debug-level messages |
The HEADER part contains the following elements:
VERSION: Version number of the syslog protocol standard. Currently this can only be 1
.
ISOTIMESTAMP: The time when the message was generated in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00
.
HOSTNAME: The machine that originally sent the message.
APPLICATION: The device or application that generated the message
PID: The process name or process ID of the syslog application that sent the message. It is not necessarily the process ID of the application that generated the message.
MESSAGEID: The ID number of the message.
|
NOTE:
The syslog-ng application supports other timestamp formats as well, like ISO, or the PIX extended format. The timestamp used in the IETF-syslog protocol is derived from RFC3339, which is based on ISO8601. For details, see the |
The syslog-ng PE application will truncate the following fields:
If APP-NAME is longer than 48 characters it will be truncated to 48 characters.
If PROC-ID is longer than 128 characters it will be truncated to 128 characters.
If MSGID is longer than 32 characters it will be truncated to 32 characters.
If HOSTNAME is longer than 255 characters it will be truncated to 255 characters.
The STRUCTURED-DATA message part may contain meta- information about the syslog message, or application-specific information such as traffic counters or IP addresses. STRUCTURED-DATA consists of data blocks enclosed in brackets ([]). Every block includes the ID of the block, and one or more name=value pairs. The syslog-ng application automatically parses the STRUCTURED-DATA part of syslog messages, which can be referenced in macros (for details, see the section called “Macros of syslog-ng PE”). An example STRUCTURED-DATA block looks like:
[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]
The MSG part contains the text of the message itself. The encoding of the text must be UTF-8 if theBOM[3]character is present in the message. If the message does not contain the BOM character, the encoding is treated as unknown. Usually messages arriving from legacy sources do not include the BOM character. CRLF characters will not be removed from the message.
When the syslog-ng PE application receives a message, it automatically parses the message. The syslog-ng PE application can automatically parse log messages that conform to the RFC3164 (BSD or legacy-syslog) or the RFC5424 (IETF-syslog) message formats. If syslog-ng PE cannot parse a message, it results in an error.
|
TIP:
In case you need to relay messages that cannot be parsed without any modifications or changes, use the To parse non-syslog messages, for example, JSON, CSV, or other messages, you can use the built-in parsers of syslog-ng PE. For details, see Chapter 15, Parsing and segmenting structured messages. |
A parsed syslog message has the following parts.
Timestamps. Two timestamps are associated with every message: one is the timestamp contained within the message (that is, when the sender sent the message), the other is the time when syslog-ng PE has actually received the message.
Severity. The severity of the message.
Facility. The facility that sent the message.
Tags. Custom text labels added to the message that are mainly used for filtering. None of the current message transport protocols adds tags to the log messages. Tags can be added to the log message only within syslog-ng PE. The syslog-ng PE application automatically adds the id of the source as a tag to the incoming messages. Other tags can be added to the message by the pattern database, or using the tags()
option of the source.
IP address of the sender. The IP address of the host that sent the message. Note that the IP address of the sender is a hard macro and cannot be modified within syslog-ng PE but the associated hostname can be modified, for example, using rewrite rules.
Hard macros. Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. The most important consideration with hard macros is that they are read-only, meaning they cannot be modified using rewrite rules or other means.
Soft macros. Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the syslog-ng pattern database or a CSV-parser. The SDATA fields of RFC5424-formatted log messages become soft macros as well. In contrast with hard macros, soft macros are writable and can be modified within syslog-ng PE, for example, using rewrite rules.
|
NOTE:
It is also possible to set the value of built-in soft macros using parsers, for example, to set the ${HOST} macro from the message using a column of a CSV-parser. The data extracted from the log messages using named pattern parsers in the pattern database are also soft macros. |
|
TIP:
For the list of hard and soft macros, see the section called “Hard vs. soft macros”. |
Internally, syslog-ng PE represents every message as UTF-8. The maximal length of the log messages is limited by the log-msg-size()
option: if a message is longer than this value, syslog-ng PE truncates the message at the location it reaches the log-msg-size()
value, and discards the rest of the message.
When encoding is set in a source (using the encoding()
option) and the message is longer (in bytes) than log-msg-size()
in UTF-8 representation, syslog-ng PE splits the message at an undefined location (because the conversion between different encodings is not trivial).
Available in syslog-ng PE 5.1 and later.
The syslog-ng PE application allows you to select and construct name-value pairs from any information already available about the log message, or extracted from the message itself. You can directly use this structured information, for example, in the following places:
format-welf()
template function
mongodb()
destination
or in other destinations using the format-json()
template function.
When using value-pairs
, there are three ways to specify which information (that is, macros or other name-value pairs) to include in the selection.
These parameters are detailed in the section called “value-pairs()”.
By default, syslog-ng PE handles every data as strings. However, certain destinations and data formats (for example, SQL, MongoDB, JSON) support other types of data as well, for example, numbers or dates. The syslog-ng PE application allows you to specify the data type in templates (this is also called type-hinting). If the destination driver supports data types, it converts the incoming data to the specified data type. For example, this allows you to store integer numbers as numbers in MongoDB, instead of strings.
|
Caution:
Hazard of data loss! If syslog-ng PE cannot convert the data into the specified type, an error occurs, and syslog-ng PE drops the message by default. To change how syslog-ng PE handles data-conversion errors, see the section called “on-error()”. |
To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>")
, for example: int("$PID")
.
Currently the mongodb()
destination and the format-json
template function supports data types.
Example 2.6. Using type-hinting
The following example stores the MESSAGE, PID, DATE, and PROGRAM fields of a log message in a MongoDB database. The DATE and PID parts are stored as numbers instead of strings.
mongodb( value-pairs(pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM")) pair("message", "$MESSAGE")) ) );
The following example formats the same fields into JSON.
$(format-json date=datetime("$UNIXTIME") pid=int64("$PID") program="$PROGRAM" message="$MESSAGE")
The syslog-ng PE application currently supports the following data-types.
boolean
: Converts the data to a boolean value. Anything that begins with a t
or 1
is converted to true, anything that begins with an f
or 0
is converted to false.
datetime
: Use it only with UNIX timestamps, anything else will likely result in an error. This means that currently you can use only the $UNIXTIME
macro for this purpose.
literal
: The data as a literal string, without adding any quotes or escape characters.
int
or int32
: 32-bit integer.
int64
: 64-bit integer.
string
: The data as a string.
Description: The value-pairs()
option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a mongodb()
destination.
Example 2.7. Using the value-pairs()
option
The following example selects every available information about the log message, except for the date-related macros (R_*
and S_*
), selects the .SDATA.meta.sequenceId
macro, and defines a new value-pair called MSGHDR
that contains the program name and PID of the application that sent the log message.
value-pairs( scope(nv_pairs core syslog all_macros selected_macros everything) exclude("R_*") exclude("S_*") key(".SDATA.meta.sequenceId") pair("MSGHDR" "$PROGRAM[$PID]: ") )
The following example selects the same information as the previous example, but converts it into JSON format.
$(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \ --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \ --pair MSGHDR="$PROGRAM[$PID]: ")
|
NOTE:
Every macro is included in the selection only once, but redundant information may appear if multiple macros include the same information (for example, including several date-related macros in the selection). |
The value-pairs()
option has the following parameters. The parameters are evaluated in the following order:
rekey() | |
Type: | <pattern-to-select-names>, <list of transformations> |
Default: | empty string |
Description: This option allows you to manipulate and modify the name of the value-pairs. You can define transformations, which are are applied to the selected name-value pairs. The first parameter of the The following transformations are available:
Example 2.8. Using the rekey() option The following sample selects every value-pair that begins with value-pairs( key(".cee.*" rekey( shift(4) add-prefix("events.") ) ) ) The $(format-json --rekey .cee.* --add-prefix events.) |
When you send your log messages from a syslog-ng PE client through the network to a syslog-ng PE server, you can use different protocols and options. Every combination has its advantages and disadvantages. The most important thing is to use matching protocols and options, so the server handles the incoming log messages properly.
In syslog-ng PE you can change many aspects of the network communication. First of all, there is the structure of the messages itself. Currently, syslog-ng PE supports two standard syslog protocols: the BSD (RFC3164) and the syslog (RFC5424) message format.
These RFCs describe the format and the structure of the log message, and add a (lightweight) framing around the messages. You can set this framing/structure by selecting the appropriate driver in syslog-ng PE. There are two drivers you can use: the network()
driver and the syslog()
driver. The syslog()
driver is for the syslog (RFC5424) protocol and the network() driver is for the BSD (RFC3164) protocol.
The tcp()
and udp()
drivers are now deprecated, they are essentially equivalent with the network(transport(tcp))
and network(transport(udp))
drivers.
In addition to selecting the driver to use, both drivers allow you to use different transport-layer protocols: TCP and UDP, and optionally also higher-level transport protocols: TLS (over TCP, and RLTP (optionally using TLS). To complicate things a bit more, you can configure the network()
driver (corresponding to the BSD (RFC3164) protocol) to send the messages in the syslog (RFC5424) format (but without the framing used in RFC5424) using the flag(syslog-protocol)
option.
Because some combination of drivers and options are invalid, you can use the following drivers and options as sources and as destinations:
syslog(transport(tcp))
syslog(transport(udp))
syslog(transport(rltp))
syslog(transport(tls))
syslog(transport(rltp(tls-required(yes)))
network(transport(tcp))
network(transport(udp))
network(transport(rltp))
network(transport(tls))
network(transport(rltp(tls-required(yes)))
network(transport(tcp) flag(syslog-protocol))
network(transport(udp) flag(syslog-protocol))
network(transport(rltp)flag(syslog-protocol))
network(transport(tls) flag(syslog-protocol))
network(transport(rltp(tls-required(yes)) flag(syslog-protocol))
If you use the same driver and options in the destination of your syslog-ng PE client and the source of your syslog-ng PE server, everything should work as expected. Unfortunately there are some other combinations, that seem to work, but result in losing parts of the messages. The following table show the combinations:
Table 2.6. Source-destination driver combinations
S \ D | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | ✔ | - | - | - | - | ! | - | - | - | - | ! | - | - | - | - |
2 | - | ✔ | - | - | - | - | ! | - | - | - | - | ! | - | - | - |
3 | - | - | ✔ | - | - | - | - | ! | - | - | - | - | ! | - | - |
4 | - | - | - | ✔ | - | - | - | - | ! | - | - | - | - | ! | - |
5 | - | - | - | - | ✔ | - | - | - | - | ! | - | - | - | - | ! |
6 | - | - | - | - | - | ✔ | - | - | - | - | ✔? | - | - | - | - |
7 | - | ✔? | - | - | - | - | ✔ | - | - | - | - | ✔? | - | - | - |
8 | - | - | ✔? | - | - | - | - | ✔ | - | - | - | - | ✔? | - | - |
9 | - | - | - | - | - | - | - | - | ✔ | - | - | - | - | ✔? | - |
10 | - | - | - | - | ✔? | - | - | - | - | ✔ | - | - | - | - | ✔? |
11 | ! | - | - | - | - | ! | - | - | - | - | ✔ | - | - | - | - |
12 | - | ! | - | - | - | - | ! | - | - | - | - | ✔ | - | - | - |
13 | - | - | ! | - | - | - | - | ! | - | - | - | - | ✔ | - | - |
14 | - | - | - | ! | - | - | - | - | ! | - | - | - | - | ✔ | - |
15 | - | - | - | - | ! | - | - | - | - | ! | - | - | - | - | ✔ |
- This method does not work. The logs will not get to the server.
✔ This method works.
! This method has some visible drawbacks. The logs go through, but some of the values are missing/misplaced/and so on.
✔? This method seems to work, but it is not recommended because this can change in a future release.
The configuration options of the syslog-ng Agent for Windows application are much more limited. The configurable elements are:
Protocol: (Legacy BSD Syslog Protocol, Syslog Protocol, Snare Protocol)
option
option
The protocol corresponds to the network()
and syslog()
drivers of syslog-ng PE, except that the syslog-ng PE server does not support the Snare protocol.
The transport(rltp)
option.
The transport(tls)
option if RLTP is not set, and to the transport(rltp(tls-required(yes))
otherwise.
Therefore with syslog-ng Agent the following cases are possible:
Table 2.7. Source-destination driver combinations in syslog-ng Agent for Windows
© 2024 One Identity LLC. ALL RIGHTS RESERVED. Terms of Use Privacy Cookie Preference Center