The Enterprise-wide message model
The following section describes the structure of log messages using the Enterprise-wide message model or EWMM message format.
The Enterprise-wide message model or EWMM allows you to deliver structured messages from the initial receiving syslog-ng component right up to the central log server, through any number of hops. It does not matter if you parse the messages on the client, on a relay, or on the central server, their structured results will be available where you store the messages. Optionally, you can also forward the original raw message as the first syslog-ng component in your infrastructure has received it, which is important if you want to forward a message for example, to a SIEM system. To make use of the enterprise-wide message model, you have to use the syslog-ng() destination on the sender side, and the default-network-drivers() source on the receiver side.
The following is a sample log message in EWMM format.
<13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
{"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
/dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
The message has the following parts:
-
The header of the complies with the RFC5424 message format, where the PROGRAM field is set to @syslog-ng, and the SDATA field is empty.
-
The MESSAGE part is in JSON format, and contains the actual message, as well as any name-value pairs that syslog-ng OSE has attached to or extracted from the message. The ${._TAGS} field contains the identifier of the syslog-ng source that has originally received the message on the first syslog-ng node.
To send a message in EWMM format, you can use the syslog-ng() destination driver, or the format-ewmm() template function.
To receive a message in EWMM format, you can use the default-destination-drivers() source driver, or the ewmm-parser() parser.
Message representation in syslog-ng OSE
When the syslog-ng OSE application receives a message, it automatically parses the message. The syslog-ng OSE application can automatically parse log messages that conform to the RFC3164 (BSD or legacy-syslog) or the RFC5424 (IETF-syslog) message formats. If syslog-ng OSE cannot parse a message, it results in an error.
TIP: In case you need to relay messages that cannot be parsed without any modifications or changes, use the flags(no-parse) option in the source definition, and a template containing only the ${MESSAGE} macro in the destination definition.
To parse non-syslog messages, for example, JSON, CSV, or other messages, you can use the built-in parsers of syslog-ng OSE. For details, see parser: Parse and segment structured messages.
A parsed syslog message has the following parts:
-
Timestamps
Two timestamps are associated with every message: one is the timestamp contained within the message (that is, when the sender sent the message), the other is the time when syslog-ng OSE has actually received the message.
-
Severity
The severity of the message.
-
Facility
The facility that sent the message.
-
Tags
Custom text labels added to the message that are mainly used for filtering. None of the current message transport protocols adds tags to the log messages. Tags can be added to the log message only within syslog-ng OSE. The syslog-ng OSE application automatically adds the id of the source as a tag to the incoming messages. Other tags can be added to the message by the pattern database, or using the tags() option of the source.
-
IP address of the sender
The IP address of the host that sent the message. Note that the IP address of the sender is a hard macro and cannot be modified within syslog-ng OSE but the associated hostname can be modified, for example, using rewrite rules.
-
Hard macros
Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. The most important consideration with hard macros is that they are read-only, meaning they cannot be modified using rewrite rules or other means.
-
Soft macros
Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the syslog-ng pattern database or a CSV-parser. The SDATA fields of RFC5424-formatted log messages become soft macros as well. In contrast with hard macros, soft macros are writable and can be modified within syslog-ng OSE, for example, using rewrite rules.
NOTE: It is also possible to set the value of built-in soft macros using parsers, for example, to set the ${HOST} macro from the message using a column of a CSV-parser.
The data extracted from the log messages using named pattern parsers in the pattern database are also soft macros.
Message size and encoding
Internally, syslog-ng OSE represents every message as UTF-8. The maximal length of the log messages is limited by the log-msg-size() option: if a message is longer than this value, syslog-ng OSE truncates the message at the location it reaches the log-msg-size() value, and discards the rest of the message.
When encoding is set in a source (using the encoding() option) and the message is longer (in bytes) than log-msg-size() in UTF-8 representation, syslog-ng OSE splits the message at an undefined location (because the conversion between different encodings is not trivial).
Structuring macros, metadata, and other value-pairs
Available in syslog-ng OSE 3.3 and later.
The syslog-ng OSE application allows you to select and construct name-value pairs from any information already available about the log message, or extracted from the message itself. You can directly use this structured information, for example, in the following places:
When using value-pairs, there are three ways to specify which information (that is, macros or other name-value pairs) to include in the selection.
-
Select groups of macros using the scope() parameter, and optionally remove certain macros from the group using the exclude() parameter.
-
List specific macros to include using the key() parameter.
-
Define new name-value pairs to include using the pair() parameter.
These parameters are detailed in value-pairs().
Specifying data types in value-pairs
By default, syslog-ng OSE handles every data as strings. However, certain destinations and data formats (for example, SQL, MongoDB, JSON, AMQP) support other types of data as well, for example, numbers or dates. The syslog-ng OSE application allows you to specify the data type in templates (this is also called type-hinting). If the destination driver supports data types, it converts the incoming data to the specified data type. For example, this allows you to store integer numbers as numbers in MongoDB, instead of strings.
|
|
Caution:
Hazard of data loss! If syslog-ng OSE cannot convert the data into the specified type, an error occurs, and syslog-ng OSE drops the message by default. To change how syslog-ng OSE handles data-conversion errors, see on-error(). |
To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>"), for example: int("$PID").
Currently the mongodb() destination and the format-json and format-flat-json() template functions support data types.
Example: Using type-hinting
The following example stores the MESSAGE, PID, DATE, and PROGRAM fields of a log message in a MongoDB database. The DATE and PID parts are stored as numbers instead of strings.
mongodb(
value-pairs(pair("date", datetime("$UNIXTIME"))
pair("pid", int64("$PID"))
pair("program", "$PROGRAM"))
pair("message", "$MESSAGE"))
)
);
The following example formats the same fields into JSON.
$(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$MESSAGE)
The following example formats the MESSAGE field as a JSON list.
$(format-json message=list($MESSAGE))"
The syslog-ng OSE application currently supports the following data-types.
-
boolean: Converts the data to a boolean value. Anything that begins with a t or 1 is converted to true, anything that begins with an f or 0 is converted to false.
-
datetime: Use it only with UNIX timestamps, anything else will likely result in an error. This means that currently you can use only the $UNIXTIME macro for this purpose.
-
double: A floating-point number.
-
literal: The data as a literal string, without adding any quotes or escape characters.
-
list: The data as a list. For details, see the list manipulation template functions in Template functions of syslog-ng OSE.
-
int or int32: 32-bit integer.
-
int64: 64-bit integer.
-
string: The data as a string.
