The following macros are available in syslog-ng PE.
|
|
Caution:
These macros are available when syslog-ng PE successfully parses the incoming message as a syslog message, or you use some other parsing method and map the parsed values to these macros.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Note that since flags(no-parse) disables message parsing, it interferes with other flags, for example, disables flags(no-multi-line). |
AMPM
Description: Typically used together with the ${HOUR12} macro, ${AMPM} returns the period of the day: AM for hours before mid day and PM for hours after mid day. In reference to a 24-hour clock format, AM is between 00:00-12:00 and PM is between 12:00-24:00. 12AM is midnight. Available in syslog-ng PE 3.2 and later.
BSDTAG
Description: Facility/priority information in the format used by the FreeBSD syslogd: a priority number followed by a letter that indicates the facility. The priority number can range from 0 to 7. The facility letter can range from A to Y, where A corresponds to facility number zero (LOG_KERN), B corresponds to facility 1 (LOG_USER), and so on.
DATE, C_DATE, R_DATE, S_DATE
Description: Date of the message using the BSD-syslog style timestamp format (month/day/hour/minute/second, each expressed in two digits). This is the original syslog time stamp without year information, for example: Jun 13 15:58:00.
DAY, C_DAY, R_DAY, S_DAY
Description: The day the message was sent.
FACILITY
Description: The name of the facility (for example, kern) that sent the message.
FACILITY_NUM
Description: The numerical code of the facility (for example, 0) that sent the message.
FILE_NAME
Description: Name of the log file (including its path) from where syslog-ng PE received the message (only available if syslog-ng PE received the message from a file or a wildcard-file source). If you need only the path or the filename, use the dirname and basename template functions.
FULLDATE, C_FULLDATE, R_FULLDATE, S_FULLDATE
Description: A nonstandard format for the date of the message using the same format as ${DATE}, but including the year as well, for example: 2006 Jun 13 15:58:00.
FULLHOST
Description: The name of the source host where the message originates from.
-
If the message traverses several hosts and the chain-hostnames() option is on, the first host in the chain is used.
-
If the keep-hostname() option is disabled (keep-hostname(no)), the value of the $FULLHOST macro will be the DNS hostname of the host that sent the message to syslog-ng PE (that is, the DNS hostname of the last hop). In this case the $FULLHOST and $FULLHOST_FROM macros will have the same value.
-
If the keep-hostname() option is enabled (keep-hostname(yes)), the value of the $FULLHOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server.
NOTE: The use-dns(), use-fqdn(), normalize-hostnames(), and dns-cache() options will have no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
For details on using name resolution in syslog-ng PE, see Using name resolution in syslog-ng.
FULLHOST_FROM
Description: The FQDN of the host that sent the message to syslog-ng as resolved by syslog-ng using DNS. If the message traverses several hosts, this is the last host in the chain.
The syslog-ng PE application uses the following procedure to determine the value of the $FULLHOST_FROM macro:
-
The syslog-ng PE application takes the IP address of the host sending the message.
-
If the use-dns() option is enabled, syslog-ng PE attempts to resolve the IP address to a hostname. If it succeeds, the returned hostname will be the value of the $FULLHOST_FROM macro. This value will be the FQDN of the host if the use-fqdn() option is enabled, but only the hostname if use-fqdn() is disabled.
-
If the use-dns() option is disabled, or the address resolution fails, the ${FULLHOST_FROM} macro will return the IP address of the sender host.
For details on using name resolution in syslog-ng PE, see Using name resolution in syslog-ng.
HOUR, C_HOUR, R_HOUR, S_HOUR
Description: The hour of day the message was sent.
HOUR12, C_HOUR12, R_HOUR12, S_HOUR12
Description: The hour of day the message was sent in 12-hour clock format. See also the ${AMPM} macro. 12AM is midnight. Available in syslog-ng PE 3.2 and later.
HOST
Description: The name of the source host where the message originates from.
-
If the message traverses several hosts and the chain-hostnames() option is on, the first host in the chain is used.
-
If the keep-hostname() option is disabled (keep-hostname(no)), the value of the $HOST macro will be the DNS hostname of the host that sent the message to syslog-ng PE (that is, the DNS hostname of the last hop). In this case the $HOST and $HOST_FROM macros will have the same value.
-
If the keep-hostname() option is enabled (keep-hostname(yes)), the value of the $HOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server.
NOTE: The use-dns(), use-fqdn(), normalize-hostnames(), and dns-cache() options will have no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
For details on using name resolution in syslog-ng PE, see Using name resolution in syslog-ng.
HOST_FROM
Description: The FQDN of the host that sent the message to syslog-ng as resolved by syslog-ng using DNS. If the message traverses several hosts, this is the last host in the chain.
The syslog-ng PE application uses the following procedure to determine the value of the $HOST_FROM macro:
-
The syslog-ng PE application takes the IP address of the host sending the message.
-
If the use-dns() option is enabled, syslog-ng PE attempts to resolve the IP address to a hostname. If it succeeds, the returned hostname will be the value of the $HOST_FROM macro. This value will be the FQDN of the host if the use-fqdn() option is enabled, but only the hostname if use-fqdn() is disabled.
-
If the use-dns() option is disabled, or the address resolution fails, the ${HOST_FROM} macro will return the IP address of the sender host.
For details on using name resolution in syslog-ng PE, see Using name resolution in syslog-ng.
ISODATE, C_ISODATE, R_ISODATE, S_ISODATE
Description: Date of the message in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00. If possible, it is recommended to use ${ISODATE} for timestamping. Note that syslog-ng PE can produce fractions of a second (for example, milliseconds) in the timestamp by using the frac-digits() global or per-destination option.
NOTE: As syslog-ng PE is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, syslog-ng PE will truncate the fraction seconds in the timestamps after 6 digits.
LEVEL_NUM
Description: The priority (also called severity) of the message, represented as a numeric value, for example, 3. For the textual representation of this value, use the ${LEVEL} macro. See PRIORITY or LEVEL for details.
LOGHOST
Description: The hostname of the computer running syslog-ng PE — it returns the same result as the hostname command.
MESSAGE
Description: Text contents of the log message without the program name and pid. The program name and the pid together are available in the ${MSGHDR} macro, and separately in the ${PROGRAM} and ${PID} macros.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Note that since flags(no-parse) disables message parsing, it interferes with other flags, for example, disables flags(no-multi-line).
The ${MSG} macro is an alias of the ${MESSAGE} macro: using ${MSG} in syslog-ng PE is equivalent to ${MESSAGE}.
Note that before syslog-ng version 3.0, the ${MESSAGE} macro included the program name and the pid. In syslog-ng 3.0, the ${MESSAGE} macro became equivalent with the ${MSGONLY} macro.
MIN, C_MIN, R_MIN, S_MIN
Description: The minute the message was sent.
MONTH, C_MONTH, R_MONTH, S_MONTH
Description: The month the message was sent as a decimal value, prefixed with a zero if smaller than 10.
MONTH_ABBREV, C_MONTH_ABBREV, R_MONTH_ABBREV, S_MONTH_ABBREV
Description: The English abbreviation of the month name (3 letters).
MONTH_NAME, C_MONTH_NAME, R_MONTH_NAME, S_MONTH_NAME
Description: The English name of the month name.
MONTH_WEEK, C_MONTH_WEEK, R_MONTH_WEEK, S_MONTH_WEEK
Description: The number of the week in the given month (0-5). The week with numerical value 1 is the first week containing a Monday. The days of month before the first Monday are considered week 0. For example, if a 31-day month begins on a Sunday, then the 1st of the month is week 0, and the end of the month (the 30th and 31st) is week 5.
MSEC, C_MSEC, R_MSEC, S_MSEC
Description: The millisecond the message was sent.
Available in syslog-ng PE version 4 F2 and later.
MSG
The ${MSG} macro is an alias of the ${MESSAGE} macro, using ${MSG} in syslog-ng PE is equivalent to ${MESSAGE}. For details on this macro, see MESSAGE.
MSGHDR
Description: The name and the PID of the program that sent the log message in PROGRAM[PID]: format. Includes a trailing whitespace. Note that the macro returns an empty value if both the PROGRAM and PID fields of the message are empty.
MSGID
Description: A string specifying the type of the message in IETF-syslog (RFC5424-formatted) messages. For example, a firewall might use the ${MSGID} "TCPIN" for incoming TCP traffic and the ${MSGID} "TCPOUT" for outgoing TCP traffic. By default, syslog-ng PE does not specify this value, but uses a dash (-) character instead. If an incoming message includes the ${MSGID} value, it is retained and relayed without modification.
MSGONLY
Description: Message contents without the program name or pid. Starting with syslog-ng PE 3.0, the following macros are equivalent: ${MSGONLY}, ${MSG}, ${MESSAGE}. For consistency, use the ${MESSAGE} macro. For details, see MESSAGE.
PID
Description: The PID of the program sending the message.
PRI
Description: The priority and facility encoded as a 2 or 3 digit decimal number as it is present in syslog messages.
PRIORITY or LEVEL
Description: The priority (also called severity) of the message, for example, error. For the textual representation of this value, use the ${LEVEL} macro. See PRIORITY or LEVEL for details.
PROGRAM
Description: The name of the program sending the message. Note that the content of the ${PROGRAM} variable may not be completely trusted as it is provided by the client program that constructed the message.
RAWMSG
Description: The original message as received from the client. Note that this macro is available only in 7.0.9 and later, and only if syslog-ng received the message using the default-network-drivers-ng() source, or the source receiving the message has the store-raw-message flag set.
RCPTID
Description: When the use-rcptid global option is set to yes, syslog-ng PE automatically assigns a unique reception ID to every received message. You can access this ID and use it in templates via the ${RCPTID} macro. The reception ID is a monotonously increasing 48-bit integer number, that can never be zero (if the counter overflows, it restarts with 1).
RUNID
Description: An ID that changes its value every time syslog-ng PE is restarted, but not when reloaded.
SDATA, .SDATA.SDID.SDNAME
Description: The syslog-ng application automatically parses the STRUCTURED-DATA part of IETF-syslog messages, which can be referenced in macros. The ${SDATA} macro references the entire STRUCTURED-DATA part of the message, while structured data elements can be referenced using the ${.SDATA.SDID.SDNAME} macro.Available only in syslog-ng Premium Edition 4.0 and later.
NOTE: When using STRUCTURED-DATA macros, consider the following:
-
When referencing an element of the structured data, the macro must begin with the dot (.) character. For example, ${.SDATA.timeQuality.isSynced}.
-
The SDID and SDNAME parts of the macro names are case sensitive: ${.SDATA.timeQuality.isSynced} is not the same as ${.SDATA.TIMEQUALITY.ISSYNCED}.
Example: Using SDATA macros
For example, if a log message contains the following structured data: [exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"] you can use macros like: ${.SDATA.exampleSDID@0.eventSource} — this would return the Application string in this case.
SEC, C_SEC, R_SEC, S_SEC
Description: The second the message was sent.
SEQNUM
Description: The ${SEQNUM} macro contains a sequence number for the log message. The value of the macro depends on the scenario, and can be one of the following:
-
If syslog-ng PE receives a message via the IETF-syslog protocol that includes a sequence ID, this ID is automatically available in the ${SEQNUM} macro.
-
If the message is a Cisco IOS log message using the extended timestamp format, then syslog-ng PE stores the sequence number from the message in this macro. If you forward this message the IETF-syslog protocol, syslog-ng PE includes the sequence number received from the Cisco device in the ${.SDATA.meta.sequenceId} part of the message.
NOTE: To enable sequence numbering of log messages on Cisco devices, use the following command on the device (available in IOS 10.0 and later): service sequence-numbers. For details, see the manual of your Cisco device.
-
For locally generated messages (that is, for messages that are received from a local source, and not from the network), syslog-ng PE calculates a sequence number when sending the message to a destination (it is not calculated for relayed messages).
-
The sequence number is not global, but per-destination. Essentially, it counts the number of messages sent to the destination.
-
This sequence number increases by one for every message sent to the destination. It not lost when syslog-ng PE is reloaded, but it is reset when syslog-ng PE is restarted.
-
This sequence number is added to every message that uses the IETF-syslog protocol (${.SDATA.meta.sequenceId}), and can be added to BSD-syslog messages using the ${SEQNUM} macro.
NOTE: If you need a sequence number for every log message that syslog-ng PE receives, use the RCPTID macro.
SOURCE
Description: The identifier of the source statement in the syslog-ng PE configuration file that received the message. For example, if syslog-ng PE received the log message from the source s_local { internal(); }; source statement, the value of the ${SOURCE} macro is s_local. This macro is mainly useful for debugging and troubleshooting purposes.
SOURCEIP
Description: IP address of the host that sent the message to syslog-ng. (That is, the IP address of the host in the ${FULLHOST_FROM} macro.) Please note that when a message traverses several relays, this macro contains the IP of the last relay.
STAMP, R_STAMP, S_STAMP
Description: A timestamp formatted according to the ts-format() global or per-destination option.
SYSUPTIME
Description: The time elapsed since the syslog-ng PE instance was started (that is, the uptime of the syslog-ng PE process). The value of this macro is an integer containing the time in 1/100th of the second.
Note that syslog-ng PE evaluates the macro every time it is processed, so even if you use the same macro for the same message, its value can be different. For example, if you use it in a filter and in a destination filename, their values will be different even for the same message.
Available in syslog-ng PE version 4 F1 and later.
TAG
Description: The priority and facility encoded as a 2 digit hexadecimal number.
TAGS
Description: A comma-separated list of the tags assigned to the message. Available only in syslog-ng Premium Edition 3.2 and later.
NOTE: Note that the tags are not part of the log message and are not automatically transferred from a client to the server. For example, if a client uses a pattern database to tag the messages, the tags are not transferred to the server. A way of transferring the tags is to explicitly add them to the log messages using a template and the ${TAGS} macro, or to add them to the structured metadata part of messages when using the IETF-syslog message format.
When sent as structured metadata, it is possible to reference to the list of tags on the central server, and for example, to add them to a database column.
TZ, C_TZ, R_TZ, S_TZ
Description: An alias of the ${TZOFFSET} macro.
TZOFFSET, C_TZOFFSET, R_TZOFFSET, S_TZOFFSET
Description: The time-zone as hour offset from GMT, for example: -07:00. In syslog-ng 1.6.x this used to be -0700 but as ${ISODATE} requires the colon it was added to ${TZOFFSET} as well.
UNIXTIME, C_UNIXTIME, R_UNIXTIME, S_UNIXTIME
Description: Standard UNIX timestamp, represented as the number of seconds since 1970-01-01T00:00:00.
UNIQID
Description: A globally unique ID generated from the HOSTID and the RCPTID in the format of HOSTID@RCPTID. For details, see use-uniqid() and RCPTID.
Available in syslog-ng PE version 5 F2 and later.
USEC, C_USEC, R_USEC, S_USEC
Description: The microsecond the message was sent.
Available in syslog-ng PE version 4 F2 and later.
YEAR, C_YEAR, R_YEAR, S_YEAR
Description: The year the message was sent.
WEEK, C_WEEK, R_WEEK, S_WEEK
Description: The week number of the year, prefixed with a zero for the first nine week of the year. (The first Monday in the year marks the first week.)
WEEK_DAY_ABBREV, C_WEEK_DAY_ABBREV, R_WEEK_DAY_ABBREV, S_WEEK_DAY_ABBREV
Description: The 3-letter English abbreviation of the name of the day the message was sent, for example, Thu.
WEEK_DAY, C_WEEK_DAY, R_WEEK_DAY, S_WEEK_DAY
Description: The day of the week as a numerical value (1-7).
WEEK_DAY_NAME, C_WEEK_DAY_NAME, R_WEEK_DAY_NAME, S_WEEK_DAY_NAME
Description: The English name of the day.
A template function is a transformation: it modifies the way macros or name-value pairs are expanded. Template functions can be used in template definitions, or when macros are used in the configuration of syslog-ng PE. Template functions use the following syntax:
$(function-name parameter1 parameter2 parameter3 ...)
For example, the $(echo) template function simply returns the value of the macro it receives as a parameter, thus $(echo ${HOST}) is equivalent to ${HOST}.
The parameters of template functions are separated by a whitespace character. A template function can have maximum 64 parameters. If you want to use a longer string or multiple macros as a single parameter, enclose the parameter in double-quotes or apostrophes. For example:
$(echo "${HOST} ${PROGRAM} ${PID}")
Template functions can be nested into each other, so the parameter of a template function can be another template function, like:
$(echo $(echo ${HOST}))
For details on the available template functions, see the descriptions of the individual template functions in Template functions of syslog-ng PE.
You can define your own template function as a regular configuration object (for example, to reuse the same function in different places in your configuration).
Declaration
template-function <name-of-the-template-function> "<template-expression-using-strings-macros-template-functions>";
Example: Using custom template functions
The following template function can be used to reformat the message. It adds the length of the message to the message template.
template-function my-template-function "${ISODATE} ${HOST} message-length=$(length "${MSG}") ${MSG}";
destination d_file {
file("/tmp/mylogs.log" template("$(my-template-function)\n")); };
You can also refer to existing templates in your template function.
template my-custom-header-template "${ISODATE} ${HOST_FROM} ${MSGHDR}";
template-function my-template-function "$(my-custom-header-template) message-length=$(length "${MSG}") ${MSG}";
The following template functions are available in syslog-ng PE.
base64-encode
Syntax:
$(base64-encode argument)
Description: You can use the base64-encode template function to base64-encode strings and macros. The template function can receive multiple parameters (maximum 64). In this case, syslog-ng PE joins the parameters into a single string and encodes this string. For example, $(base64-encode string1 string2) is equivalent to $(base64-encode string1string2).
Available in syslog-ng PE version 7.0.11 and later.
basename
Syntax:
$(basename argument)
Description: Returns the filename from an argument (for example, a macro) that contains a filename with a path. For example, $(basename "/var/log/messages.log") returns messages.log. To extract the path, use the dirname template function.
Available in syslog-ng PE version 7.0.3 and later.
dirname
Syntax:
$(dirname argument)
Description: Returns the path (without the filename) from an argument (for example, a macro) that contains a filename with a path. For example, $(dirname "/var/log/messages.log") returns /var/log path. To extract the filename, use the basename template function.
Available in syslog-ng PE version 7.0.3 and later.
echo
Syntax:
$(echo argument)
Description: Returns the value of its argument. Using $(echo ${HOST}) is equivalent to ${HOST}.
env
Syntax:
$(env <environment-variable>)
Description: Returns the value of the specified environment variable. Available in syslog-ng PE7 and later.
format-cef-extension
syslog-ng PE includes a template function (format-cef-extension) to format name-value pairs as ArcSight Common Event Format extensions. Note that the template function only formats the selected name-value pairs, it does not provide any mapping. There is no special support for creating the prefix part of a Common Event Format (CEF) message. Note that the order of the elements is random. For details on the CEF extension escaping rules format, see the ArcSight Common Event Format.
You can use the value-pairs that syslog-ng PE stores about the log message as CEF fields. Using value-pairs, you can:
-
select which value-pairs to use as CEF fields,
-
add custom value-pairs as CEF fields,
-
rename value-pairs, and so on.
For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-* template functions is different from the syntax of value-pairs(): these template functions use a syntax similar to command lines.
Using the format-cef-extension template function, has the following prerequisites:
-
Load the the cef module in your configuration:
@module cef
-
Set the on-error global option to drop-property, otherwise if the name of a name-value pair includes an invalid character, syslog-ng PE drops the entire message. (Key name in CEF extensions can contain only the A-Z, a-z and 0-9 characters.)
options {
on-error("drop-property");
};
-
The log messages must be encoded in UTF-8. Use the encoding() option or the validate-utf8 flag in the message source.
Example: Using the format-cef-extension template function
The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).
$(format-cef-extension --scope syslog,all_macros,selected_macros \
--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
--pair MSGHDR=\"$PROGRAM[$PID]: \")
The following example selects every value-pair that has a name beginning with .cef., but removes the .cef. prefix from the key names.
template("$(format-cef-extension --subkeys .cef.)\n")
The following example shows how to use this template function to store log messages in CEF format:
destination d_cef_extension {
file("/var/log/messages.cef" template("${ISODATE} ${HOST} $(format-cef-extension --scope selected_macros --scope nv_pairs)\n"));
};
format-cim
Syntax:
$(format-cim)
Description: Formats the message into Splunk Common Information Model (CIM) format. Applications that can receive messages in CIM format include Kibana, logstash, and Splunk. Applications that can be configured to log into CIM format include nflog and the Suricata IDS engine.
destination d_cim {
network("192.168.1.1" template("$(format-cim)\n"));
};
You can find the exact source of this template function in the syslog-ng PE GitHub repository.
NOTE: To use the format-cim() template function, syslog-ng PE must be compiled with JSON support. To see if your syslog-ng PE binary was compiled with JSON support, execute the syslog-ng --version command.
format-ewmm
Syntax:
$(format-ewmm)
Description: The format-ewmm template function converts the message into the Enterprise-wide message model (EWMM) format. Available in version 7.0.9 and later.
The following is a sample log message in EWMM format.
<13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
{"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
/dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
format-flat-json
Syntax:
$(format-flat-json parameters)
Description: The format-flat-json template function is identical to the format-json template function, but nested JSON objects are flattened in the output. If you have to forward your log messages in JSON format, but the receiving application cannot handle nested JSON objects, use the format-flat-json template function.
Example: Flattened JSON output
The following example shows the difference between nested and flattened JSON objects.
-
The output of $(format-json a.b.c=1) is a nested JSON object (whitespace added for better readability):
{
"a": {
"b": {
"c": "1"
}
}
}
-
The output of $(format-flat-json a.b.c=1) is a flattened JSON object (whitespace added for better readability):
{
"a.b.c": "1"
}
For details on formatting log messages into JSON format, see .
format-json
Syntax:
$(format-json parameters)
Description: The format-json template function receives value-pairs as parameters and converts them into JavaScript Object Notation (JSON) format. Including the template function in a message template allows you to store selected information about a log message (that is, its content, macros, or other metadata) in JSON format. Note that the input log message does not have to be in JSON format to use format-json, you can reformat any incoming message as JSON.
You can use the value-pairs that syslog-ng PE stores about the log message as JSON fields. Using value-pairs, you can:
-
select which value-pairs to use as JSON fields,
-
add custom value-pairs as JSON fields,
-
rename value-pairs, and so on.
For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-json is different from the syntax of value-pairs(): format-json uses a syntax similar to command lines.
NOTE: By default, syslog-ng PE handles every message field as a string. For details on how to send selected fields as other types of data (for example, handle the PID as a number), see Specifying data types in value-pairs.
Example: Using the format-json template function
The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).
$(format-json --scope syslog,all_macros,selected_macros \
--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
--pair MSGHDR=\"$PROGRAM[$PID]: \")
The following example shows how to use this template function to store log messages in JSON format:
destination d_json {
file("/var/log/messages.json" template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
};
NOTE: In case of syslog-ng macros starting with a dot (for example, ".SDATA.meta.sequenceID") an empty key name is added at the top level of the JSON structure. You can work around this by adding --shift 1 as a parameter to the template function. For example, in case of ".SDATA.meta.sequenceID", an empty key name is added at the top level of the JSON structure:
{"":
{"SDATA" :
{"meta" :
{"sequenceID": "123"}
}
}
}
format-welf
This template function converts value-pairs into the WebTrends Enhanced Log file Format (WELF). The WELF format is a comma-separated list of name=value elements. Note that the order of the elements is random. If the value contains whitespace, it is enclosed in double-quotes, for example, name="value". For details on the WELF format, see https://www3.trustwave.com/support/kb/article.aspx?id=10899.
To select which value-pairs to convert, use the command-line syntax of the value-pairs() option. For details on selecting value-pairs, see value-pairs().
Example: Using the format-welf() template function
The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).
$(format-welf --scope syslog,all_macros,selected_macros \
--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
--pair MSGHDR=\"$PROGRAM[$PID]: \")
The following example shows how to use this template function to store log messages in WELF format:
destination d_welf {
file("/var/log/messages.welf" template("$(format-welf --scope selected_macros --scope nv_pairs)\n"));
};
geoip2
Syntax:
$(geoip2 --database <path-to-geoip2-database-file>
[ --field "registered_country.names.ru" ] ${HOST})
Description: This template function extracts specific fields from the mmdb database using the --field parameter. If you omit this parameter, it returns the 2-letter country code of any IPv4/IPv6 address or host.
To retrieve additional GeoIP information, see Looking up GeoIP2 data from IP addresses.
Starting with version 7.0.17, syslog-ng PE tries to automatically detect the location of the database. If that is successful, the database() option is not mandatory.
graphite-output
Syntax:
$(graphite-output parameters)
Description: Available in syslog-ng PE7.0 and later. This template function converts value-pairs from the incoming message to the Graphite plain text protocol format. It is ideal to use with the messages generated by the monitor-source plugin (currently available in the syslog-ng incubator project).
For details on selecting value-pairs in syslog-ng PE and for possibilities to specify which information to convert to Graphite plain text protocol format, see Structuring macros, metadata, and other value-pairs. Note that the syntax of graphite-output is different from the syntax of value-pairs(): graphite-output uses a the command-line syntax used in the format-json template function.
Example: Using the graphite-output template function
The following configuration example shows, how to send value-pairs with names starting with "vmstat." to Graphite running on localhost, port 2003:
destination d_graphite {
network( host("localhost") port(2003) template("$(graphite-output --key vmstat.*)"));
};
grep
Syntax:
$(grep condition value-to-select)
Description: The grep template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching message (for example, the ${MESSAGE} field of the message).
Example: Using the grep template function
The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the auth_method name-value pair.
$(grep ("${username}" == "root") ${auth_method})
You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of grep, these will be returned also separated by commas. This can be used for example, to collect the email recipients from postfix messages.
hash
Syntax:
$(<method> [opts] $arg1 $arg2 $arg3...)
Options:
--length N, -l N
Truncate the hash to the first N characters.
Description: Calculates a hash of the string or macro received as argument using the specified hashing method. If you specify multiple arguments, effectively you receive the hash of the first argument salted with the subsequent arguments.
<method> can be one of md5, md4, sha1, sha256, sha512 and "hash", which is equivalent to md5. Macros are expected as arguments, and they are concatenated without the use of additional characters.
This template function can be used for anonymizing sensitive parts of the log message (for example, username) that were parsed out using PatternDB before storing or forwarding the message. This way, the ability of correlating messages along this value is retained.
Also, using this template, quasi-unique IDs can be generated for data, using the --length option. This way, IDs will be shorter than a regular hash, but there is a very small possibility of them not being as unique as a non-truncated hash.
NOTE: These template functions are available only if the tfhash module has been loaded.
By default, syslog-ng PE loads every available module. For details, see Loading modules.
Example: Using the $(hash) template function
The following example calculates the SHA1 hash of the hostname of the message:
$(sha1 $HOST)
The following example calculates the SHA256 hash of the hostname, using the salted string to salt the result:
$(sha1 $HOST salted)
To use shorter hashes, set the --length:
$(sha1 --length 6 $HOST)
To replace the hostname with its hash, use a rewrite rule:
rewrite r_rewrite_hostname{set("$(sha1 $HOST)", value("HOST"));};
Example: Anonymizing IP addresses
The following example replaces every IPv4 address in the MESSAGE part with its SHA-1 hash:
rewrite pseudonymize_ip_addresses_in_message {subst ("((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])[.]){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))", "$(sha1 $0)", value("MESSAGE"));};
if
Syntax:
$(if (<condition>) <true template> <false template>)
Description: Returns the value of the <true template> parameter if the <condition> is true. If the <condition> is false, the value of <false template> is returned.
Example: Using pattern databases and the if template function
The following example returns violation if the username name-value pair of a message is root, and system otherwise.
$(if ("${username}" == "root") "violation" "system")
This can be used to set the class of a message in pattern database rules based on the condition.
<value name="username">$(if ("${username}" == "root") "violation" "system")</value>
Since template functions can be embedded into each other, it is possible to use another template function as the template of the first one. For example, the following expression returns root if the username is root, admin if the username is joe, and normal user otherwise.
<value name="username">
$(if ("${username}" == "root")
"root"
$(if ("${username}" == "joe") "admin" "normal user"))</value>
indent-multi-line
Syntax:
$(indent-multi-line parameter)
Description: This template function makes it possible to write multi-line log messages into a file. The first line is written like a regular message, subsequent lines are indented with a tab, in compliance with RFC822.
Example: Using the indent-multi-line template function
The following example writes multi-line messages into a text file.
destination d_file {
file ("/var/log/messages"
template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n") );
};
ipv4-to-int
Syntax:
$(ipv4-to-int parameter)
Description: Converts the specified IPv4 address to its numeric representation. The numerical value of an IPv4 address is calculated by treating the IP address as a 4-byte hexadecimal value. For example, the 192.168.1.1 address equals to: 192=C0, 168=A8, 1=01, 1=01, or C0A80101, which is 3232235777 in decimal representation.
NOTE: This template function is available only if the convertfuncs module has been loaded.
By default, syslog-ng PE loads every available module. For details, see Loading modules.
List manipulation
The list-* template functions allow you to manipulate comma-separated lists. Such lists represent a simple array type in syslog-ng PE. Note the following about formatting lists:
-
Values are separated by commas, for example, "item1","item2","item3". The single-element list is an element without a comma.
-
You can use shell-like quotation to embed commas, for example, "item1","ite\,m2","item3".
-
Empty values are skipped (except if they are quoted)
These template functions return a well-formed list, properly encoding and quoting all elements. If a template function returns a single element, all quotation is decoded and the value contains the literal value.
Starting with syslog-ng PE version 7.0.15, the following list-related template functions are available. Certain functions allow you to reference an element using its number: note that the list index starts with zero, so the index of the first element is 0, the second element is 1, and so on.
list-append
Syntax:
$(list-append ${list} ${name-value-pair1} ${name-value-pair2} ... )
Description: Returns a list and appends the values of the specified name-value pairs to the end of the list. You can also append elements to an empty list, for example, $(list-append '' 'element-to-add')
list-concat
Syntax:
$(list-concat ${name-value-pair1} ${name-value-pair2} ... )The commas between the parameters are optional.
Description: This template function creates (concatenates) a list of the values it receives as parameter. The values can be single values (for example, ${HOST}) or lists.
For example, the value of the $(list-concat ${HOST}, ${PROGRAM}, ${PID}) is a comma-separated list.
You can concatenate existing lists into a single list using:
$(list-concat ${list1} ${list2})
list-count
Syntax:
$(list-count ${list} )
Description: Returns the number of elements in the list.
list-head
Syntax:
$(list-head ${list} )
Description: Returns the first element of the list, unquoted.
list-nth
Syntax:
$(list-nth <index-number> ${list} )
Description: Returns the nth element of the list, unquoted. Note that the list index starts with zero, so (list-nth 1 ${list} ) returns the second element, and so on.
list-tail
Syntax:
$(list-tail ${list} )
Description: Returns the list without the first element. For example, if the ${mylist} list contains the one, two, three elements, then $(list-tail ${mylist} ) returns two, three.
list-slice
Syntax:
$(list-slice <from>:<to> ${list} )
Description: Returns the specified subset of the list. Note that the list index starts with zero, for example, $(list-slice 1:2 ${list} ) returns the second and third element of the list, and so on.
You can omit the from or to index if you want to start the subset from the beginning or end of the list, for example: 3: returns the list starting with the 4th element, while :3 returns the first four elements.
Negative numbers select an element from the end of the list, for example, -3: returns the last three element of the list.
length
Syntax:
$(length "<macro>")
Description: Returns the length of the macro in characters, for example, the length of the message. For example, the following filter selects messages that are shorter than 16 characters:
f_short {
match ('-', value ("$(if ($(length "${MESSAGE}") <= 16) "-" "+")"));
};
lowercase
Syntax:
$(lowercase "<macro>")
Description: Returns the lowercase version of the specified string or macro. For example, the following example uses the lowercase version of the hostname in a directory name:
destination d_file {
file ("/var/log/${MONTH}/${DAY}/$(lowercase "${HOST}")/messages");
};
Available in syslog-ng PE7.0 and later.
Numerical operations
Syntax:
$(<operation> "<value1>" "<value2>")
Description: These template functions allow you to manipulate numbers, that is, to perform addition (+), substraction (-), multiplication (*), division (/), and modulus (%). All of them require two numeric arguments. The result is NaN (Not-a-Number) if the parameters are not numbers, cannot be parsed, or if a division by zero would occur. For example, to add the value of two macros, use the following template function:
$(+ "${<MACRO1>}" "${<MACRO2>}");
When you are correlating messages and a name-value pair contains numerical values in the messages, you can calculate the lowest (min), highest (max), total (sum), and mean (average) values. These calculations process every message of the correlation context. For details on message correlation, see Correlating log messages. For example, if the messages of the context have a .myfields.load name-value pair, you can find the highest load value using the following template function.
$(max ${.myfields.load})
or
Syntax:
$(or <macro1> <macro2>)
Description: This template function returns the first non-empty argument.
padding
Syntax:
$(padding <macro> <width> <prepended-character-or-string>)
Description: This template function returns the value of its first parameter (a string or macro), prepended with a string. This string is <width> long, and repeats the character or string set in the third parameter. If you use a single character, it is added <width> times. If you use a string, it is repeated until its length reaches <width>. The default padding character is ' ' (space). For example:
Example: Using the padding template function
If the value of the ${MESSAGE} macro is mymessage, then the output of the padding() template function is the following:
$(padding ${MESSAGE} 10 X)
Output: XXXXXXXXXXmymessage
$(padding ${MESSAGE} 10 foo)
Output: foofoofoofmymessage
python
Syntax:
$(python <name-of-the-python-method-to-use> <arguments-of-the-method>)
Description: This template function enables you to write a custom template function in Python. You can define a Python block in your syslog-ng PE configuration file, define one or more Python functions in it, and use the methods as template functions. If you use a Python block, syslog-ng PE embeds a Python interpreter to process the messages.
The following points apply to using Python blocks in syslog-ng PE in general:
-
Only the default Python modules are available (that is, you cannot import external Python modules, and One Identity does not support using external Python modules).
-
The syslog-ng PE application uses its own Python interpreter (shipped with the default syslog-ng PE installation) instead of the system's Python interpreter.
-
The syslog-ng PE application is shipped with Python version 3.8.
-
The Python block must be a top-level block in the syslog-ng PE configuration file.
-
If you store the Python code in a separate Python file and only include it in the syslog-ng PE configuration file, make sure that the PYTHON_PATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start syslog-ng PE manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc
In production, when syslog-ng PE starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting syslog-ng PE. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc"
-
The Python object is initiated every time when syslog-ng PE is started or reloaded.
|
|
Caution:
If you reload syslog-ng PE, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of syslog-ng PE typically involves a reload. |
-
The Python block can contain multiple Python functions.
-
Using Python code in syslog-ng PE can significantly decrease the performance of syslog-ng PE, especially if the Python code is slow. In general, the features of syslog-ng PE are implemented in C, and are faster than implementations of the same or similar features in Python.
-
Validate and lint the Python code before using it. The syslog-ng PE application does not do any of this.
-
Python error messages are available in the internal() source of syslog-ng PE.
-
You can access the name-value pairs of syslog-ng PE directly through a message object or a dictionary.
-
To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of syslog-ng PE. For details, see Logging from your Python code.
-
Support disclaimer
|
|
Caution:
This is a Preview Feature, which provides an insight to planned enhancements to functionality in the product. Consider this Preview Feature a work in progress, as it may not represent the final design and functionality.
This feature has completed QA release testing, but its full impact on production systems has not been determined yet, and potential future changes in functionality and the user interface may result in compatibility issues in your current settings.
One Identity recommends the following:
- Consider the potential risks when using this functionality in a production environment.
- Consider the Support Policy on Product Preview Features before using this functionality in a production environment.
- Closely and regularly keep track of official One Identity announcements about potential changes in functionality and the user interface. If these potential changes affect your configuration, check the changes you have to make in your configuration, otherwise your syslog-ng PE application may not start after upgrade.
- Always perform tests prior to upgrades in order to avoid the risks mentioned.
However, you are welcome to try this feature and if you have any feedback, Contact One Identity.
Support Policy on Product Preview Features
The One Identity Support Team will:
-
Accept and review each service request opened regarding a Preview Feature.
-
Consider all service requests relating to a Preview Features as severity level 3.
-
Provide best effort support to resolve any issues relating to a Preview Feature.
-
Work with customers to log any product defects or enhancements relating to Preview Features.
-
Not accept requests for escalations regarding Preview Features.
-
Not provide after-hours support for Preview Features. |
Using Python in syslog-ng PE is recommended only if you are familiar with both Python and syslog-ng PE. One Identity is not responsible for the quality, resource requirements, or any bugs in the Python code, nor any syslog-ng PE crashes, message losses, or any other damage caused by the improper use of this feature, unless explicitly stated in a contract with One Identity.
The following points apply to Python parsers.
-
The first argument in the definition of the Python function is the actual log message. This is implicitly passed to the function, you do not have to use it in the template function.
-
The value of the template function is return value of the Python function.
-
To reference a name-value pair or a macro in the Python code, use the following format. For example, if the first argument in the definition of the function is called log-message, the value of the HOST macro is log-message['HOST'], and so on. (The log-message contains the entire log message (not just the text body) in a structure similar to a Python dict, but it is actually an object.)
-
You can define new name-value pairs in the Python function. For example, if the first argument in the definition of the function is called log-message, you can create a new name-value pair like this: log_message["new-macro-name"]="value". This is useful when you parse a part of the message from Python, or lookup a value based on data extracted from the log message.
Note that the names of the name-value pairs are case-sensitive. If you create a new name-value pair called new-macro-name in Python, and want to reference it in another part of the syslog-ng PE configuration file (for example, in a template), use the ${new-macro-name} macro.
-
You cannot override hard macros (see Hard versus soft macros).
-
To list all available keys (names of name-value pairs), use the log_message.keys() function.
Declaration
python {
def <name_of_the_python_function>(<log_message>, <optional_other_arguments>):
# <your-python-code>
return <value_of_the_template_function>
};
template <template-name> {
template($(python <name_of_the_python_function>));
};
Example: Writing template functions in Python
The following example creates a Python template function called return_message that returns the MESSAGE part of the log message.
@version: 7.0
python {
def return_message(log_message):
return log_message['MESSAGE']
};
destination d_local {
file("/tmp/logs.txt" template("[$(python return_message)]\n"));
};
The following example creates a Python template function called resolve_host that receives an IP address as an argument, and attempts to resolve it into a hostname.
@version: 7.0
python {
import socket
def resolve_host(log_message, hostname):
try:
return socket.gethostbyaddr(hostname)[0]
except (socket.herror, socket.error):
return 'unknown'
};
destination d_local {
file("/tmp/logs.txt" template("${ISODATE} $(python resolve_host ${SOURCE_IP}) ${MESSAGE}\n"));
};
replace-delimiter
Syntax:
$(replace-delimiter "<old-delimiters>" "<new-delimiter>" "<macro>")
Description: Replaces the delimiter character with a new one. For example, the following example replaces the tabulators (\t) in the message with semicolons (;):
$(replace-delimiter "\t" ";" "${MESSAGE}")
Available in syslog-ng PE7.0 and later.
sanitize
Syntax:
$(sanitize <options> "<macro1>" "<macro2> ...")
Description: This file replaces the special characters in macro values, for example, it can replace the slash (/) characters in a filename with the underscore (_) character. If you specify multiple arguments, they will be concatenated using the / character, so they can be used as separate directory levels when used in filenames.
The function has the following options:
- --ctrl-chars or -c
-
Filter control characters (characters that have an ASCII code of 32 or lower). This option is used by default.
- --invalid-chars <characterlist> or -i <characterlist>
-
The list of characters to be replaced with underscores (_). The default list contains the / character. The following example replaces the \ and @ characters, so for example, fo\o@bar becomes foobar:
$(sanitize -i \@ $PROGRAM)
- --no-ctrl-chars or -C
-
Do not filter the control characters (characters that have an ASCII code of 32 or lower).
- --replacement <replacement-character> or -r <replacement-character>
-
The character used to replace invalid characters. By default, this is the underscore (_). The following example replaces invalid characters with colons instead of underscores, so for example, foo/bar becomes foo;bar:
$(sanitize -r ; $PROGRAM)
Example: Using the sanitize template function
The following example uses the sanitize function on two macros, and the results are used as directory names in a file destination.
file("/var/log/$(sanitize $HOST $PROGRAM)/messages");
This is equivalent to file("/var/log/$HOST/$PROGRAM/messages");, but any slashes in the values of the $HOST and $PROGRAM macros are replaced with underscores.
strip
Syntax:
$(strip "<macro>")
Description: Deletes whitespaces from the beginning and the end of a macro. You can specify multiple macros separated with whitespace in a single template function, for example:
$(strip "${MESSAGE}" "${PROGRAM}")
substr
Syntax:
$(substr "<argument>" "<offset>" "<length>")
Description: This function extracts a substring of a string.
- argument
-
The string to extract the substring from, for example, "${MESSAGE}"
- offset
-
Specifies where the substring begins (in characters). 0 means to start from the beginning of the string, 5 means to skip the first 5 characters of the string, and so on. Use negative numbers to specify where to start from the end of the string, for example, -1 means the last character, -5 means to start five characters before the end of the string.
- length
-
(Optional parameter): The number of characters to extract. If not specified, the substring will be extracted from the offset to the end of the string. Use negative numbers to stop the substring before the end of the string, for example, -5 means the substring ends five characters before the end of the string.
Example: Using the substr template function
Skip the first 15 characters of the message, and select the rest:
$(substr "${MESSAGE}" "15");
Select characters 16-30 of the message (15 characters with offset 15):
$(substr "${MESSAGE}" "15" "15");
Select the last 15 characters of the message:
$(substr "${MESSAGE}" "-15");
A template that converts the message to RFC3164 (BSD-syslog) format and truncates the messages to 1023 characters:
template t_truncate_messages {
template("$(substr \"<$PRI>$DATE $HOST $MSGHDR$MESSAGE\" \"0\" \"1023\")\n");
template-escape(no);
};
uppercase
Syntax:
$(uppercase "<macro>")
Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:
destination d_file {
file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
};
Available in syslog-ng PE7.0 and later.
url-decode
Syntax:
$(url-decode <string-pr-macro-1> <string-pr-macro-2> ... )
Description: You can use the url-decode template function to decode url-encoded strings and macros. For example, $(url-decode %3C%3E) yields <>. The url-decode can receive multiple parameters (maximum 64). In this case, each parameter is decoded separately, and simply concatenated.
Available in syslog-ng PE version 7.0.11 and later.
url-encode
Syntax:
$(url-encode ${MESSAGE} )\n")
Description: You can use the url-encode template function together with the telegram() destination to send syslog messages to Telegram. The url-encode template function escapes strings. All input characters that are not a-z, A-Z, 0-9, '-', '.', '_' or '~' are converted to their "URL escaped" version.
Available in syslog-ng PE version 7.0.11 and later.
uuid
Syntax:
$(uuid)
Description: Generates a Universally Unique IDentifier (UUID) that complies with RFC4122. That way, an UUID can be added to the message soon after it is received, so messages stored in multiple destinations can be identified. For example, when storing messages in a database and also in files, the UUID can be used to find a particular message both in the database and the files.
To generate a UUID, you can use a rewrite rule to create a new value-pair for the message.
Example: Using Universally Unique Identifiers
The following example adds a value-pair called MESSAGE_UUID to the message using a rewrite rule and a template.
rewrite r_add_uuid { set("$(uuid)" value("MESSAGE_UUID")); };
destination d_file {
file ("/var/log/messages"
template("$MESSAGE_UUID $ISODATE $HOST $MSG\n")
template-escape(no)
);
};
log { source(s_network);
rewrite(r_add_uuid);
destination(d_file);
};
NOTE: This template function is available only if the tfuuid module has been loaded.
By default, syslog-ng PE loads every available module. For details, see Loading modules.
Macros, templates, and template functions allow you to fully customize the format of the message. This flexibility makes it possible to use syslog-ng PE in some unexpected way if needed, for example, to emulate simple, plain-text protocols. The following example shows you how to send LPUSH commands to a Redis server.
NOTE: The purpose of this example is to demonstrate the flexibility of syslog-ng PE. A dedicated Redis destination is available in syslog-ng PE version 3.5. For details, see Storing name-value pairs in Redis.
The following template is a valid LPUSH command in accordance with the Redis protocol, and puts the $MESSAGE into a separate list for every $PROGRAM:
template t_redis_lpush {
template("*3\r\n$$5\r\nLPUSH\r\n$$$(length ${PROGRAM})\r\n${PROGRAM}\r\n$$$(length ${MESSAGE})\r\n${MESSAGE}\r\n");
};
If you use this template in a network() destination, syslog-ng PE formats the message according to the template, and sends it to the Redis server.
destination d_redis_tcp {
network("127.0.0.1" port(6379) template(t_redis_lpush));
};
