syslog-ng Open Source Edition 3.16 - Administration Guide

AMPM

Description: Typically used together with the ${HOUR12} macro, ${AMPM} returns the period of the day: AM for hours before mid day and PM for hours after mid day. In reference to a 24-hour clock format, AM is between 00:00-12:00 and PM is between 12:00-24:00. 12AM is midnight. Available in syslog-ng OSE 3.4 and later.

BSDTAG

Description: Facility/priority information in the format used by the FreeBSD syslogd: a priority number followed by a letter that indicates the facility. The priority number can range from 0 to 7. The facility letter can range from A to Y, where A corresponds to facility number zero (LOG_KERN), B corresponds to facility 1 (LOG_USER), and so on.

Custom macros

Description: CSV parsers and pattern databases can also define macros from the content of the messages, for example, a pattern database rule can extract the username from a login message and create a macro that references the username. For details on using custom macros created with CSV parsers and pattern databases, see parser: Parse and segment structured messages and Using parser results in filters and templates, respectively.

FACILITY

Description: The name of the facility (for example, kern) that sent the message.

FACILITY_NUM

Description: The numerical code of the facility (for example, 0) that sent the message.

FILE_NAME

Description: Name of the log file (including its path) from where syslog-ng OSE received the message (only available if syslog-ng OSE received the message from a file or a wildcard-file source). If you need only the path or the filename, use the dirname and basename template functions.

FULLHOST

Description: The name of the source host where the message originates from.

• If the message traverses several hosts and the chain-hostnames() option is on, the first host in the chain is used.

• If the keep-hostname() option is disabled (keep-hostname(no)), the value of the $FULLHOST macro will be the DNS hostname of the host that sent the message to syslog-ng OSE (that is, the DNS hostname of the last hop). In this case the $FULLHOST and $FULLHOST_FROM macros will have the same value.

• If the keep-hostname() option is enabled (keep-hostname(yes)), the value of the $FULLHOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server.

  NOTE: The use-dns(), use-fqdn(), normalize-hostnames(), and dns-cache() options will have no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.

For details on using name resolution in syslog-ng OSE, see Using name resolution in syslog-ng.

FULLHOST_FROM

Description: The FQDN of the host that sent the message to syslog-ng as resolved by syslog-ng using DNS. If the message traverses several hosts, this is the last host in the chain.

The syslog-ng OSE application uses the following procedure to determine the value of the $FULLHOST_FROM macro:

1. The syslog-ng OSE application takes the IP address of the host sending the message.

2. If the use-dns() option is enabled, syslog-ng OSE attempts to resolve the IP address to a hostname. If it succeeds, the returned hostname will be the value of the $FULLHOST_FROM macro. This value will be the FQDN of the host if the use-fqdn() option is enabled, but only the hostname if use-fqdn() is disabled.

3. If the use-dns() option is disabled, or the address resolution fails, the ${FULLHOST_FROM} macro will return the IP address of the sender host.

For details on using name resolution in syslog-ng OSE, see Using name resolution in syslog-ng.

HOUR12, C_HOUR12, R_HOUR12, S_HOUR12

Description: The hour of day the message was sent in 12-hour clock format. See also the ${AMPM} macro. 12AM is midnight. Available in syslog-ng OSE 3.4 and later.

HOST

Description: The name of the source host where the message originates from.

• If the message traverses several hosts and the chain-hostnames() option is on, the first host in the chain is used.

• If the keep-hostname() option is disabled (keep-hostname(no)), the value of the $HOST macro will be the DNS hostname of the host that sent the message to syslog-ng OSE (that is, the DNS hostname of the last hop). In this case the $HOST and $HOST_FROM macros will have the same value.

• If the keep-hostname() option is enabled (keep-hostname(yes)), the value of the $HOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server.

  NOTE: The use-dns(), use-fqdn(), normalize-hostnames(), and dns-cache() options will have no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.

For details on using name resolution in syslog-ng OSE, see Using name resolution in syslog-ng.

HOST_FROM

Description: The FQDN of the host that sent the message to syslog-ng as resolved by syslog-ng using DNS. If the message traverses several hosts, this is the last host in the chain.

The syslog-ng OSE application uses the following procedure to determine the value of the $HOST_FROM macro:

1. The syslog-ng OSE application takes the IP address of the host sending the message.

2. If the use-dns() option is enabled, syslog-ng OSE attempts to resolve the IP address to a hostname. If it succeeds, the returned hostname will be the value of the $HOST_FROM macro. This value will be the FQDN of the host if the use-fqdn() option is enabled, but only the hostname if use-fqdn() is disabled.

3. If the use-dns() option is disabled, or the address resolution fails, the ${HOST_FROM} macro will return the IP address of the sender host.

For details on using name resolution in syslog-ng OSE, see Using name resolution in syslog-ng.

ISODATE, C_ISODATE, R_ISODATE, S_ISODATE

Description: Date of the message in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00. If possible, it is recommended to use ${ISODATE} for timestamping. Note that syslog-ng can produce fractions of a second (for example milliseconds) in the timestamp by using the frac-digits() global or per-destination option.

LEVEL_NUM

Description: The priority (also called severity) of the message, represented as a numeric value, for example, 3. For the textual representation of this value, use the ${LEVEL} macro. See PRIORITY or LEVEL for details.

LOGHOST

Description: The hostname of the computer running syslog-ng OSE — it returns the same result as the hostname command.

MESSAGE

Description: Text contents of the log message without the program name and pid. The program name and the pid together are available in the ${MSGHDR} macro, and separately in the ${PROGRAM} and ${PID} macros.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, syslog-ng OSE generates a new syslog header (timestamp, host, and so on) automatically. Note that since flags(no-parse) disables message parsing, it interferes with other flags, for example, disables flags(no-multi-line).

The ${MSG} macro is an alias of the ${MESSAGE} macro: using ${MSG} in syslog-ng OSE is equivalent to ${MESSAGE}.

Note that before syslog-ng version 3.0, the ${MESSAGE} macro included the program name and the pid. In syslog-ng 3.0, the ${MESSAGE} macro became equivalent with the ${MSGONLY} macro.

MONTH_ABBREV, C_MONTH_ABBREV, R_MONTH_ABBREV, S_MONTH_ABBREV

Description: The English abbreviation of the month name (3 letters).

MONTH_NAME, C_MONTH_NAME, R_MONTH_NAME, S_MONTH_NAME

Description: The English name of the month name.

MONTH_WEEK, C_MONTH_WEEK, R_MONTH_WEEK, S_MONTH_WEEK

Description: The number of the week in the given month (0-5). The week with numerical value 1 is the first week containing a Monday. The days of month before the first Monday are considered week 0. For example, if a 31-day month begins on a Sunday, then the 1st of the month is week 0, and the end of the month (the 30th and 31st) is week 5.

MSEC, C_MSEC, R_MSEC, S_MSEC

Description: The millisecond the message was sent.

Available in syslog-ng OSE version 3.4 and later.

MSG

The ${MSG} macro is an alias of the ${MESSAGE} macro, using ${MSG} in syslog-ng OSE is equivalent to ${MESSAGE}. For details on this macro, see MESSAGE.

MSGID

Description: A string specifying the type of the message in IETF-syslog (RFC5424-formatted) messages. For example, a firewall might use the ${MSGID} "TCPIN" for incoming TCP traffic and the ${MSGID} "TCPOUT" for outgoing TCP traffic. By default, syslog-ng OSE does not specify this value, but uses a dash (-) character instead. If an incoming message includes the ${MSGID} value, it is retained and relayed without modification.

MSGONLY

Description: Message contents without the program name or pid. Starting with syslog-ng OSE 3.0, the following macros are equivalent: ${MSGONLY}, ${MSG}, ${MESSAGE}. For consistency, use the ${MESSAGE} macro. For details, see MESSAGE.

PID

Description: The PID of the program sending the message.

PRI

Description: The priority and facility encoded as a 2 or 3 digit decimal number as it is present in syslog messages.

PRIORITY or LEVEL

Description: The priority (also called severity) of the message, for example, error. For the textual representation of this value, use the ${LEVEL} macro. See PRIORITY or LEVEL for details.

PROGRAM

Description: The name of the program sending the message. Note that the content of the ${PROGRAM} variable may not be completely trusted as it is provided by the client program that constructed the message.

RAWMSG

Description: The original message as received from the client. Note that this macro is available only in 3.16 and later, and only if syslog-ng received the message using the default-network-drivers-ng() source, or the source receiving the message has the store-raw-message flag set.

RCPTID

Description: When the use-rcptid global option is set to yes, syslog-ng OSE automatically assigns a unique reception ID to every received message. You can access this ID and use it in templates via the ${RCPTID} macro. The reception ID is a monotonously increasing 48-bit integer number, that can never be zero (if the counter overflows, it restarts with 1).

RUNID

Description: An ID that changes its value every time syslog-ng OSE is restarted, but not when reloaded.

SDATA, .SDATA.SDID.SDNAME

Description: The syslog-ng application automatically parses the STRUCTURED-DATA part of IETF-syslog messages, which can be referenced in macros. The ${SDATA} macro references the entire STRUCTURED-DATA part of the message, while structured data elements can be referenced using the ${.SDATA.SDID.SDNAME} macro.

SEQNUM

Description: The ${SEQNUM} macro contains a sequence number for the log message. The value of the macro depends on the scenario, and can be one of the following:

• If syslog-ng OSE receives a message via the IETF-syslog protocol that includes a sequence ID, this ID is automatically available in the ${SEQNUM} macro.

• If the message is a Cisco IOS log message using the extended timestamp format, then syslog-ng OSE stores the sequence number from the message in this macro. If you forward this message the IETF-syslog protocol, syslog-ng OSE includes the sequence number received from the Cisco device in the ${.SDATA.meta.sequenceId} part of the message.

  NOTE: To enable sequence numbering of log messages on Cisco devices, use the following command on the device (available in IOS 10.0 and later): service sequence-numbers. For details, see the manual of your Cisco device.

• For locally generated messages (that is, for messages that are received from a local source, and not from the network), syslog-ng OSE calculates a sequence number when sending the message to a destination (it is not calculated for relayed messages).

  • The sequence number is not global, but per-destination. Essentially, it counts the number of messages sent to the destination.

  • This sequence number increases by one for every message sent to the destination. It not lost when syslog-ng OSE is reloaded, but it is reset when syslog-ng OSE is restarted.

  • This sequence number is added to every message that uses the IETF-syslog protocol (${.SDATA.meta.sequenceId}), and can be added to BSD-syslog messages using the ${SEQNUM} macro.

SOURCE

Description: The identifier of the source statement in the syslog-ng OSE configuration file that received the message. For example, if syslog-ng OSE received the log message from the source s_local { internal(); }; source statement, the value of the ${SOURCE} macro is s_local. This macro is mainly useful for debugging and troubleshooting purposes.

SOURCEIP

Description: IP address of the host that sent the message to syslog-ng. (That is, the IP address of the host in the ${FULLHOST_FROM} macro.) Please note that when a message traverses several relays, this macro contains the IP of the last relay.

STAMP, R_STAMP, S_STAMP

Description: A timestamp formatted according to the ts-format() global or per-destination option.

SYSUPTIME

Description: The time elapsed since the syslog-ng OSE instance was started (that is, the uptime of the syslog-ng OSE process). The value of this macro is an integer containing the time in 1/100th of the second.

Available in syslog-ng OSE version 3.4 and later.

TAG

Description: The priority and facility encoded as a 2 digit hexadecimal number.

TAGS

Description: A comma-separated list of the tags assigned to the message.

.TLS.X509

Description: When using a transport that uses TLS, these macros contain information about the peer's certificate. That way, you can use information from the client certificate in filenames, database values, or as other metadata. If you clients have their own certificates, then these values are unique per client, but unchangeable by the client. The following macros are available in syslog-ng OSE version 3.9 and later.

• .TLS.X509_CN: The Common Name of the certificate.

• .TLS.X509_O: The value of the Organization field.

• .TLS.X509_OU: The value of the Organization Unit field.

UNIQID

Description: A globally unique ID generated from the HOSTID and the RCPTID in the format of HOSTID@RCPTID. For details, see use-uniqid() and RCPTID.

Available in syslog-ng OSE version 3.7 and later.

USEC, C_USEC, R_USEC, S_USEC

Description: The microsecond the message was sent.

Available in syslog-ng OSE version 3.4 and later.

WEEK_DAY_ABBREV, C_WEEK_DAY_ABBREV, R_WEEK_DAY_ABBREV, S_WEEK_DAY_ABBREV

Description: The 3-letter English abbreviation of the name of the day the message was sent, for example Thu.

WEEK_DAY, C_WEEK_DAY, R_WEEK_DAY, S_WEEK_DAY

Description: The day of the week as a numerical value (1-7).

WEEKDAY, C_WEEKDAY, R_WEEKDAY, S_WEEKDAY

Description: These macros are deprecated, use ${WEEK_DAY_ABBREV}, ${R_WEEK_DAY_ABBREV}, ${S_WEEK_DAY_ABBREV} instead. The 3-letter name of the day of week the message was sent, for example Thu.

WEEK_DAY_NAME, C_WEEK_DAY_NAME, R_WEEK_DAY_NAME, S_WEEK_DAY_NAME

Description: The English name of the day.

Declaration:

template-function <name-of-the-template-function> "<template-expression-using-strings-macros-template-functions>";

basename

Syntax:

$(basename argument)

Description: Returns the filename from an argument (for example, a macro: $(basename ${FILE_NAME})) that contains a filename with a path. For example, $(basename "/var/log/messages.log") returns messages.log. To extract the path, use the dirname template function.

Available in syslog-ng OSE version 3.10 and later.

context-lookup

Syntax:

$(context-lookup [option] condition value-to-select)

Description: The context-lookup template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching messages (for example, the ${MESSAGE}) as a list. It works similarly to the $(grep) template function, but it escapes its output properly, so that the returned value is a list that can be processed with other template functions that work on lists, for example, $(list-slice).

$(context-lookup ("${username}" == "root") ${tags})

To limit the number of matches that the template function returns, use the --max-count option, for example, $(context-lookup --max-count 5 ("${username}" == "root") ${tags}). If you do not want to limit the number of matches, use --max-count 0.

You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of context-lookup, these will be returned also separated by commas. This can be used for example to collect the e-mail recipients from postfix messages.

Available in syslog-ng OSE version 3.10 and later.

context-values

Syntax:

$(context-values $name-value1 $name-value2 ...)

Description: The context-values template function returns a list of every occurrence of the specified name-value pairs from the entire context. For example, if the context contains multiple messages, the $(context-values ${HOST}) template function will return a comma-separated list of the ${HOST} values that appear in the context.

Available in syslog-ng OSE version 3.10 and later.

dirname

Syntax:

$(dirname argument)

Description: Returns the path (without the filename) from an argument (for example, a macro: $(basename ${FILE_NAME}) that contains a filename with a path. For example, $(dirname "/var/log/messages.log") returns /var/log path. To extract the filename, use the basename template function.

Available in syslog-ng OSE version 3.10 and later.

echo

Syntax:

$(echo argument)

Description: Returns the value of its argument. Using $(echo ${HOST}) is equivalent to ${HOST}.

env

Syntax:

$(env <environment-variable>)

Description: Returns the value of the specified environment variable. Available in syslog-ng OSE 3.5 and later.

format-cef-extension

syslog-ng OSE version 3.8 includes a new template function (format-cef-extension) to format name-value pairs as ArcSight Common Event Format extensions. Note that the template function only formats the selected name-value pairs, it does not provide any mapping. There is no special support for creating the prefix part of a Common Event Format (CEF) message. Note that the order of the elements is random. For details on the CEF extension escaping rules format, see the ArcSight Common Event Format.

You can use the value-pairs that syslog-ng OSE stores about the log message as CEF fields. Using value-pairs, you can:

• select which value-pairs to use as CEF fields,

• add custom value-pairs as CEF fields,

• rename value-pairs, and so on.

For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-* template functions is different from the syntax of value-pairs(): these template functions use a syntax similar to command lines.

Using the format-cef-extension template function, has the following prerequisites:

• Load the the cef module in your configuration:

  @module cef

• Set the on-error global option to drop-property, otherwise if the name of a name-value pair includes an invalid character, syslog-ng OSE drops the entire message. (Key name in CEF extensions can contain only the A-Z, a-z and 0-9 characters.)

  options {
     on-error("drop-property");
  };

• The log messages must be encoded in UTF-8. Use the encoding() option or the validate-utf8 flag in the message source.

$(format-cef-extension --scope syslog,all_macros,selected_macros \
  --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
  --pair MSGHDR=\"$PROGRAM[$PID]: \")

template("$(format-cef-extension --subkeys .cef.)\n")

destination d_cef_extension {
    file("/var/log/messages.cef" template("${ISODATE} ${HOST} $(format-cef-extension --scope selected_macros --scope nv_pairs)\n"));
};

format-cim

Syntax:

$(format-cim)

Description: Formats the message into Splunk Common Information Model (CIM) format. Applications that can receive messages in CIM format include Kibana, logstash, and Splunk. Applications that can be configured to log into CIM format include nflog and the Suricata IDS engine.

destination d_cim {
    network(
        "192.168.1.1"
        template("$(format-cim)\n")
    );
};

You can find the exact source of this template function in the syslog-ng OSE GitHub repository.

format-ewmm

Syntax:

$(format-ewmm)

Description: The format-ewmm template function converts the message into the Enterprise-wide message model (EWMM) format. Available in version 3.16 and later.

The follwing is a sample log message in EWMM format.

<13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
{"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
/dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}

format-gelf

Syntax:

$(format-gelf)

Description: Available in syslog-ng OSE 3.13 and later.

You can use the Graylog Extended Log Format (GELF) template together with the graylog2() destination to send syslog messages to Graylog. GELF is the native data format of Graylog.

destination graylog2 { 
    network(
        "127.0.0.1"
        port(12201)
        transport(tcp)
        template("$(format-gelf)")
    );
};

format-json

Syntax:

$(format-json parameters)

Description: The format-json template function receives value-pairs as parameters and converts them into JavaScript Object Notation (JSON) format. Including the template function in a message template allows you to store selected information about a log message (that is, its content, macros, or other metadata) in JSON format. Note that the input log message does not have to be in JSON format to use format-json, you can reformat any incoming message as JSON.

You can use the value-pairs that syslog-ng OSE stores about the log message as JSON fields. Using value-pairs, you can:

• select which value-pairs to use as JSON fields,

• add custom value-pairs as JSON fields,

• rename value-pairs, and so on.

For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-json is different from the syntax of value-pairs(): format-json uses a syntax similar to command lines.

$(format-json --scope syslog,all_macros,selected_macros \
  --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
  --pair MSGHDR=\"$PROGRAM[$PID]: \")

destination d_json {
    file(
        "/var/log/messages.json"
        template("$(format-json --scope selected_macros --scope nv_pairs)\n")
    );
};

format-welf

This template function converts value-pairs into the WebTrends Enhanced Log file Format (WELF). The WELF format is a comma-separated list of name=value elements. Note that the order of the elements is random. If the value contains whitespace, it is enclosed in double-quotes, for example, name="value". For details on the WELF format, see https://www3.trustwave.com/support/kb/article.aspx?id=10899.

To select which value-pairs to convert, use the command-line syntax of the value-pairs() option. For details on selecting value-pairs, see value-pairs().

$(format-welf --scope syslog,all_macros,selected_macros \
  --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
  --pair MSGHDR=\"$PROGRAM[$PID]: \")

destination d_welf {
    file(
        "/var/log/messages.welf"
        template("$(format-welf --scope selected_macros --scope nv_pairs)\n")
    );
};

geoip (DEPRECATED)

This template function is deprecated. Use geoip2 instead.

Syntax:

$(geoip <IPv4-address>)

Description: This template function returns the 2-letter country code of any IPv4 address or host. IPv6 addresses are not supported. Currently only the 2-letter codes are supported, and only from the default database. For example, $(geoip $HOST)

To retrieve additional GeoIP information, see Looking up GeoIP data from IP addresses (DEPRECATED).

geoip2

Syntax:

$(geoip2 --database <path-to-geoip2-database-file>
    [ --field "registered_country.names.ru" ] ${HOST})

Description: This template function exctracts specific fields from the mmdb database using the --field parameter. If you omit this parameter, the it returns the 2-letter country code of any IPv4/IPv6 address or host.

To retrieve additional GeoIP information, see Looking up GeoIP2 data from IP addresses.

getent

Syntax:

$(getent)

Description: Available in syslog-ng OSE 3.13 and later.

You can use the getent template function to look up entries from the Name Service Switch libraries, such as, passwd, services, or protocols.

The following databases are supported:

• passwd

  Use this database to query data related to a user. Specify the user by either username or user ID. You can query the following data: username, user ID, group ID, GECOS field, home directory, or user shell.

  $(getent passwd testuser name)
  $(getent passwd testuser uid)
  $(getent passwd testuser gid)
  $(getent passwd testuser gecos)
  $(getent passwd testuser dir)
  $(getent passwd testuser shell)

  or

  $(getent passwd 1000 name)
  $(getent passwd 1000 uid)
  $(getent passwd 1000 gid)
  $(getent passwd 1000 gecos)
  $(getent passwd 1000 dir)
  $(getent passwd 1000 shell)

  The queried data is optional. When you do not query any data, the default behavior applies, which is as follows: user ID is returned for username, or username is returned for user ID.

  • Username $(getent passwd testuser) returns user ID 1000.

  • User ID $(getent passwd 1000) returns username testuser.

• group

  Use this database to query group-related data. The group can be specified using either group ID or group name. You can query the following data: group name, group ID, and members.

  $(getent group adm name)
  $(getent group adm gid)
  $(getent group adm members)

  The queried data is optional. The default behavior is as follows: group ID is returned for group name, or group name is returned for user ID.

  • Group name $(getent group adm) returns group ID 4.

  • Group ID $(getent group 4) returns group name adm.

• protocols

  Use this database to translate protocol name to protocol ID, or protocol ID to protocol string.

  $(getent protocols tcp)
  $(getent protocols 6)

• services

  Use this database to translate service name to service ID, or service ID to service name.

  $(getent services http)
  $(getent services 80)

graphite-output

Syntax:

$(graphite-output parameters)

Description: Available in syslog-ng OSE 3.6 and later (Originally appeared in the syslog-ng OSE incubator for syslog-ng 3.5). This template function converts value-pairs from the incoming message to the Graphite plain text protocol format. It is ideal to use with the messages generated by the monitor-source plugin (currently available in the syslog-ng incubator project).

For details on selecting value-pairs in syslog-ng OSE and for possibilities to specify which information to convert to Graphite plain text protocol format, see Structuring macros, metadata, and other value-pairs. Note that the syntax of graphite-output is different from the syntax of value-pairs(): graphite-output uses a the command-line syntax used in the format-json template function.

destination d_graphite {
    network( host("localhost") port(2003) template("$(graphite-output --key vmstat.*)"));
};

grep

Syntax:

$(grep condition value-to-select)

Description: The grep template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching message (for example, the ${MESSAGE} field of the message).

$(grep ("${username}" == "root") ${auth_method})

You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of grep, these will be returned also separated by commas. This can be used for example to collect the e-mail recipients from postfix messages.

hash

Syntax:

$(<method> [opts] $arg1 $arg2 $arg3...)

Options:

--length N, -l N

Description: Calculates a hash of the string or macro received as argument using the specified hashing method. If you specify multiple arguments, effectively you receive the hash of the first argument salted with the subsequent arguments.

<method> can be one of md5, md4, sha1, sha256, sha512 and "hash", which is equivalent to md5. Macros are expected as arguments, and they are concatenated without the use of additional characters.

This template function can be used for anonymizing sensitive parts of the log message (for example username) that were parsed out using PatternDB before storing or forwarding the message. This way, the ability of correlating messages along this value is retained.

Also, using this template, quasi-unique IDs can be generated for data, using the --length option. This way, IDs will be shorter than a regular hash, but there is a very small possibility of them not being as unique as a non-truncated hash.

$(sha1 $HOST)

$(sha1 $HOST salted)

$(sha1 --length 6 $HOST)

rewrite r_rewrite_hostname{set("$(sha1 $HOST)", value("HOST"));};

rewrite pseudonymize_ip_addresses_in_message {subst ("((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])[.]){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))", "$(sha1 $0)", value("MESSAGE"));};

if

Syntax:

$(if (<condition>) <true template> <false template>)

Description: Returns the value of the <true template> parameter if the <condition> is true. If the <condition> is false, the value of <false template> is returned.

$(if ("${username}" == "root") "violation" "system")

<value name="username">$(if ("${username}" == "root") "violation" "system")</value>

<value name="username">
    $(if ("${username}" == "root")
        "root"
        $(if ("${username}" == "joe") "admin" "normal user"))</value>

indent-multi-line

Syntax:

$(indent-multi-line parameter)

Description: This template function makes it possible to write multi-line log messages into a file. The first line is written like a regular message, subsequent lines are indented with a tab, in compliance with RFC822.

destination d_file {
    file (
        "/var/log/messages"
        template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n")
    );
};

ipv4-to-int

Syntax:

$(ipv4-to-int parameter)

Description: Converts the specified IPv4 address to its numeric representation. The numerical value of an IPv4 address is calculated by treating the IP address as a 4-byte hexadecimal value. For example, the 192.168.1.1 address equals to: 192=C0, 168=A8, 1=01, 1=01, or C0A80101, which is 3232235777 in decimal representation.

List manipulation

The list-* template functions allow you to manipulate comma-separated lists. Such lists represent a simple array type in syslog-ng OSE. Note the following about formatting lists:

• Values are separated by commas, for example, "item1","item2","item3". The single-element list is an element without a comma.

• You can use shell-like quotation to embed commas, for example, "item1","ite\,m2","item3".

• Empty values are skipped (except if they are quoted)

These template functions return a well-formed list, properly encoding and quoting all elements. If a template function returns a single element, all quotation is decoded and the value contains the literal value.

Starting with syslog-ng OSE version 3.10, the following list-related template functions are available. Certain functions allow you to reference an element using its number: note that the list index starts with zero, so the index of the first element is 0, the second element is 1, and so on.

$(list-append ${list} ${name-value-pair1} ${name-value-pair2} ... )

$(list-concat ${name-value-pair1} ${name-value-pair2} ... )

$(list-concat ${list1} ${list2})

$(list-count ${list} )

$(list-head ${list} )

$(list-nth <index-number> ${list} )

$(list-tail ${list} )

$(list-slice <from>:<to> ${list} )

length

Syntax:

$(length "<macro>")

Description: Returns the length of the macro in characters, for example, the length of the message. For example, the following filter selects messages that are shorter than 16 characters:

f_short {
    match ('-', value ("$(if ($(length "${MESSAGE}") <= 16) "-" "+")"));
};

lowercase

Syntax:

$(lowercase "<macro>")

Description: Returns the lowercase version of the specified string or macro. For example, the following example uses the lowercase version of the hostname in a directory name:

destination d_file {
    file ("/var/log/${MONTH}/${DAY}/$(lowercase "${HOST}")/messages");
};

Available in syslog-ng OSE 3.5 and later.

Numerical operations

Syntax:

$(<operation> "<value1>" "<value2>")

Description: These template functions allow you to manipulate numbers, that is, to perform addition (+), substraction (-), multiplication (*), division (/), and modulus (%). All of them require two numeric arguments. The result is NaN (Not-a-Number) if the parameters are not numbers, cannot be parsed, or if a division by zero would occur. For example, to add the value of two macros, use the following template function:

$(+ "${<MACRO1>}" "${<MACRO2>}");

When you are correlating messages and a name-value pair contains numerical values in the messages, you can calculate the lowest (min), highest (max), total (sum), and mean (average) values. These calculations process every message of the correlation context. For details on message correlation, see Correlating log messages. For example, if the messages of the context have a .myfields.load name-value pair, you can find the highest load value using the following template function.

$(max ${.myfields.load})

or

Syntax:

$(or <macro1> <macro2>)

Description: This template function returns the first non-empty argument.

padding

Syntax:

 $(padding <macro> <width> <prepended-character-or-string>)

Description: This template function returns the value of its first parameter (a string or macro), prepended with a string. This string is <width> long, and repeats the character or string set in the third parameter. If you use a single character, it is added <width> times. If you use a string, it is repeated until its length reaches <width>. The default padding character is ' ' (space). For example:

$(padding ${MESSAGE} 10 X)

$(padding ${MESSAGE} 10 foo)

python

Syntax:

 $(python <name-of-the-python-method-to-use> <arguments-of-the-method>)

Description: This template function enables you to write a custom template function in Python. You can define a Python block in your syslog-ng OSE configuration file, define one or more Python functions in it, and use the methods as template functions. If you use a Python block, syslog-ng OSE embeds a Python interpreter to process the messages. Note the following points:

• Currently only Python 2.7 is supported.

• The Python block must be a top-level block in the syslog-ng OSE configuration file.

• The Python block can contain multiple Python functions.

• The first argument in the definition of the Python function is the actual log message. This is implicitly passed to the function, you do not have to use it in the template function.

• The value of the template function is return value of the Python function.

• To reference a name-value pair or a macro in the Python function, use the dot-notation. For example, if the first argument in the definition of the function is called log-message, the value of the HOST macro is log-message.HOST, and so on.

• You can define new name-value pairs in the Python function. For example, if the first argument in the definition of the function is called log-message, you can create a new name-value pair like this: log_message["new-macro-name"]="value". This is useful when you parse a part of the message from Python, or lookup a value based on data extracted from the log message.

python {
def <name_of_the_python_function>(<log_message>, <optional_other_arguments>):
    # <your-python-code>
    return <value_of_the_template_function>
};

template <template-name> {
    template($(python <name_of_the_python_function>));
};

@version: 3.16

python {
def return_message(log_message):
    return log_message.MESSAGE
};

destination d_local {
    file("/tmp/logs.txt" template("[$(python return_message)]\n"));
};

@version: 3.16

python {
import socket

def resolve_host(log_message, hostname):
    try:
        return socket.gethostbyaddr(hostname)[0]
    except (socket.herror, socket.error):
        return 'unknown'
};

destination d_local {
    file(
        "/tmp/logs.txt"
        template("${ISODATE} $(python resolve_host(${SOURCE_IP})) ${MESSAGE}\n")
    );
};

replace-delimiter

Syntax:

$(replace-delimiter "<old-delimiters>" "<new-delimiter>" "<macro>")

Description: Replaces the delimiter character with a new one. For example, the following example replaces the tabulators (\t) in the message with semicolons (;):

$(replace-delimiter "\t" ";" "${MESSAGE}")

Available in syslog-ng OSE 3.5 and later.

sanitize

Syntax:

$(sanitize <options> "<macro1>" "<macro2> ...")

Description: This file replaces the special characters in macro values, for example, it can replace the slash (/) characters in a filename with the underscore (_) character. If you specify multiple arguments, they will be concatenated using the / character, so they can be used as separate directory levels when used in filenames.

The function has the following options:

• --ctrl-chars or -c

• Filter control characters (characters that have an ASCII code of 32 or lower). This option is used by default.

• --invalid-chars <characterlist> or -i <characterlist>

• The list of characters to be replaced with underscores (_). The default list contains the / character. The following example replaces the \ and @ characters, so for example, fo\o@bar becomes foobar:

  $(sanitize -i \@ $PROGRAM)

• --no-ctrl-chars or -C

• Do not filter the control characters (characters that have an ASCII code of 32 or lower).

• --replacement <replacement-character> or -r <replacement-character>

• The character used to replace invalid characters. By default, this is the underscore (_). The following example replaces invalid characters with colons instead of underscores, so for example, foo/bar becomes foo;bar:

  $(sanitize -r ; $PROGRAM)

file("/var/log/$(sanitize $HOST $PROGRAM)/messages");

stardate

Syntax:

$(stardate [option] "<date-in-unixtime>")

Description: Converts a date in UNIXTIME (for example, ${UNIXTIME}) into stardate, displaying the year and the progress of the year in a number of digits (YYYY.NNN). You can set the number of digits using the --digits option, for example:

$(stardate --digits 2 "${R_UNIXTIME}")

strip

Syntax:

$(strip "<macro>")

Description: Deletes whitespaces from the beginning and the end of a macro. You can specify multiple macros separated with whitespace in a single template function, for example:

$(strip "${MESSAGE}" "${PROGRAM}")

substr

Syntax:

$(substr "<argument>" "<offset>" "<length>")

Description: This function extracts a substring of a string.

• argument

• The string to extract the substring from, for example, "${MESSAGE}"

• offset

• Specifies where the substring begins (in characters). 0 means to start from the beginning of the string, 5 means to skip the first 5 characters of the string, and so on. Use negative numbers to specify where to start from the end of the string, for example, -1 means the last character, -5 means to start five characters before the end of the string.

• length

• Optional parameter: The number of characters to extract. If not specified, the substring will be extracted from the offset to the end of the string. Use negative numbers to stop the substring before the end of the string, for example, -5 means the substring ends five characters before the end of the string.

$(substr "${MESSAGE}" "15");

$(substr "${MESSAGE}" "15" "15");

$(substr "${MESSAGE}" "-15");

template t_truncate_messages {
    template("$(substr \"<$PRI>$DATE $HOST $MSGHDR$MESSAGE\" \"0\" \"1023\")\n");
    template-escape(no);
};

uppercase

Syntax:

$(uppercase "<macro>")

Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:

destination d_file {
    file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
};

Available in syslog-ng OSE 3.5 and later.

urlencode

Syntax:

$(urlencode ${MSG} )\n")

Description: You can use the urlencode template together with the telegram() destination to send syslog messages to Telegram. The urlencode template function escapes strings. All input characters that are not a-z, A-Z, 0-9, '-', '.', '_' or '~' are converted to their "URL escaped" version.

Available in syslog-ng OSE version 3.16 and later.

uuid

Syntax:

$(uuid)

Description: Generates a Universally Unique IDentifier (UUID) that complies with RFC4122. That way, an UUID can be added to the message soon after it is received, so messages stored in multiple destinations can be identified. For example, when storing messages in a database and also in files, the UUID can be used to find a particular message both in the database and the files.

To generate a UUID, you can use a rewrite rule to create a new value-pair for the message.

rewrite r_add_uuid {
    set("$(uuid)" value("MESSAGE_UUID"));
};

destination d_file {
    file (
        "/var/log/messages"
        template("$MESSAGE_UUID $ISODATE $HOST $MSG\n")
        template-escape(no)
    );
};

log {
    source(s_network);
    rewrite(r_add_uuid);
    destination(d_file);
};