syslog-ng Open Source Edition 3.25 - Administration Guide

Structuring macros, metadata, and other value-pairs

Available in syslog-ng OSE 3.3 and later.

The syslog-ng OSE application allows you to select and construct name-value pairs from any information already available about the log message, or extracted from the message itself. You can directly use this structured information, for example, in the following places:

• amqp() destination

• format-welf() template function

• mongodb() destination

• stomp() destination

• or in other destinations using the format-json() template function.

When using value-pairs, there are three ways to specify which information (that is, macros or other name-value pairs) to include in the selection.

• Select groups of macros using the scope() parameter, and optionally remove certain macros from the group using the exclude() parameter.

• List specific macros to include using the key() parameter.

• Define new name-value pairs to include using the pair() parameter.

These parameters are detailed in value-pairs().

Specifying data types in value-pairs

By default, syslog-ng OSE handles every data as strings. However, certain destinations and data formats (for example, SQL, MongoDB, JSON, AMQP) support other types of data as well, for example, numbers or dates. The syslog-ng OSE application allows you to specify the data type in templates (this is also called type-hinting). If the destination driver supports data types, it converts the incoming data to the specified data type. For example, this allows you to store integer numbers as numbers in MongoDB, instead of strings.

To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>"), for example: int("$PID").

Currently the mongodb() destination and the format-json and format-flat-json() template functions support data types.

mongodb(
    value-pairs(pair("date", datetime("$UNIXTIME"))
        pair("pid", int64("$PID"))
        pair("program", "$PROGRAM"))
        pair("message", "$MESSAGE"))
    )
);

$(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$MESSAGE)

$(format-json message=list($MESSAGE))"

The syslog-ng OSE application currently supports the following data-types.

• boolean: Converts the data to a boolean value. Anything that begins with a t or 1 is converted to true, anything that begins with an f or 0 is converted to false.

• datetime: Use it only with UNIX timestamps, anything else will likely result in an error. This means that currently you can use only the $UNIXTIME macro for this purpose.

• double: A floating-point number.

• literal: The data as a literal string, without adding any quotes or escape characters.

• list: The data as a list. For details, see the list manipulation template functions in Template functions of syslog-ng OSE.

• int or int32: 32-bit integer.

• int64: 64-bit integer.

• string: The data as a string.

value-pairs()

empty string

Description: The value-pairs() option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a mongodb() destination.

value-pairs(
    scope(nv_pairs core syslog all_macros selected_macros everything)
    exclude("R_*")
    exclude("S_*")
    key(".SDATA.meta.sequenceId")
    pair("MSGHDR" "$PROGRAM[$PID]: ")
)

$(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \
    --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
    --pair MSGHDR="$PROGRAM[$PID]: ")

The value-pairs() option has the following parameters. The parameters are evaluated in the following order:

1. scope()

2. exclude()

3. key()

4. pair()

value-pairs(
    scope(rfc5424 selected_macros)
    exclude(".SDATA*")
)

value-pairs(
    scope(rfc3164)
    key("HOST")
)

value-pairs(
    scope(rfc3164)
    key("HOST", "PROGRAM")
)

value-pairs(
    scope(rfc3164 selected-macros)
    omit-empty-values()
)

value-pairs(
    scope(rfc3164)
    pair("TIME" "$HOUR:$MIN")
    pair("MSGHDR" "$PROGRAM[$PID]: ")
)

value-pairs(
    key(".cee.*"
        rekey(
            shift(4)
            add-prefix("events.")
        )
    )
)

$(format-json --rekey .cee.* --add-prefix events.)

value-pairs(
    scope(rfc3164 selected-macros)
)

Things to consider when forwarding messages between syslog-ng OSE hosts

When you send your log messages from a syslog-ng OSE client through the network to a syslog-ng OSE server, you can use different protocols and options. Every combination has its advantages and disadvantages. The most important thing is to use matching protocols and options, so the server handles the incoming log messages properly.

In syslog-ng OSE you can change many aspects of the network communication. First of all, there is the structure of the messages itself. Currently, syslog-ng OSE supports two standard syslog protocols: the BSD (RFC3164) and the syslog (RFC5424) message format.

These RFCs describe the format and the structure of the log message, and add a (lightweight) framing around the messages. You can set this framing/structure by selecting the appropriate driver in syslog-ng OSE. There are two drivers you can use: the network() driver and the syslog() driver. The syslog() driver is for the syslog (RFC5424) protocol and the network() driver is for the BSD (RFC3164) protocol.

The tcp() and udp() drivers are now deprecated, they are essentially equivalent with the network(transport(tcp)) and network(transport(udp)) drivers.

In addition to selecting the driver to use, both drivers allow you to use different transport-layer protocols: TCP and UDP, and optionally also higher-level transport protocols: TLS (over TCP. To complicate things a bit more, you can configure the network() driver (corresponding to the BSD (RFC3164) protocol) to send the messages in the syslog (RFC5424) format (but without the framing used in RFC5424) using the flag(syslog-protocol) option.

Because some combination of drivers and options are invalid, you can use the following drivers and options as sources and as destinations:

1. syslog(transport(tcp))

2. syslog(transport(udp))

3. syslog(transport(rltp))

4. syslog(transport(tls))

5. syslog(transport(rltp(tls-required(yes)))

6. network(transport(tcp))

7. network(transport(udp))

8. network(transport(rltp))

9. network(transport(tls))

10. network(transport(rltp(tls-required(yes)))

11. network(transport(tcp) flag(syslog-protocol))

12. network(transport(udp) flag(syslog-protocol))

13. network(transport(rltp)flag(syslog-protocol))

14. network(transport(tls) flag(syslog-protocol))

15. network(transport(rltp(tls-required(yes)) flag(syslog-protocol))

If you use the same driver and options in the destination of your syslog-ng OSE client and the source of your syslog-ng OSE server, everything should work as expected. Unfortunately there are some other combinations, that seem to work, but result in losing parts of the messages. The following table show the combinations:

• - This method does not work. The logs will not get to the server.

• ✔ This method works.

• ! This method has some visible drawbacks. The logs go through, but some of the values are missing/misplaced/and so on.

• ✔? This method seems to work, but it is not recommended because this can change in a future release.