syslog-ng Open Source Edition 3.18 - Administration Guide

template and rewrite: Format, modify, and manipulate log messages

This chapter explains the methods that you can use to customize, reformat, and modify log messages using syslog-ng Open Source Edition.

• Customize message format using macros and templates explains how to use templates and macros to change the format of log messages, or the names of logfiles and database tables.

• Modifying messages using rewrite rules describes how to use rewrite rules to search and replace certain parts of the message content.

• Regular expressions lists the different types of regular expressions that can be used in various syslog-ng OSE objects like filters and rewrite rules.

Customize message format using macros and templates

The following sections describe how to customize the names of logfiles, and also how to use templates, macros, and template functions.

• Formatting messages, filenames, directories, and tablenames explains how macros work.

• Modifying messages using rewrite rules describes how to use macros and templates to format log messages or change the names of logfiles and database tables.

• Macros of syslog-ng OSE lists the different types of macros available in syslog-ng OSE.

• Using template functions explains what template functions are and how to use them.

• Template functions of syslog-ng OSE lists the template functions available in syslog-ng OSE.

Formatting messages, filenames, directories, and tablenames

The syslog-ng OSE application can dynamically create filenames, directories, or names of database tables using macros that help you organize your log messages. Macros refer to a property or a part of the log message, for example, the ${HOST} macro refers to the name or IP address of the client that sent the log message, while ${DAY} is the day of the month when syslog-ng has received the message. Using these macros in the path of the destination log files allows you for example to collect the logs of every host into separate files for every day.

A set of macros can be defined as a template object and used in multiple destinations.

Another use of macros and templates is to customize the format of the syslog message, for example, to add elements of the message header to the message text.

• For details on using templates and macros, see Templates and macros.

• For a list and description of the macros available in syslog-ng OSE, see Macros of syslog-ng OSE.

• For details on using custom macros created with CSV parsers and pattern databases, see parser: Parse and segment structured messages and Using parser results in filters and templates, respectively.

Templates and macros

The syslog-ng OSE application allows you to define message templates, and reference them from every object that can use a template. Templates can include strings, macros (for example date, the hostname, and so on), and template functions. For example, you can use templates to create standard message formats or filenames. For a list of macros available in syslog-ng Open Source Edition, see Macros of syslog-ng OSE. Fields from the structured data (SD) part of messages using the new IETF-syslog standard can also be used as macros.

template <template-name> {
    template("<template-expression>") <template-escape(yes)>;
};

Template objects have a single option called template-escape(), which is disabled by default (template-escape(no)). This behavior is useful when the messages are passed to an application that cannot handle escaped characters properly. Enabling template escaping (template-escape(yes)) causes syslog-ng to escape the ', ", and backslash characters from the messages.

If you do not want to enable the template-escape() option (which is rarely needed), you can define the template without the enclosing braces.

template <template-name> "<template-expression>";

You can also refer to an existing template from within a template. The result of the referred template will be pasted into the second template.

template first-template "sample-text";
template second-template "The result of the first-template is: $(template first-template)";

If you want to use a template only once, you can define the template inline, for example:

destination d_file {
    file ("/var/log/messages" template("${ISODATE} ${HOST} ${MESSAGE}\n") );
};

Macros can be included by prefixing the macro name with a $ sign, just like in Bourne compatible shells. Although using braces around macro names is not mandatory, and the "$MESSAGE" and "${MESSAGE}" formats are equivalent, using the "${MESSAGE}" format is recommended for clarity.

To use a literal $ character in a template, you have to escape it. In syslog-ng OSE versions 3.4 and earlier, use a backslash (\$). In version 3.5 and later, use $$.

Default values for macros can also be specified by appending the :- characters and the default value of the macro. If a message does not contain the field referred to by the macro, or it is empty, the default value will be used when expanding the macro. For example, if a message does not contain a hostname, the following macro can specify a default hostname.

${HOST:-default_hostname}

By default, syslog-ng sends messages using the following template: ${ISODATE} ${HOST} ${MSGHDR}${MESSAGE}\n. (The ${MSGHDR}${MESSAGE} part is written together because the ${MSGHDR} macro includes a trailing whitespace.)

template t_demo_filetemplate {
    template("${ISODATE} ${HOST} ${MESSAGE}\n");
};
destination d_file {
    file("/var/log/messages" template(t_demo_filetemplate));
};

template t_demo_template-with-braces {
    template("${ISODATE} ${HOST} ${MESSAGE}\n");
};
template t_demo_template-without-braces "${ISODATE} ${HOST} ${MESSAGE}\n";

destination d_file {
    file ("/var/log/messages" template("${ISODATE} ${HOST} ${MESSAGE}\n") );
};

destination d_file {
        file("/var/log/${YEAR}.${MONTH}.${DAY}/${HOST}.log");
};