Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.18 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql: Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

The configuration syntax in detail

Every syslog-ng configuration file must begin with a line containing the version information of syslog-ng. For syslog-ng version 7, this line looks like:

@version: 7.0

Versioning the configuration file was introduced in syslog-ng 3.0. If the configuration file does not contain the version information, syslog-ng assumes that the file is for syslog-ng version 2.x. In this case it interprets the configuration and sends warnings about the parts of the configuration that should be updated. Version 3.0 and later will correctly operate with configuration files of version 2.x, but the default values of certain parameters have changed since 3.0.

Example: A simple configuration file

The following is a very simple configuration file for syslog-ng: it collects the internal messages of syslog-ng and the messages from /dev/log into the /var/log/messages_syslog-ng.log file.

@version: 7.0

source s_local {
    unix-dgram("/dev/log");
    internal();
};

destination d_file {
    file("/var/log/messages_syslog-ng.log");
};

log {
    source(s_local);
    destination(d_file);
};

As a syslog-ng user described on a mailing list:

Alan McKinnon

The syslog-ng's config file format was written by programmers for programmers to be understood by programmers. That may not have been the stated intent, but it is how things turned out. The syntax is exactly that of C, all the way down to braces and statement terminators.

  • The main body of the configuration file consists of object definitions: sources, destinations, logpaths define which log message are received and where they are sent. All identifiers, option names and attributes, and any other strings used in the syslog-ng configuration file are case sensitive. Object definitions (also called statements) have the following syntax:

    type-of-the-object identifier-of-the-object {<parameters>};
    • Type of the object: One of source, destination, log, filter, parser, rewrite rule, or template.

    • Identifier of the object: A unique name identifying the object. When using a reserved word as an identifier, enclose the identifier in quotation marks.

      All identifiers, attributes, and any other strings used in the syslog-ng configuration file are case sensitive.

      TIP:

      Use identifiers that refer to the type of the object they identify. For example, prefix source objects with s_, destinations with d_, and so on.

      NOTE:

      Repeating a definition of an object (that is, defining the same object with the same id more than once) is not allowed, unless you use the @define allow-config-dups 1 definition in the configuration file.

    • Parameters: The parameters of the object, enclosed in braces {parameters}.

    • Semicolon: Object definitions end with a semicolon (;).

    For example, the following line defines a source and calls it s_internal.

    source s_internal {
        internal();
    };

    The object can be later referenced in other statements using its ID, for example, the previous source is used as a parameter of the following log statement:

    log {
        source(s_internal);
        destination(d_file);
    };
  • The parameters and options within a statement are similar to function calls of the C programming language: the name of the option followed by a list of its parameters enclosed within brackets and terminated with a semicolon.

    option(parameter1, parameter2);
    option2(parameter1, parameter2);

    For example, the file() driver in the following source statement has three options: the filename (/var/log/apache/access.log), follow-freq(), and flags(). The follow-freq() option also has a parameter, while the flags() option has two parameters.

    source s_tail {
        file("/var/log/apache/access.log"
            follow-freq(1)
            flags(no-parse, validate-utf8)
        );
    };

    Objects may have required and optional parameters. Required parameters are positional, meaning that they must be specified in a defined order. Optional parameters can be specified in any order using the option(value) format. If a parameter (optional or required) is not specified, its default value is used. The parameters and their default values are listed in the reference section of the particular object.

    Example: Using required and optional parameters

    The unix-stream() source driver has a single required argument: the name of the socket to listen on. Optional parameters follow the socket name in any order, so the following source definitions have the same effect:

    source s_demo_stream1 {
        unix-stream("<path-to-socket>"
            max-connections(10)
            group(log)
        );
    };
    source s_demo_stream2 {
        unix-stream("<path-to-socket>"
            group(log)
            max-connections(10)
        );
    };
  • Some options are global options, or can be set globally, for example, whether syslog-ng PE should use DNS resolution to resolve IP addresses. Global options are detailed in Global options of syslog-ng PE.

    options {
        use-dns(no);
    };
  • Objects can be used before definition.

  • Objects can be defined inline as well. This is useful if you use the object only once (for example, a filter). For details, see Defining configuration objects inline.

  • To add comments to the configuration file, start a line with # and write your comments. These lines are ignored by syslog-ng.

    # Comment: This is a stream source
    source s_demo_stream {
        unix-stream("<path-to-socket>"
            max-connections(10)
            group(log)
        );
    };
TIP:

Before activating a new configuration, check that your configuration file is syntactically correct using the syslog-ng --syntax-only command.

To activate the configuration, reload the configuration of syslog-ng using the /etc/init.d/syslog-ng reload command.

Notes about the configuration syntax

When you are editing the syslog-ng configuration file, note the following points:

  • The configuration file can contain a maximum of 6665 source / destination / log elements.

  • When writing the names of options and parameters (or other reserved words), the hyphen (-) and underscore (_) characters are equivalent, for example, max-connections(10) and max_connections(10) are both correct.

  • Numbers can be prefixed with + or - to indicate positive or negative values. Numbers beginning with zero (0) or 0x are treated as octal or hexadecimal numbers, respectively.

    Starting with syslog-ng PE version 7.0, you can use suffixes for kilo-, mega-, and gigabytes. Use the Kb, Mb, or Gb suffixes for the base-10 version, and Kib, Mib, or Gib for the base-2 version. That is, 2MB means 2000000, while 2MiB means 2097152. For example, to set the log-msg-size() option to 2000000 bytes, use log-msg-size(2Mb).

  • You can use commas (,) to separate options or other parameters for readability, syslog-ng completely ignores them. The following declarations are equivalent:

    source s_demo_stream {
        unix-stream("<path-to-socket>"
            max-connections(10)
            group(log)
        );
    };
    source s_demo_stream {
        unix-stream("<path-to-socket>",
            max-connections(10),
            group(log)
        );
    };
  • When enclosing object IDs (for example, the name of a destination) between double-quotes ("mydestination"), the ID can include whitespace as well, for example:

    source "s demo stream" {
        unix-stream("<path-to-socket>"
            max-connections(10)
            group(log)
        );
    };
  • For notes on using regular expressions, see Regular expressions.

  • You can use if {}, elif {}, and else {} blocks to configure conditional expressions. For details, see if-else-elif: Conditional expressions.

Defining configuration objects inline

Starting with syslog-ng PE 7.0, you can define configuration objects inline, where they are actually used, without having to define them in a separate object. This is useful if you need an object only once, for example, a filter or a rewrite rule, because it makes the configuration much easier to read. Every object can be defined inline: sources, destinations, filters, parsers, rewrite rules, and so on.

To define an object inline, use braces instead of parentheses. That is, instead of <object-type> (<object-id>);, you use <object-type> {<object-definition>};

Example: Using inline definitions

The following two configuration examples are equivalent. The first one uses traditional statements, while the second uses inline definitions.

source s_local {
    system();
    internal();
};
destination d_local {
    file("/var/log/messages");
};
log {
    source(s_local);
    destination(d_local);
};
log {
    source {
        system();
        internal();
    };
    destination {
        file("/var/log/messages");
    };
};

Using channels in configuration objects

Starting with syslog-ng PE7.0, every configuration object is a log expression. Every configuration object is essentially a configuration block, and can include multiple objects. To reference the block, only the top-level object must be referenced. That way you can use embedded log statements, junctions and in-line object definitions within source, destination, filter, rewrite and parser definitions. For example, a source can include a rewrite rule to modify the messages received by the source, and that combination can be used as a simple source in a log statement. This feature allows you to preprocess the log messages very close to the source itself.

To embed multiple objects into a configuration object, use the following syntax. Note that you must enclose the configuration block between braces instead of parenthesis.

<type-of-top-level-object> <name-of-top-level-object> {
    channel {
        <configuration-objects>
    };
};
Example: Using channels

For example, to process a log file in a specific way, you can define the required processing rules (parsers and rewrite expressions) and combine them in a single object:

source s_apache {
    channel {
        source { file("/var/log/apache/error.log"); };
        parser(p_apache_parser);
    };
};

log { source(s_apache); ... };

The s_apache source uses a file source (the error log of an Apache webserver) and references a specific parser to process the messages of the error log. The log statement references only the s_apache source, and any other object in the log statement can already use the results of the p_apache_parserparser.

NOTE:

You must start the object definition with a channel even if you will use a junction, for example:

parser demo-parser() {
    channel {
        junction {
            channel { ... };
            channel { ... };
        };
    };
};

If you want to embed configuration objects into sources or destinations, always use channels, otherwise the source or destination will not behave as expected. For example, the following configuration is good:

source s_filtered_hosts {
    channel{
        source {
            pipe("/dev/pipe");
            syslog(
                ip(192.168.0.1)
                transport("tcp")
            );
            syslog(
                ip(127.0.0.1)
                transport("tcp")
            );
        };
        filter {
            netmask(10.0.0.0/16);
        };
    };
};;
Related Documents