Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.20 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel: Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql: Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

smtp: Generating SMTP messages (email) from logs

The destination is aimed at a fully controlled local, or near-local, trusted SMTP server. The goal is to send mail to trusted recipients, through a controlled channel. It hands mails over to an SMTP server, and that is all it does, therefore the resulting solution is as reliable as sending an email in general. For example, syslog-ng PE does not verify whether the recipient exists.

The smtp() driver sends email messages triggered by log messages. The smtp() driver uses SMTP, without needing external applications. You can customize the main fields of the email, add extra headers, send the email to multiple recipients, and so on.

The subject(), body(), and header() fields may include macros which get expanded in the email. For more information on available macros see Macros of syslog-ng PE.

The smtp() driver has the following required parameters: host(), port(), from(), to(), subject(), and body(). For the list of available optional parameters, see smtp() destination options.

NOTE: To use this destination, syslog-ng PE must run in server mode. Typically, only the central syslog-ng PE server uses this destination. For more information on the server mode, see Server mode.

NOTE:

The smtp() destination driver is available only in 5 F2 and later.

Declaration
smtp(host() port() from() to() subject() body() options());
Example: Using the smtp() driver

The following example defines an smtp() destination using only the required parameters.

destination d_smtp {
    smtp(
        host("localhost")
        port(25)
        from("syslog-ng alert service" "noreply@example.com")
        to("Admin #1" "admin1@example.com")
        subject("[ALERT] Important log message of $LEVEL condition received from $HOST/$PROGRAM!")
        body("Hi!\nThe syslog-ng alerting service detected the following important log message:\n $MSG\n-- \nsyslog-ng\n")
    );
};

The following example sets some optional parameters as well.

destination d_smtp {
    smtp(
        host("localhost")
        port(25)
        from("syslog-ng alert service" "noreply@example.com")
        to("Admin #1" "admin1@example.com")
        to("Admin #2" "admin2@example.com")
        cc("Admin BOSS" "admin.boss@example.com")
        bcc("Blind CC" "blindcc@example.com")
        subject("[ALERT] Important log message of $LEVEL condition received from $HOST/$PROGRAM!")
        body("Hi!\nThe syslog-ng alerting service detected the following important log message:\n $MSG\n-- \nsyslog-ng\n")
        header("X-Program", "$PROGRAM")
    );
};
Example: Simple email alerting with the smtp() driver

The following example sends an email alert if the eth0 network interface of the host is down.

filter f_linkdown {
    match("eth0: link down" value("MESSAGE"));
};
destination d_alert {
    smtp(
        host("localhost") port(25)
        from("syslog-ng alert service" "syslog@localhost")
        reply-to("Admins" "root@localhost")
        to("Ennekem" "me@localhost")
        subject("[SYSLOG ALERT]: eth0 link down")
        body("Syslog received an alert:\n$MSG")
    );
};

log {
    source(s_local);
    filter(f_linkdown);
    destination(d_alert);
};

smtp() destination options

The smtp() sends email messages using SMTP, without needing external applications. The smtp() destination has the following options:

body()
Type: string
Default: n/a

Description: The BODY field of the email. You can also use macros in the string. Use \n to start a new line. For example:

body("syslog-ng PE received the following alert from $HOST:\n$MSG")
bcc()
Type: string
Default: n/a

Description: The BCC recipient of the email (contents of the BCC field). You can specify the email address, or the name and the email address. Set the bcc() option multiple times to send the email to multiple recipients. For example: bcc("admin@example.com") or bcc("Admin" "admin@example.com") or bcc("Admin" "admin@example.com") bcc("Admin2" "admin2@example.com")

You can also use macros to set the value of this parameter.

cc()
Type: string
Default: n/a

Description: The CC recipient of the email (contents of the CC field). You can specify the email address, or the name and the email address. Set the cc() option multiple times to send the email to multiple recipients. For example: cc("admin@example.com") or cc("Admin" "admin@example.com") or cc("Admin" "admin@example.com") cc("Admin2" "admin2@example.com")

You can also use macros to set the value of this parameter.

disk-buffer()

Description: This option enables putting outgoing messages into the disk-buffer file of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:

reliable()
Type: yes|no
Default: no

Description: If set to yes, syslog-ng PE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng PE crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer option will be used. This provides a faster, but less reliable disk-buffer option.

Caution:

Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer file, the messages stored in the disk-buffer file will be lost.

dir()
Type: string
Default: N/A

Description: Defines the folder where the disk-buffer files are stored.

Note that changing the value the dir() option will not move or copy existing files from the old directory to the new one.

Caution:

When creating a new dir() option for a disk-buffer file, or modifying an existing one, make sure you delete the persist file.

syslog-ng PE creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, syslog-ng PE will look for or create disk-buffer files in their old location. To ensure that syslog-ng PE uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.

disk-buf-size()
Type: number [bytes]
Default:

Description: This is a required option. The maximum size of the disk-buffer file in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
mem-buf-length()
Type: number [messages]
Default: 10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
mem-buf-size()
Type: number [bytes]
Default: 163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk-buffer file. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
qout-size()
Type: number [messages]
Default: 64
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer file already exists, the change will take effect when the disk-buffer file becomes empty.

Options reliable() and disk-buf-size() are required options.

Example: Examples for using disk-buffer()

In the following case reliable disk-buffer() is used.

destination d_demo {
    network("127.0.0.1"
        port(3333)
        disk-buffer(
            mem-buf-size(10000)
            disk-buf-size(2000000)
            reliable(yes)
            dir("/tmp/disk-buffer")
        )
    );
};

In the following case normal disk-buffer() is used.

destination d_demo {
    network("127.0.0.1"
        port(3333)
        disk-buffer(
            mem-buf-length(10000)
            disk-buf-size(2000000)
            reliable(no)
            dir("/tmp/disk-buffer")
        )
    );
};
from()
Type: string
Default: n/a

Description: The sender of the email (contents of the FROM field). You can specify the email address, or the name and the email address. For example:

from("admin@example.com")
or
from("Admin" "admin@example.com")

If you specify the from() option multiple times, the last value will be used. Instead of the from() option, you can also use sender(), which is just an alias of the from() option.

You can also use macros to set the value of this parameter.

header()
Type: string
Default: n/a

Description: Adds an extra header to the email with the specified name and content. The first parameter sets the name of the header, the second one its value. The value of the header can contain macros. Set the header() option multiple times to add multiple headers. For example:

header("X-Program", "$PROGRAM")

When using the header option, note the following points:

  • Do not use the header() option to set the values of headers that have dedicated options. Use it only to add extra headers.

  • If you set the same custom header multiple times, only the first will be added to the email, other occurrences will be ignored.

  • It is not possible to set the DATE, Return-Path, Original-Recipient, Content-*, MIME-*, Resent-*, Received headers.

host()
Type: hostname or IP address
Default: n/a

Description: Hostname or IP address of the SMTP server.

NOTE:

If you specify host="localhost", syslog-ng PE will use a socket to connect to the local SMTP server. Use host="127.0.0.1" to force TCP communication between syslog-ng PE and the local SMTP server.

log-fifo-size()
Type: number
Default: Use global setting.

Description: The number of messages that the output queue can store.

port()
Type: number
Default: 25

Description: The port number of the SMTP server.

reply-to()
Type: string
Default: n/a

Description: Replies of the recipient will be sent to this address (contents of the REPLY-TO field). You can specify the email address, or the name and the email address. Set the reply-to() option multiple times to send the email to multiple recipients. For example: reply-to("admin@example.com") or reply-to("Admin" "admin@example.com") or reply-to("Admin" "admin@example.com") reply-to("Admin2" "admin2@example.com")

You can also use macros to set the value of this parameter.

retries()
Type: number [of attempts]
Default: 3

Description: The number of times syslog-ng PE attempts to send a message to this destination. If syslog-ng PE could not send a message, it will try again until the number of attempts reaches retries, then drops the message.

subject()
Type: string
Default: n/a

Description: The SUBJECT field of the email. You can also use macros. For example:

subject("[SYSLOG ALERT]: Critical error message received from $HOST")

If you specify the subject() option multiple times, the last value will be used.

throttle()
Type: number
Default: 0

Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using the disk-buffer option as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

to()
Type: string
Default: localhost

Description: The recipient of the email (contents of the TO field). You can specify the email address, or the name and the email address. Set the to() option multiple times to send the email to multiple recipients. For example: to("admin@example.com") or to("Admin" "admin@example.com") or to("Admin" "admin@example.com") to("Admin2" "admin2@example.com")

You can also use macros to set the value of this parameter.

splunk-hec: Sending messages to Splunk HTTP Event Collector

Version 7.0.12 of syslog-ng PE can directly post log messages to a Splunk deployment using the HTTP Event Collector (HEC) over the HTTP and Secure HTTP (HTTPS) protocols.

HTTPS connection, as well as password- and certificate-based authentication is supported. The content of the events is sent in JSON format.

NOTE: Typically, only the central syslog-ng PE server uses this destination. For more information on the server mode, see Server mode.

Declaration
d_splunk_hec {
    splunk_hec(
        #mandatory
        index("<splunk-index-to-store-messages>")
        token("<event-collector-tokens>")
        url("http://your-splunk-server:8088/services/collector/event")
    );
};
Prerequisites
  • On your Splunk deployment, you must enable HTTP Event Collector (HEC).

  • On your Splunk deployment, you must create a token for syslog-ng PE. You must use this token in the token() option of your splunk-hec() destination. We recommend to use the syslog source type for the token.

For details, see Set up and use HTTP Event Collector in Splunk Web.

Example: Sending log data to Splunk

The following example defines a splunk-hec() destination.

d_splunk_hec {
    splunk_hec(
        # mandatory
        index("<splunk-index-to-store-messages>")
        token("<event-collector-tokens>")
        url("http://your-splunk-server:8088/services/collector/event")

        # optional
        batch_lines(25)
        workers(4)
        source("syslog-ng")
        sourcetype("${.app.name:-syslog}")
        delimiter("\n")
        time("$S_UNIXTIME.$S_MSEC")
        host("$HOST")
        event("$S_ISODATE $HOST $MSGHDR$MSG\n")
        timeout(10));
    );
};

log {
    source(s_file);
    destination(d_splunk_hec);
    flags(flow-control);
};

Batch mode and load balancing

The splunk-hec() destination automatically sends multiple log messages in a single HTTP request, increasing the rate of messages that your Splunk deployment can consume. For details on adjusting and fine-tuning the batch mode of the splunk-hec() destination, see the following section.

Batch size

The batch-lines(), batch-lines(), and batch-timeout() options of the destination determine how many log messages syslog-ng PE sends in a batch. The batch-lines() option determines the maximum number of messages syslog-ng PE puts in a batch in. This can be limited based on size and time:

  • syslog-ng PE sends a batch every batch-timeout() milliseconds, even if the number of messages in the batch is less than batch-lines(). This ensures that the destination receives every message in a timely manner even if suddenly there are no more messages.

  • syslog-ng PE sends the batch if the total size of the messages in the batch reaches batch-bytes() bytes.

To increase the performance of the destination, increase the number of worker threads for the destination using the workers() option, or adjust the batch-bytes(), batch-lines(), batch-timeout() options.

Example: HTTP batch mode

In the following example, a batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds).

destination d_splunk-hec {
    splunk-hec(url("http://your-splunk-server:8088/services/collector/event")
        index("<splunk-index-to-store-messages>")
        token("<event-collector-tokens>")
        url("http://your-splunk-server:8088/services/collector/event")
        batch-lines(100)
        batch-bytes(512Kb)
        batch-timeout(10000)
    );
};
Load balancing between multiple Splunk indexers

Starting with version 7.0.12, you can specify multiple URLs, for example, url("site1" "site2"). In this case, syslog-ng PE sends log messages to the specified URLs in a load-balance fashion. This means that syslog-ng PE sends each message to only one URL. For example, you can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. Note that the order of the messages as they arrive on the servers can differ from the order syslog-ng PE has received them, so use load-balancing only if your server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the order of the messages will be incorrect.

Caution:

If you set multiple URLs in the url() option, set the persist-name() option as well to avoid data loss.

Example: HTTP load balancing

The following destination sends log messages to 3 different Splunk indexer nodes. Each node is assigned a separate worker thread. A batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds).

destination d_splunk-hec {
    splunk-hec(url("http://your-splunk-server1:8088/services/collector/event" "http://your-splunk-server2:8088/services/collector/event" "http://your-splunk-server3:8088/services/collector/event")
        batch-lines(100)
        batch-bytes(512Kb)
        batch-timeout(20000)
        persist-name("d_splunk-hec-load-balance")
    );
};

If you are using load-balancing (that is, you have configured multiple servers in the url() option), increase the number of worker threads at least to the number of servers. For example, if you have set three URLs (url("site1" "site2" "site3")), set the workers() option to 3 or more.

Related Documents