Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.9 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files network: Collecting messages using the RFC3164 protocol (network() driver) osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch: Sending messages directly to Elasticsearch version 1.x elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher file: Storing messages in plain-text files hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications smtp: Generating SMTP messages (e-mail) from logs Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng: Forwarding messages and tags to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Reliable Log Transfer Protocol Manipulating messages Parsers and segmenting structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages About us

system: Collecting the system-specific log messages of a platform

Starting with version 4 F1, syslog-ng PE can automatically collect the system-specific log messages of the host on a number of platforms using the system() driver. If the system() driver is included in the syslog-ng PE configuration file, syslog-ng PE automatically adds the following sources to the syslog-ng PE configuration.

NOTE:

syslog-ng PE versions 4.1-5.0 used an external script to generate the system() source, but this was problematic in certain situations, for example, when the host used a strict AppArmor profile. Therefore, the system() source is now generated internally in syslog-ng PE.

The system() driver is also used in the default configuration file of syslog-ng PE. For details on the default configuration file, see Example: The default configuration file of syslog-ng PE. Starting with syslog-ng PE version , you can use the system-expand command-line utility (which is a shell script, located in the modules/system-source/ directory) to display the configuration that the system() source will use.

Caution:

If syslog-ng PE does not recognize the platform it is installed on, it does not add any sources.

Starting with version 7.0, syslog-ng PE parses messages complying with the Splunk Common Information Model (CIM) and marked with @cim as JSON messages (for example, the ulogd from the netfilter project can emit such messages). That way, you can forward such messages without losing any information to CIM-aware applications (for example, Splunk).

Table 10: Sources automatically added by syslog-ng Premium Edition
Platform Message source
AIX and Tru64
unix-dgram("/dev/log");
FreeBSD
unix-dgram("/var/run/log");
unix-dgram("/var/run/logpriv" perm(0600));
file("/dev/klog" follow-freq(0) program-override("kernel") flags(no-parse));

For FreeBSD versions earlier than 9.1, follow-freq(1) is used.

GNU/kFreeBSD
unix-dgram("/var/run/log");
file("/dev/klog" follow-freq(0) program-override("kernel"));
HP-UX
pipe("/dev/log" pad-size(2048));
Linux
unix-dgram("/dev/log");
file("/proc/kmsg" program-override("kernel") flags(kernel));

Note that on Linux, the so-rcvbuf() option of the system() source is automatically set to 8192.

If the host is running under systemd, syslog-ng PE reads directly from the systemd journal file using the systemd-journal() source.

If the kernel of the host is version 3.5 or newer, and /dev/kmsg is seekable, syslog-ng PE will use that instead of /proc/kmsg, using the multi-line-mode(indented), keep-timestamp(no), and the format(linux-kmsg) options.

If syslog-ng PE is running in a jail or a Linux Container (LXC), it will not read from the /dev/kmsg or /proc/kmsg files.

Solaris 8
sun-streams("/dev/log");

NOTE:

Starting with version , the syslog-ng PE system() driver automatically extracts the msgid from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the extract-solaris-msgid() parser. You can find the exact source of this parser in the syslog-ng PE GitHub repository.

Solaris 9
sun-streams("/dev/log" door("/etc/.syslog_door"));

NOTE:

Starting with version , the syslog-ng PE system() driver automatically extracts the msgid from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the extract-solaris-msgid() parser. You can find the exact source of this parser in the syslog-ng PE GitHub repository.

Solaris 10
sun-streams("/dev/log" door("/var/run/syslog_door"));

NOTE:

Starting with version , the syslog-ng PE system() driver automatically extracts the msgid from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the extract-solaris-msgid() parser. You can find the exact source of this parser in the syslog-ng PE GitHub repository.

systemd-journal: Collecting messages from the systemd-journal system log storage

The systemd-journal() source is used on various Linux distributions, such as RHEL (from RHEL7) and CentOS. The systemd-journal() source driver can read the structured name-value format of the journald system service, making it easier to reach the custom fields in the message. By default, syslog-ng PE adds the .journald. prefix to the name of every parsed value.

The systemd-journal() source driver is designed to read only local messages through the systemd-journal API. It is not possible to set the location of the journal files, or the directories.

NOTE:

The log-msg-size() option is not applicable for this source. Use the max-field-size() option instead.

NOTE:

This source will not handle the following cases:

  • corrupted journal file

  • incorrect journal configuration

  • any other journald-related bugs

NOTE:

If you are using RHEL-7, the default source in the configuration is systemd-journal() instead of unix-dgram("/dev/log") and file("/proc/kmsg"). If you are using unix-dgram("/dev/log") or unix-stream("/dev/log") in your configuration as a source, syslog-ng PE will revert to using systemd-journal() instead.

Caution:

Only one systemd-journal() source can be configured in the configuration file. If there are more than one systemd-journal() sources configured, syslog-ng PE will not start.

Declaration:
systemd-journal(options);
Example: Sending all fields through syslog protocol using the systemd-journal() driver

To send all fields through the syslog protocol, enter the prefix in the following format: ".SDATA.<name>".

@version: 7.0

source s_journald {
    systemd-journal(prefix(".SDATA.journald."));
};

destination d_network {
    syslog("server.host");
};

log {
    source(s_journald);
    destination(d_network);
};
Example: Filtering for a specific field using the systemd-journal() driver
@version: 7.0

source s_journald {
    systemd-journal(prefix(".SDATA.journald."));
};

filter f_uid {"${.SDATA.journald._UID}" eq "1000"};

destination d_network {
    syslog("server.host");
};

log {
    source(s_journald);
    filter(f_uid);
    destination(d_network);
};
Example: Sending all fields in value-pairs using the systemd-journal() driver
@version: 7.0

source s_local {
    systemd-journal(prefix("journald."));
};

destination d_network {
    network("server.host" template("$(format_json --scope rfc5424 --key journald.*)\n"));
};

log {
    source(s_local);
    destination(d_network);
};

The journal contains credential information about the process that sent the log message. The syslog-ng PE application makes this information available in the following macros:

Journald field syslog-ng predefined macro
MESSAGE $MESSAGE
_HOSTNAME $HOST
_PID $PID
_COMM or SYSLOG_IDENTIFIER $PROGRAM If both _COMM and SYSLOG_IDENTIFIER exists, syslog-ng PE uses SYSLOG_IDENTIFIER
SYSLOG_FACILITY $FACILITY_NUM
PRIORITY $LEVEL_NUM

systemd-journal() source options

The systemd-journal() driver has the following options:

default-facility()
Type: facility string
Default: local0

Description: The default facility value if the SYSLOG_FACILITY entry does not exist.

default-level()
Type: string
Default: notice

Description: The default level value if the PRIORITY entry does not exist.

host-override()
Type: string
Default:

Description: Replaces the ${HOST} part of the message with the parameter string.

keep-hostname()
Type: yes or no
Default: no

Description: Enable or disable hostname rewriting.

  • If enabled (keep-hostname(yes)), syslog-ng PE will retain the hostname information read from the systemd journal messages.

  • If disabled (keep-hostname(no)), syslog-ng PE will use the hostname that has been set up for the operating system instance that syslog-ng is running on. To query or set this value, use the hostnamectl command.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-fetch-limit()
Type: number
Default: 100

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

max-field-size()
Type: number (characters)
Default: 65536

Description: The maximum length of a field's value.

prefix()
Type: string
Default: .journald.

Description: If this option is set, every non-built-in mapped names get a prefix (for example: ".SDATA.journald."). By default, syslog-ng PE adds the .journald. prefix to every value.

read-old-records()
Type: yes|no
Default: yes

Description: If set to yes, syslog-ng PE will start reading the records from the beginning of the journal, if the journal has not been read yet. If set to no, syslog-ng PE will read only the new records. If the source has a state in the persist file, this option will have no effect.

time-zone()
Type: name of the timezone, or the timezone offset
Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

use-fqdn()
Type: yes or no
Default: no

Description: Add Fully Qualified Domain Name instead of short hostname. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

NOTE:

This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.

systemd-syslog: Collecting systemd messages using a socket

On platforms running systemd, the systemd-syslog() driver reads the log messages of systemd using the /run/systemd/journal/syslog socket. Note the following points about this driver:

  • If possible, use the more reliable systemd-journal() driver instead.

  • The socket activation of systemd is buggy, causing some log messages to get lost during system startup.

  • If syslog-ng PE is running in a jail or a Linux Container (LXC), it will not read from the /dev/kmsg or /proc/kmsg files.

Declaration:
systemd-syslog();
Example: Using the systemd-syslog() driver
@version: 7.0

source s_systemdd {
    systemd-syslog();
};

destination d_network {
    syslog("server.host");
};

log {
    source(s_systemdd);
    destination(d_network);
};
Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating