Chat now with support
Chat with Support

syslog-ng Open Source Edition 3.38 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files kubernetes: Collecting and parsing the Kubernetes CRI (Container Runtime Interface) format linux-audit: Collecting messages from Linux audit logs mqtt: receiving messages from an MQTT broker network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local email messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP collectd: sending metrics to collectd discord: Sending alerts and notifications to Discord elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka (Java implementation) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) loggly: Using Loggly logmatic: Using Logmatic.io mongodb(): Storing messages in a MongoDB database mqtt() destination: sending messages from a local network to an MQTT broker network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (email) from logs snmp: Sending SNMP traps Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP Sumo Logic destinations: sumologic-http() and sumologic-syslog() syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages
Parsing syslog messages Parsing messages with comma-separated and similar values Parsing key=value pairs JSON parser XML parser Parsing dates and timestamps Python parser Parsing tags Apache access log parser Linux audit parser Cisco parser Parsing enterprise-wide message model (EWMM) messages iptables parser Netskope parser panos-parser(): parsing PAN-OS log messages Sudo parser MariaDB parser Websense parser Fortigate parser Check Point Log Exporter parser Regular expression (regexp) parser db-parser: Process message content with a pattern database (patterndb)
Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License The syslog-ng Open Source Edition Documentation License Glossary

system() source options

The system() driver has the following options:

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable syslog-ng OSE to execute external applications.

Using the hook-commands() when syslog-ng OSE starts or stops

To execute an external program when syslog-ng OSE starts or stops, use the following options:

startup()
Type: string
Default: N/A

Description: Defines the external program that is executed as syslog-ng OSE starts.

shutdown()
Type: string
Default: N/A

Description: Defines the external program that is executed as syslog-ng OSE stops.

Using the hook-commands() when syslog-ng OSE reloads

To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:

setup()
Type: string
Default: N/A

Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload.

teardown()
Type: string
Default: N/A

Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as syslog-ng OSE is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed.

source {
   network(transport(udp)
	hook-commands(
          startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
          shutdown("iptables -D LOGCHAIN 1")
        )
     );
};

systemd-journal: Collecting messages from the systemd-journal system log storage

The systemd-journal() source is used on various Linux distributions, such as RHEL (from RHEL7) and CentOS. The systemd-journal() source driver can read the structured name-value format of the journald system service, making it easier to reach the custom fields in the message. By default, syslog-ng OSE adds the .journald. prefix to the name of every parsed value. For a list and description of name-value pairs that journald provides, see the documentation of journald for your platform (for example, man systemd.journal-fields).

The systemd-journal() source driver is designed to read only local messages through the systemd-journal API. It is not possible to set the location of the journal files, or the directories.

NOTE: The log-msg-size() option is not applicable for this source. Use the max-field-size() option instead.

NOTE: This source will not handle the following cases:

  • Corrupted journal file

  • Incorrect journal configuration

  • Any other journald-related bugs

NOTE: If you are using RHEL-7, the default source in the configuration is systemd-journal() instead of unix-dgram("/dev/log") and file("/proc/kmsg"). If you are using unix-dgram("/dev/log") or unix-stream("/dev/log") in your configuration as a source, syslog-ng OSE will revert to using systemd-journal() instead.

Caution:

Only one systemd-journal() source can be configured in the configuration file. If there is more than one systemd-journal() source configured, syslog-ng OSE will not start.

Declaration:
systemd-journal(options);
Example: Sending all fields through syslog protocol using the systemd-journal() driver

To send all fields through the syslog protocol, enter the prefix in the following format: ".SDATA.<name>".

@version: 3.38

source s_journald {
    systemd-journal(prefix(".SDATA.journald."));
};

destination d_network {
    syslog("server.host");
};

log {
    source(s_journald);
    destination(d_network);
};
Example: Filtering for a specific field using the systemd-journal() driver
@version: 3.38

source s_journald {
    systemd-journal(prefix(".SDATA.journald."));
};

filter f_uid {"${.SDATA.journald._UID}" eq "1000"};

destination d_network {
    syslog("server.host");
};

log {
    source(s_journald);
    filter(f_uid);
    destination(d_network);
};
Example: Sending all fields in value-pairs using the systemd-journal() driver
@version: 3.38

source s_local {
    systemd-journal(prefix("journald."));
};

destination d_network {
    network("server.host" template("$(format_json --scope rfc5424 --key journald.*)\n"));
};

log {
    source(s_local);
    destination(d_network);
};

The journal contains credential information about the process that sent the log message. The syslog-ng OSE application makes this information available in the following macros:

Table 9: Predefined macros
Journald field syslog-ng predefined macro
MESSAGE $MESSAGE
_HOSTNAME $HOST
_PID $PID
_COMM or SYSLOG_IDENTIFIER $PROGRAM If both _COMM and SYSLOG_IDENTIFIER exists, syslog-ng OSE uses SYSLOG_IDENTIFIER
SYSLOG_FACILITY $FACILITY_NUM
PRIORITY $LEVEL_NUM

systemd-journal() source options

The systemd-journal() driver has the following options:

default-facility()
Type: facility string
Default: local0

Description: The default facility value if the SYSLOG_FACILITY entry does not exist.

default-level()
Type: string
Default: notice

Description: The default level value if the PRIORITY entry does not exist.

default-level()
Type: string
Default: notice

Description: The default level value if the PRIORITY entry does not exist.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable syslog-ng OSE to execute external applications.

Using the hook-commands() when syslog-ng OSE starts or stops

To execute an external program when syslog-ng OSE starts or stops, use the following options:

startup()
Type: string
Default: N/A

Description: Defines the external program that is executed as syslog-ng OSE starts.

shutdown()
Type: string
Default: N/A

Description: Defines the external program that is executed as syslog-ng OSE stops.

Using the hook-commands() when syslog-ng OSE reloads

To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:

setup()
Type: string
Default: N/A

Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload.

teardown()
Type: string
Default: N/A

Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as syslog-ng OSE is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed.

source {
   network(transport(udp)
	hook-commands(
          startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
          shutdown("iptables -D LOGCHAIN 1")
        )
     );
};
host-override()
Type: string
Default:

Description: Replaces the ${HOST} part of the message with the parameter string.

keep-hostname()
Type: yes or no
Default: no

Description: Enable or disable hostname rewriting.

  • If enabled (keep-hostname(yes)), syslog-ng OSE will retain the hostname information read from the systemd journal messages.

  • If disabled (keep-hostname(no)), syslog-ng OSE will use the hostname that has been set up for the operating system instance that syslog-ng is running on. To query or set this value, use the hostnamectl command.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

max-field-size()
Type: number (characters)
Default: 65536

Description: The maximum length of a field's value.

namespace()
Type: string
Default: "*"

Description: The namespace() option works exactly the same way as the respective option of the Journalctl command line tool.

The following modes of operation are available:

  • If you do not specify the namespace() option in your configuration, or if you specify an empty string, the systemd-journal() source reads and displays log data from all namespaces.

  • If you specify the namespace() option as namespace("*"), the systemd-journal() source reads and displays log data from all namespaces, interleaved.

  • If namespace(<specified-namespace>) is specified, the systemd-journal() source only reads and displays log data from the specified namespace.

  • If the namespace identifier is prefixed with "+" when you specify your namespace() option, the systemd-journal()source only reads and displays log data from the specified namespace and the default namespace, interleaved.

Syntax:namespace(string)

Example: configuration examples for using the namespace() option

The following configuration example uses the default value for the namespace() option:

source s_journal
{ 
  systemd-journal(namespace("*"));
};

The following configuration example uses a prefixed namespace identifier in the namespace() option:

source s_journal
{ 
  systemd-journal(namespace("+foobar"));
};

NOTE: Namespace support was introduced to the Journalctl command line tool in Systemd version 2.45. The syslog-ng OSE application supports the namespace() option from version 3.29. For further information about namespaces on the Systemd side, see Journal Namespaces.

prefix()
Type: string
Default: .journald.

Description: If this option is set, every non-built-in mapped names get a prefix (for example: ".SDATA.journald."). By default, syslog-ng OSE adds the .journald. prefix to every value.

read-old-records()
Type: yes|no
Default: yes

Description: If set to yes, syslog-ng OSE will start reading the records from the beginning of the journal, if the journal has not been read yet. If set to no, syslog-ng OSE will read only the new records. If the source has a state in the persist file, this option will have no effect.

time-zone()
Type: name of the timezone, or the timezone offset
Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

use-fqdn()
Type: yes or no
Default: no

Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.

TIP: Set use-fqdn() to yes if you want to use the custom-domain() global option.

NOTE: This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.

systemd-syslog: Collecting systemd messages using a socket

On platforms running systemd, the systemd-syslog() driver reads the log messages of systemd using the /run/systemd/journal/syslog socket. Note the following points about this driver:

  • If possible, use the more reliable systemd-journal() driver instead.

  • The socket activation of systemd is buggy, causing some log messages to get lost during system startup.

  • If syslog-ng OSE is running in a jail or a Linux Container (LXC), it will not read from the /dev/kmsg or /proc/kmsg files.

Declaration:
systemd-syslog();
Example: Using the systemd-syslog() driver
@version: 3.38

source s_systemdd {
    systemd-syslog();
};

destination d_network {
    syslog("server.host");
};

log {
    source(s_systemdd);
    destination(d_network);
};
Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating