Chat now with support
Chat with Support

syslog-ng Open Source Edition 3.30 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local email messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP collectd: sending metrics to collectd elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka (Java implementation) kafka: Publishing messages to Apache Kafka (C implementation, using the librdkafka client) loggly: Using Loggly logmatic: Using Logmatic.io mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (email) from logs snmp: Sending SNMP traps Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP Sumo Logic destinations: sumologic-http() and sumologic-syslog() syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages db-parser: Process message content with a pattern database (patterndb) Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License Glossary

graphite: Sending metrics to Graphite

The graphite() destination can send metrics to a Graphite server to store numeric time-series data. There are many ways to feed the Graphite template function with name value pairs. The syslog-ng OSE CSV and PatternDB parsers (for details, see Using pattern parsers) can parse log messages and generate name value pairs based on message content. The CSV parser (for details, see Parsing messages with comma-separated and similar values) can be used for logs that have a constant field based structure, like the Apache web server access logs. The patterndb parser can parse information and can extract important fields from free form log messages, as long as patterns describing the log messages are available. Another way is to send JSON-based log messages (for details, see JSON parser) to syslog-ng OSE, like running a simple shell script collecting metrics and running it from cron regularly.

To see an example of how the graphite() destination is used to collect statistics coming from syslog-ng, see the blog post Collecting syslog-ng statistics to Graphite.

Declaration:
graphite(payload());
Example: Using the graphite() driver

To use the graphite() destination, the only mandatory parameter is payload, which specifies the value pairs to send to graphite. In the following example any value pairs starting with "monitor." are forwarded to graphite.

destination d_graphite { graphite(payload("--key monitor.*")); };

NOTE: The graphite() destination is only a wrapper around the network() destination and the graphite-output template function. If you want to fine-tune the TCP parameters, use the network() destination instead, as described in graphite-output.

graphite() destination options

The graphite() destination has the following options:

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable syslog-ng OSE to execute external applications.

Using the hook-commands() when syslog-ng OSE starts or stops

To execute an external program when syslog-ng OSE starts or stops, use the following options:

startup()
Type: string
Default: N/A

Description: Defines the external program that is executed as syslog-ng OSE starts.

shutdown()
Type: string
Default: N/A

Description: Defines the external program that is executed as syslog-ng OSE stops.
Using the hook-commands() when syslog-ng OSE reloads

To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:

setup()
Type: string
Default: N/A

Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload.
teardown()
Type: string
Default: N/A

Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload.
Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as syslog-ng OSE is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed.

source {
   network(transport(udp)
	hook-commands(
          startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
          shutdown("iptables -D LOGCHAIN 1")
        )
     );
};
host()
Type: hostname or IP address
Default: localhost

Description: The hostname or IP address of the Graphite server.

port()
Type: number
Default: 2003

Description: The port number of the Graphite server.

payload()
Type: parameter list of the payload() option
Default: 
empty string

Description: The payload() option allows you to select which value pairs to forward to graphite.

The syntax of payload is different from the syntax of value-pairs(): use the command-line syntax used in the format-json template function. For details on using the payload() option, see graphite-output.

NOTE: If left empty, there is no data to be forwarded to Graphite.

Sending logs to Graylog

graylog2(): Sending logs to Graylog

You can use the graylog2() destination and a Graylog Extended Log Format (GELF) template to send syslog messages to Graylog.

You can forward simple name-value pairs where the name starts with a dot or underscore. If names of your name-value pairs include dots other than the first character, you should use JSON formatting directly instead of the GELF template and send logs to a raw TCP port in Graylog, which can then extract fields from nested JSON. Version 3.21 and later also supports TLS-encrypted connection to the Graylog server.

Declaration:
graylog2();
Example: Using the graylog2() driver

You can send syslog messages to Graylog using the graylog2() destination. The graylog2() destination uses the GELF template, the native data format of Graylog.

  1. On the Graylog side, configure a GELF TCP input. For more information, see the relevant Graylog documentation.

  2. On the syslog-ng side, configure the name or IP address of the host running Graylog.

    destination d_graylog {
  graylog2(
    host("172.16.146.142")
    transport(tcp)
  );
};

    If you parsed your messages using syslog-ng, the template also forwards any name-value pairs where the name starts with a dot or underscore.

NOTE: If there is a dot in a field name other than the first character, syslog-ng creates nested JSON while formatting the message. Nested JSON is not automatically parsed in GELF messages.

Sending nested JSON to Graylog

While sending nested JSON inside GELF is possible, it is not convenient. If you use parsing and normalization in syslog-ng and dot notation in field names, use pure JSON instead of GELF to forward your messages.

  1. On the Graylog side, create a new raw TCP input.

  2. Still in Graylog, once the raw TCP input is ready, add a JSON extractor to it.

  3. On the syslog-ng side, use a network destination combined with a template utilizing format-json as shown in the example below:

    destination d_jsontcp {
  network(
    "172.16.146.142"
    port("5555")
    transport(tcp)
    template("$(format-json --scope all-nv-pairs)\n")
  );
};

graylog() destination options

The graylog2() destination has the following options:

ca-dir()
Accepted values: Directory name
Default: none

Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject's name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the syslog-ng clients. The syslog-ng OSE application uses the CA certificates in this directory to validate the certificate of the peer.

This option can be used together with the optional ca-file() option.

ca-file()
Accepted values: File name
Default: empty

Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The syslog-ng OSE application uses the CA certificates in this file to validate the certificate of the peer.

Example format in configuration:

ca-file("/etc/pki/tls/certs/ca-bundle.crt")

NOTE: The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable syslog-ng OSE to execute external applications.

Using the hook-commands() when syslog-ng OSE starts or stops

To execute an external program when syslog-ng OSE starts or stops, use the following options:

startup()
Type: string
Default: N/A

Description: Defines the external program that is executed as syslog-ng OSE starts.

shutdown()
Type: string
Default: N/A

Description: Defines the external program that is executed as syslog-ng OSE stops.
Using the hook-commands() when syslog-ng OSE reloads

To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:

setup()
Type: string
Default: N/A

Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload.
teardown()
Type: string
Default: N/A

Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload.
Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as syslog-ng OSE is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed.

source {
   network(transport(udp)
	hook-commands(
          startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
          shutdown("iptables -D LOGCHAIN 1")
        )
     );
};
tls()
Type: tls options
Default: n/a

Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.

transport()
Type: udp, tcp, or tls
Default: tcp

Description: Specifies the protocol used to send messages to the destination server.

If you use the udp transport, syslog-ng OSE automatically sends multicast packets if a multicast destination address is specified. The tcp transport does not support multicasting.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating