Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.12 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files network: Collecting messages using the RFC3164 protocol (network() driver) osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch: Sending messages directly to Elasticsearch version 1.x elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher file: Storing messages in plain-text files hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations smtp: Generating SMTP messages (e-mail) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql: Storing messages in an SQL database syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng: Forwarding messages and tags to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages About us

Configuring syslog-ng on server hosts

Purpose:

The following describes how to configure syslog-ng on a server host.

To configure syslog-ng on a server host

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Installing syslog-ng.

  2. Starting with version 3.2, syslog-ng PE automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collecting the system-specific log messages of a platform.

  3. Configure the network sources that collect the log messages sent by the clients and relays. How the network sources should be configured depends also on the capabilities of your client hosts: many older networking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:

    source s_network { syslog(ip(10.1.2.3) transport("udp")); };

    However, if possible, use the much more reliable TCP transport:

    source s_network { syslog(ip(10.1.2.3) transport("tcp")); };

    For other options, see syslog: Collecting messages using the IETF syslog protocol (syslog() driver) and tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol.

    NOTE:

    Starting with syslog-ng PE version 3.2, the syslog() source driver can handle both BSD-syslog (RFC 3164) and IETF-syslog (RFC 5424-26) messages.

  4. Create local destinations that will store the log messages, for example file- or program destinations. The default configuration of syslog-ng PE places the collected messages into the /var/log/messages file:

    destination d_local {
        file("/var/log/messages"); };

    If you want to create separate logfiles for every client host, use the ${HOST} macro when specifying the filename, for example:

    destination d_local {
        file("/var/log/messages_${HOST}"); };

    For details on further macros and how to use them, see Manipulating messages.

  5. Create a log statement connecting the sources to the local destinations.

    log {
            source(s_local); source(s_network); destination(d_local); };
  6. Set filters, options (for example TLS encryption) and other advanced features as necessary.

    NOTE:

    By default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server, use the keep-hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option can be set individually for every source if needed.

    If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.

    Example: A simple configuration for servers

    The following is a simple configuration file for syslog-ng Premium Edition that collects incoming log messages and stores them in a text file.

    @version: 7.0
    @include "scl.conf"
        options {
            time-reap(30);
            mark-freq(10);
            keep-hostname(yes);
            };
        source s_local { system(); internal(); };
        source s_network {
            syslog(transport(tcp));
            };
        destination d_logs {
            file(
                "/var/log/syslog-ng/logs.txt"
                owner("root")
                group("root")
                perm(0777)
                ); };
        log { source(s_local); source(s_network); destination(d_logs); };

    If you experience difficulties, see Troubleshooting syslog-ng for tips on solving common problems.

Configuring syslog-ng relays

This section describes how to configure syslog-ng PE as a relay.

Configuring syslog-ng on relay hosts

Purpose:

The following describes how to configure syslog-ng on a relay host.

To configure syslog-ng on a relay host

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Installing syslog-ng.

  2. Configure the network sources that collect the log messages sent by the clients.

  3. Create a network destination that points to the syslog-ng server. Make sure that you use a destination that matches the source you configured in the previous step. For details, see Things to consider when forwarding messages between syslog-ng PE hosts.

  4. Create a log statement connecting the network sources to the syslog-ng server.

  5. Configure the local sources that collect the log messages of the relay host.

  6. Create a log statement connecting the local sources to the syslog-ng server.

  7. Enable the keep-hostname() and disable the chain-hostnames() options. (For details on how these options work, see chain-hostnames().)

    NOTE:

    It is recommended to use these options on your syslog-ng PE server as well.

  8. Set filters and options (for example TLS encryption) as necessary.

    NOTE:

    By default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server, use the keep-hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option can be set individually for every source if needed.

    If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.

    Example: A simple configuration for relays

    The following is a simple configuration file that collects local and incoming log messages and forwards them to a logserver using the IETF-syslog protocol.

    @version: 7.0
    @include "scl.conf"
        options {
            time-reap(30);
            mark-freq(10);
            keep-hostname(yes);
            chain-hostnames(no);
            };
        source s_local { system(); internal(); };
        source s_network {
            syslog(transport(tcp));
            };
        destination d_syslog_tcp {
            syslog("192.168.1.5" transport("tcp") port(2010));
        };
    
        log { source(s_local); source(s_network);
              destination(d_syslog_tcp);
            };

How relaying log messages works

Depending on your exact needs about relaying log messages, there are many scenarios and syslog-ng PE options that influence how the log message will look like on the logserver. Some of the most common cases are summarized in the following example.

Consider the following example: client-host > syslog-ng-relay > syslog-ng-server, where the IP address of client-host is 192.168.1.2. The client-host device sends a syslog message to syslog-ng-relay. Depending on the settings of syslog-ng-relay, the following can happen.

  • By default, the keep-hostname() option is disabled, so syslog-ng-relay writes the IP address of the sender host (in this case, 192.168.1.2) to the HOST field of the syslog message, discarding any IP address or hostname that was originally in the message.

  • If the keep-hostname() option is enabled on syslog-ng-relay, but name resolution is disabled (the use-dns() option is set to no), syslog-ng-relay uses the HOST field of the message as-is, which is probably 192.168.1.2.

  • To resolve the 192.168.1.2 IP address to a hostname on syslog-ng-relay using a DNS server, use the keep-hostname(no) and use-dns(yes) options. If the DNS server is properly configured and reverse DNS lookup is available for the 192.168.1.2 address, syslog-ng PE will rewrite the HOST field of the log message to client-host.

    NOTE:

    It is also possible to resolve IP addresses locally, without relying on the DNS server. For details on local name resolution, see Resolving hostnames locally.

  • The above points apply to the syslog-ng PE server (syslog-ng-server) as well, so if syslog-ng-relay is configured properly, use the keep-hostname(yes) option on syslog-ng-server to retain the proper HOST field. Setting keep-hostname(no) on syslog-ng-server would result in syslog-ng PE rewriting the HOST field to the address of the host that sent the message to syslog-ng-server, which is syslog-ng-relay in this case.

  • If you cannot or do not want to resolve the 192.168.1.2 IP address on syslog-ng-relay, but want to store your log messages on syslog-ng-server using the IP address of the original host (that is, client-host), you can enable the spoof-source() option on syslog-ng-relay. However, spoof-source() works only under the following conditions:

    • The syslog-ng PE binary has been compiled with the --enable-spoof-source option.

    • The log messages are sent using the highly unreliable UDP transport protocol. (Extremely unrecommended.)

Related Documents