Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.14 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations smtp: Generating SMTP messages (e-mail) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql: Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages

Configuring syslog-ng on client hosts

The following describes how to configure syslog-ng on a client host.

To configure syslog-ng on a client host

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Installing syslog-ng.

  2. Configure the local sources to collect the log messages of the host. Starting with version 3.2, syslog-ng PE automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collecting the system-specific log messages of a platform.

    Add sources to collect the messages from your log files. File sources look like this:

    source s_myfilesource {
            file("/var/log/myapplication.log" follow-freq(1)); };

    Name every source uniquely. For details on configuring file sources, see file: Collecting messages from text files.

    TIP:

    Many applications send log messages to logfiles by default (for example, the Roundcube webmail client, or the ProFTPD FTP server), but can be configured to send them to syslog instead. If possible, it is recommended to reconfigure the application that way.

    NOTE:

    The default configuration file of syslog-ng PE collects platform-specific log messages and the internal log messages of syslog-ng PE.

    source s_local {
            system();
            internal();
    };
  3. Create a network destination that points directly to the syslog-ng server, or to a local relay. The network destination greatly depends on the protocol that your log server or relay accepts messages. Many systems still use the legacy BSD-syslog protocol (RFC3162) over the unreliable UDP transport:

    destination d_network { network("10.1.2.3" transport("udp")); };

    However, if possible, use the much more reliable IETF-syslog protocol over TCP transport:

    destination d_network { syslog("10.1.2.3" transport("tcp")); };

    Make sure to use a destination that matches the source you configure on your syslog-ng server or relay. For details, see Things to consider when forwarding messages between syslog-ng PE hosts.

  4. Create a log statement connecting the local sources to the syslog-ng server or relay. For example:

    log {
            source(s_local); destination(d_network); };
  5. If the logs will also be stored locally on the host, create local file destinations.

    NOTE:

    The default configuration of syslog-ng PE places the collected messages into the /var/log/messages file:

    destination d_local {
        file("/var/log/messages"); };
  6. Create a log statement connecting the local sources to the file destination.

    NOTE:

    The default configuration of syslog-ng PE has only one log statement:

    log {
        source(s_local); destination(d_local); };
  7. Set filters, macros and other features and options (for example TLS encryption) as necessary.

    Example: The default configuration file of syslog-ng PE

    The following is the default configuration file of syslog-ng PE 7. It collects local log messages and the log messages of syslog-ng PE and saves them in the /var/log/messages file.

    @version: 7.0
    @include "scl.conf"
    source s_local { system(); internal(); };
    destination d_local {
                file("/var/log/messages"); };
    log { source(s_local); destination(d_local); };
    Example: A simple configuration for clients

    The following is a simple configuration file that collects local log messages and forwards them to a logserver using the IETF-syslog protocol.

    @version: 7.0
    @include "scl.conf"
    source s_local { system(); internal(); };
    destination d_syslog_tcp {
                 syslog("192.168.1.1" transport("tcp") port(2010)); };
    log { source(s_local);destination(d_syslog_tcp); };

    If you experience difficulties, see Troubleshooting syslog-ng for tips on solving common problems.

Configuring syslog-ng on server hosts

The following describes how to configure syslog-ng on a server host.

To configure syslog-ng on a server host

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Installing syslog-ng.

  2. Starting with version 3.2, syslog-ng PE automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collecting the system-specific log messages of a platform.

  3. Configure the network sources that collect the log messages sent by the clients and relays. How the network sources should be configured depends also on the capabilities of your client hosts: many older networking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:

    source s_network { syslog(ip(10.1.2.3) transport("udp")); };

    However, if possible, use the much more reliable TCP transport:

    source s_network { syslog(ip(10.1.2.3) transport("tcp")); };

    For other options, see syslog: Collecting messages using the IETF syslog protocol (syslog() driver) and tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol.

    NOTE:

    Starting with syslog-ng PE version 3.2, the syslog() source driver can handle both BSD-syslog (RFC 3164) and IETF-syslog (RFC 5424-26) messages.

  4. Create local destinations that will store the log messages, for example file- or program destinations. The default configuration of syslog-ng PE places the collected messages into the /var/log/messages file:

    destination d_local {
        file("/var/log/messages"); };

    If you want to create separate logfiles for every client host, use the ${HOST} macro when specifying the filename, for example:

    destination d_local {
        file("/var/log/messages_${HOST}"); };

    For details on further macros and how to use them, see Manipulating messages.

  5. Create a log statement connecting the sources to the local destinations.

    log {
            source(s_local); source(s_network); destination(d_local); };
  6. Set filters, options (for example TLS encryption) and other advanced features as necessary.

    NOTE:

    By default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server, use the keep-hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option can be set individually for every source if needed.

    If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.

    Example: A simple configuration for servers

    The following is a simple configuration file for syslog-ng Premium Edition that collects incoming log messages and stores them in a text file.

    @version: 7.0
    @include "scl.conf"
        options {
            time-reap(30);
            mark-freq(10);
            keep-hostname(yes);
            };
        source s_local { system(); internal(); };
        source s_network {
            syslog(transport(tcp));
            };
        destination d_logs {
            file(
                "/var/log/syslog-ng/logs.txt"
                owner("root")
                group("root")
                perm(0777)
                ); };
        log { source(s_local); source(s_network); destination(d_logs); };

    If you experience difficulties, see Troubleshooting syslog-ng for tips on solving common problems.

Configuring syslog-ng relays

This section describes how to configure syslog-ng PE as a relay.

Configuring syslog-ng on relay hosts

The following describes how to configure syslog-ng on a relay host.

To configure syslog-ng on a relay host

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Installing syslog-ng.

  2. Configure the network sources that collect the log messages sent by the clients.

  3. Create a network destination that points to the syslog-ng server. Make sure that you use a destination that matches the source you configured in the previous step. For details, see Things to consider when forwarding messages between syslog-ng PE hosts.

  4. Create a log statement connecting the network sources to the syslog-ng server.

  5. Configure the local sources that collect the log messages of the relay host.

  6. Create a log statement connecting the local sources to the syslog-ng server.

  7. Enable the keep-hostname() and disable the chain-hostnames() options. (For details on how these options work, see chain-hostnames().)

    NOTE:

    It is recommended to use these options on your syslog-ng PE server as well.

  8. Set filters and options (for example TLS encryption) as necessary.

    NOTE:

    By default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server, use the keep-hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option can be set individually for every source if needed.

    If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.

    Example: A simple configuration for relays

    The following is a simple configuration file that collects local and incoming log messages and forwards them to a logserver using the IETF-syslog protocol.

    @version: 7.0
    @include "scl.conf"
        options {
            time-reap(30);
            mark-freq(10);
            keep-hostname(yes);
            chain-hostnames(no);
            };
        source s_local { system(); internal(); };
        source s_network {
            syslog(transport(tcp));
            };
        destination d_syslog_tcp {
            syslog("192.168.1.5" transport("tcp") port(2010));
        };
    
        log { source(s_local); source(s_network);
              destination(d_syslog_tcp);
            };
Related Documents