Chat now with support
Chat with Support

syslog-ng Premium Edition 6.0.18 - Administration Guide

Preface Chapter 1. Introduction to syslog-ng Chapter 2. The concepts of syslog-ng Chapter 3. Installing syslog-ng Chapter 4. The syslog-ng PE quick-start guide Chapter 5. The syslog-ng PE configuration file Chapter 6. Collecting log messages — sources and source drivers Chapter 7. Sending and storing log messages — destinations and destination drivers Chapter 8. Routing messages: log paths, reliability, and filters Chapter 9. Global options of syslog-ng PE Chapter 10. TLS-encrypted message transfer Chapter 11. FIPS-compliant syslog-ng Chapter 12.  Reliable Log Transfer Protocol™ Chapter 13. Reliability and minimizing the loss of log messages Chapter 14. Manipulating messages Chapter 15. Parsing and segmenting structured messages Chapter 16. Processing message content with a pattern database Chapter 17. Statistics and metrics of syslog-ng Chapter 18. Multithreading and scaling in syslog-ng PE Chapter 19. Troubleshooting syslog-ng Chapter 20. Best practices and examples

Installing syslog-ng PE using .pkg installer

The syslog-ng PE application can be installed with user-interaction or in silent mode.

The generic syslog service of the operating system is stopped and deregistered automatically by the installer before installation.

Caution:

Remove any other syslog service before starting the installation. Failing to do so might have unpredictable results.

The syslog-ng PE application is compatible with the standard Solaris logrotate mechanism by default.

Procedure 3.6. Installing syslog-ng PE with user-interaction

Purpose: 

The syslog-ng PE application can be installed in an interactive way.

Steps: 

  1. Unpack the package with the following command:

    gunzip syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg.gz
  2. Run the following command:

    pkgadd -d syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg BBsyslng
  3. Answer the questions with y or n. Besides these, the following commands are available: ? displays the help, q exits the installer. The installer generates a default configuration file for syslog-ng PE based on the answers (if an old configuration file is not used):

    • the path of the license file

    • whether an old configuration file is to be used (if the answer is yes, the rest of the questions will be skipped)

    • if a new configuration file is requested, whether syslog-ng PE will send or receive logs over the network

  4. The installer finishes the installation and the replaces the default syslog service with syslog-ng PE.

Procedure 3.7. Installing syslog-ng PE without user-interaction

Purpose: 

The syslog-ng PE application can be installed in silent mode without any user-interaction by specifying the required parameters from the command line. Answers to every question of the installer can be set in advance using command-line parameters.

Steps: 

  1. Generate a response file that contains the answers to the questions.

    pkgask -r <responsefile> -d syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg BBsyslng

    For example:

    pkgask -r response_file -d syslog-ng-premium-edition-4.2.1-solaris10-amd64.pkg BBsyslng

    Caution:

    The pkgask will fail if the response file already exists.

  2. Run the pkgadd command with the added response file answering y to all questions.

    yes | pkgadd -r <responsefile> -d syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg BBsyslng

    Caution:

    pkgask runs the request script of the package under the current condition of the operating system. The installation might fail if something changes (another syslog-ng PE package has been installed, configuration file has been removed), because the response file's contents will be invalid.

Procedure 3.8. Installing syslog-ng PE from a transformed PKG package

Purpose: 

The syslog-ng PE application can be installed from a normal file structure. To perform this, transform the .pkg package.

Steps: 

  1. To transform the .pkg package, execute the following command:

    pkgtrans syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg <outdirectory>

    In this case, the .pkg package will be extracted to the <outdirectory> directory.

    Example 3.1. Extracting syslog-ng PE from a transformed PKG package

    pkgtrans syslog-ng-premium-edition-4.2.0-solaris-9-sparc.pkg /tmp/out

    This will extract the files to the /tmp/out/BBsyslng directory.


  2. To install syslog-ng PE, execute the following command:

    pkgadd <options> -d <outdirectory> BBsyslng

    In this case, <options> stand for the pkgadd options, for example -r <responsefile>.

Upgrading syslog-ng PE

This section describes the possible upgrade paths of syslog-ng PE.

Upgrading from previous syslog-ng PE versions to 6 LTS

Upgrading is supported from the following syslog-ng PE versions:

  • syslog-ng PE 5 LTS (5.0.x)

  • syslog-ng PE 5 F6 (5.6.x)

To upgrade an existing syslog-ng PE installation, see Procedure 3.11, “Upgrading to syslog-ng PE 6 LTS”.

Procedure 3.11. Upgrading to syslog-ng PE 6 LTS

Purpose: 

To upgrade to syslog-ng PE 6 LTS, complete the following steps:

Steps: 

  1. Download the new installer package. Use the same package type as you used for the installation (for example, use the .run package for the upgrade if you have originally installed syslog-ng PE using a .run installer).

  2. Install syslog-ng PE and check the warnings. Upgrade the respective parts of your configuration if needed.

  3. Set the version of the configuration file to 6.0.

Upgrading syslog-ng PE to other package versions

This scenario is not supported and will fail with the following error messages.

Upgrading from platform-specific package to .run

Upgrading from rpm package to .run package. 

Unsupported. Installation stops and the following error message is displayed:

Incompatible syslog-ng package already installed

Upgrading from deb package to .run package. 

Unsupported. Installation stops and the following error message is displayed:

Incompatible syslog-ng package already installed

Upgrading from pkg package to .run package. 

Unsupported. Installation stops and the following error message is displayed:

Incompatible syslog-ng package already installed to <syslog-ng path>

Upgrading from .run to a platform-specific package

This scenario is not supported and will fail with the following error messages. To replace a .run package with a platform-specific package, create a backup of your configuration and persist files, uninstall the .run package using the --purge option, then install the platform-specific package.

Upgrading from .run package to rpm package. 

Unsupported. Installation stops and the following error message is displayed:

Incompatible standalone (.run) installer of syslog-ng Premium Edition

Caution:

Hazard of data loss! Installing rpm package syslog-ng PE on AIX platform is possible even if the upgrade conditions are not met, since the rpm package installs before checking the upgrade conditions and therefore no error message is displayed. This might result in overwriting the old configuration file.

Upgrading from .run package to deb package. 

Unsupported. Installation stops and the following error message is displayed:

Errors were encountered while processing

Upgrading from .run package to pkg package. 

Unsupported. Installation stops and the following error message is displayed:

Please remove the conflicting package before installing this package. Installation aborted.

Upgrading from syslog-ng PE to syslog-ng OSE

Upgrading from syslog-ng PE to syslog-ng OSE is unsupported since it counts as downgrading.

Upgrading from complete syslog-ng PE to client setup version of syslog-ng PE

The installer displays the following message if you try to upgrade from complete syslog-ng PE to client setup syslog-ng PE with .run package.

This version of syslog-ng Premium Edition doesn't support storing messages in SQL servers, while the installed one did.

Uninstalling syslog-ng PE

If you need to uninstall syslog-ng PE for some reason, you have the following options:

  • If you have installed syslog-ng PE using the .run installer: Execute the uninstall.sh script located at /opt/syslog-ng/bin/uninstall.sh. The uninstall script will automatically restore the syslog daemon used before installing syslog-ng. To completely remove syslog-ng PE, including the configuration files, use the uninstall.sh --purge command.

  • If you have installed syslog-ng PE from a .deb package: Execute the dpkg -r syslog-ng-premium-edition command to remove syslog-ng, or the dpkg -P syslog-ng-premium-edition command to remove syslog-ng PE and the configuration files as well. Note that removing syslog-ng PE does not restore the syslog daemon used before syslog-ng.

  • If you have installed syslog-ng PE from an .rpm package: Execute the rpm -e syslog-ng-premium-edition command to remove syslog-ng PE. Note that removing syslog-ng PE does not restore the syslog daemon used before syslog-ng PE.

  • If you have installed syslog-ng PE from a .pkg package: Execute the pkgrm BBsyslng command to remove syslog-ng PE. Note that removing syslog-ng PE does not restore the syslog daemon used before syslog-ng.

    For automatic uninstall (answering y to all questions): Execute the yes | pkgrm BBsyslng command.

    The following files have to be deleted manually:

    • <syslog-ng path>/etc/syslog-ng.conf

    • <syslog-ng path>/var/syslog-ng.persist

    • <syslog-ng path>/var/syslog-ng-00000.qf

    • anything else under the <syslog-ng path>/var directory

Chapter 4. The syslog-ng PE quick-start guide

This chapter provides a very brief introduction into configuring the syslog-ng PE application. For details on the format of the configuration file and how to configure sources, destinations, and other features, refer to the subsequent chapters.

Procedure 4.1. Configuring syslog-ng on client hosts

Purpose: 

To configure syslog-ng on a client host, complete the following steps.

Steps: 

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Chapter 3, Installing syslog-ng.

  2. Configure the local sources to collect the log messages of the host. Starting with version 3.2, syslog-ng PE automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see the section called “Collecting the system-specific log messages of a platform”.

    Add sources to collect the messages from your log files. File sources look like this:

    source s_myfilesource {
            file("/var/log/myapplication.log" follow-freq(1)); };

    Name every source uniquely. For details on configuring file sources, see the section called “Collecting messages from text files”.

    TIP:

    Many applications send log messages to logfiles by default (for example, the Roundcube webmail client, or the ProFTPD FTP server), but can be configured to send them to syslog instead. If possible, it is recommended to reconfigure the application that way.

    NOTE:

    The default configuration file of syslog-ng PE collects platform-specific log messages and the internal log messages of syslog-ng PE.

    source s_local {
            system();
            internal();
    };
  3. Create a network destination that points directly to the syslog-ng server, or to a local relay. The network destination greatly depends on the protocol that your log server or relay accepts messages. Many systems still use the legacy BSD-syslog protocol (RFC3162) over the unreliable UDP transport:

    destination d_network { network("10.1.2.3" transport("udp")); };

    However, if possible, use the much more reliable IETF-syslog protocol over TCP transport:

    destination d_network { syslog("10.1.2.3" transport("tcp")); };
  4. Create a log statement connecting the local sources to the syslog-ng server or relay. For example:

    log {
            source(s_local); destination(d_network); };
  5. If the logs will also be stored locally on the host, create local file destinations.

    NOTE:

    The default configuration of syslog-ng PE places the collected messages into the /var/log/messages file:

    destination d_local {
        file("/var/log/messages"); };
  6. Create a log statement connecting the local sources to the file destination.

    NOTE:

    The default configuration of syslog-ng PE has only one log statement:

    log {
        source(s_local); destination(d_local); };
  7. Set filters, macros and other features and options (for example TLS encryption) as necessary.

    Example 4.1. The default configuration file of syslog-ng PE

    The following is a simple configuration file that collects local log messages to the /var/log/messages file.

    @version: 6.0
    @include "scl.conf"
    source s_local { system(); internal(); };
    destination d_local {
                file("/var/log/messages"); };
    log { source(s_local); destination(d_local); };

    Example 4.2. A simple configuration for clients

    The following configuration file collects local log messages and the log messages of syslog-ng PE, and forwards them to a log server using the IETF-syslog protocol.

    @version: 6.0
    @include "scl.conf"
    source s_local { system(); internal(); };
    destination d_syslog_tcp {
                 syslog("192.168.1.1" transport("tcp") port(2010)); };
    log { source(s_local);destination(d_syslog_tcp); };

    If you experience difficulties, see Chapter 19, Troubleshooting syslog-ng for tips on solving common problems.

Procedure 4.2. Configuring syslog-ng on server hosts

Purpose: 

To configure syslog-ng on a server host, complete the following steps.

Steps: 

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Chapter 3, Installing syslog-ng.

  2. Starting with version 3.2, syslog-ng PE automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see the section called “Collecting the system-specific log messages of a platform”.

  3. Configure the network sources that collect the log messages sent by the clients and relays. How the network sources should be configured depends also on the capabilities of your client hosts: many older networking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:

    source s_network { syslog(ip(10.1.2.3) transport("udp")); };

    However, if possible, use the much more reliable TCP transport:

    source s_network { syslog(ip(10.1.2.3) transport("tcp")); };

    For other options, see the section called “Collecting messages using the IETF syslog protocol (syslog() driver)” and the section called “Collecting messages from remote hosts using the BSD syslog protocol”.

    NOTE:

    Starting with syslog-ng PE version 3.2, the syslog() source driver can handle both BSD-syslog (RFC 3164) and IETF-syslog (RFC 5424-26) messages.

  4. Create local destinations that will store the log messages, for example file- or program destinations. The default configuration of syslog-ng PE places the collected messages into the /var/log/messages file:

    destination d_local {
        file("/var/log/messages"); };

    If you want to create separate logfiles for every client host, use the ${HOST} macro when specifying the filename, for example:

    destination d_local {
        file("/var/log/messages_${HOST}"); };

    For details on further macros and how to use them, see Chapter 14, Manipulating messages.

  5. Create a log statement connecting the sources to the local destinations.

    log {
            source(s_local); source(s_network); destination(d_local); };
  6. Set filters, options (for example TLS encryption) and other advanced features as necessary.

    NOTE:

    By default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server, use the keep-hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option can be set individually for every source if needed.

    If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.

    Example 4.3. A simple configuration for servers

    The following is a simple configuration file for syslog-ng Premium Edition that collects incoming log messages and stores them in a text file.

    @version: 6.0
    @include "scl.conf"
        options {
            time-reap(30);
            mark-freq(10);
            keep-hostname(yes);
            };
        source s_local { system(); internal(); };
        source s_network {
            syslog(transport(tcp));
            };
        destination d_logs {
            file(
                "/var/log/syslog-ng/logs.txt"
                owner("root")
                group("root")
                perm(0777)
                ); };
        log { source(s_local); source(s_network); destination(d_logs); };

    If you experience difficulties, see Chapter 19, Troubleshooting syslog-ng for tips on solving common problems.

Configuring syslog-ng relays

This section describes how to configure syslog-ng PE as a relay.

Procedure 4.3. Configuring syslog-ng on relay hosts

Purpose: 

To configure syslog-ng on a relay host, complete the following steps:

Steps: 

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Chapter 3, Installing syslog-ng.

  2. Configure the network sources that collect the log messages sent by the clients.

  3. Create a network destination that points to the syslog-ng server.

  4. Create a log statement connecting the network sources to the syslog-ng server.

  5. Configure the local sources that collect the log messages of the relay host.

  6. Create a log statement connecting the local sources to the syslog-ng server.

  7. Enable the keep-hostname() and disable the chain-hostnames() options. (For details on how these options work, see the section called “chain-hostnames()”.)

    NOTE:

    It is recommended to use these options on your syslog-ng PE server as well.

  8. Set filters and options (for example TLS encryption) as necessary.

    NOTE:

    By default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server, use the keep-hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option can be set individually for every source if needed.

    If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.

    Example 4.4. A simple configuration for relays

    The following is a simple configuration file that collects local and incoming log messages and forwards them to a logserver using the IETF-syslog protocol.

    @version: 6.0
    @include "scl.conf"
        options {
            time-reap(30);
            mark-freq(10);
            keep-hostname(yes);
            chain-hostnames(no);
            };
        source s_local { system(); internal(); };
        source s_network {
            syslog(transport(tcp));
            };
    
        destination d_syslog_tcp {
            syslog("192.168.1.5" transport("tcp") port(2010));
        };
    
        log { source(s_local); source(s_network);
              destination(d_syslog_tcp);
            };

How relaying log messages works

Depending on your exact needs about relaying log messages, there are many scenarios and syslog-ng PE options that influence how the log message will look like on the log server. Some of the most common cases are summarized in the following example.

Consider the following example: client-host > syslog-ng-relay > syslog-ng-server, where the IP address of client-host is 192.168.1.2. The client-host device sends a syslog message to syslog-ng-relay. Depending on the settings of syslog-ng-relay, the following can happen.

  • By default, the keep-hostname() option is disabled, so syslog-ng-relay writes the IP address of the sender host (in this case, 192.168.1.2) to the HOST field of the syslog message, discarding any IP address or hostname that was originally in the message.

  • If the keep-hostname() option is enabled on syslog-ng-relay, but name resolution is disabled (the use-dns() option is set to no), syslog-ng-relay uses the HOST field of the message as-is, which is probably 192.168.1.2.

  • To resolve the 192.168.1.2 IP address to a hostname on syslog-ng-relay using a DNS server, use the keep-hostname(no) and use-dns(yes) options. If the DNS server is properly configured and reverse DNS lookup is available for the 192.168.1.2 address, syslog-ng PE will rewrite the HOST field of the log message to client-host.

    NOTE:

    It is also possible to resolve IP addresses locally, without relying on the DNS server. For details on local name resolution, see Procedure 20.1, “Resolving hostnames locally”.

  • The above points apply to the syslog-ng PE server (syslog-ng-server) as well, so if syslog-ng-relay is configured properly, use the keep-hostname(yes) option on syslog-ng-server to retain the proper HOST field. Setting keep-hostname(no) on syslog-ng-server would result in syslog-ng PE rewriting the HOST field to the address of the host that sent the message to syslog-ng-server, which is syslog-ng-relay in this case.

  • If you cannot or do not want to resolve the 192.168.1.2 IP address on syslog-ng-relay, but want to store your log messages on syslog-ng-server using the IP address of the original host (that is, client-host), you can enable the spoof-source() option on syslog-ng-relay. However, spoof-source() works only under the following conditions:

    • The syslog-ng PE binary has been compiled with the --enable-spoof-source option.

    • The log messages are sent using the highly unreliable UDP transport protocol. (Extremely unrecommended.)

Related Documents