Chat now with support
Chat with Support

syslog-ng Premium Edition 6.0.21 - Administration Guide

Preface Chapter 1. Introduction to syslog-ng Chapter 2. The concepts of syslog-ng Chapter 3. Installing syslog-ng Chapter 4. The syslog-ng PE quick-start guide Chapter 5. The syslog-ng PE configuration file Chapter 6. Collecting log messages — sources and source drivers Chapter 7. Sending and storing log messages — destinations and destination drivers Chapter 8. Routing messages: log paths, reliability, and filters Chapter 9. Global options of syslog-ng PE Chapter 10. TLS-encrypted message transfer Chapter 12.  Reliable Log Transfer Protocol™ Chapter 13. Reliability and minimizing the loss of log messages Chapter 14. Manipulating messages Chapter 15. Parsing and segmenting structured messages Chapter 16. Processing message content with a pattern database Chapter 17. Statistics and metrics of syslog-ng Chapter 18. Multithreading and scaling in syslog-ng PE Chapter 19. Troubleshooting syslog-ng Chapter 20. Best practices and examples

NFS file system for log files

Using the NFS network file system can lead to problems if NFS connection is not stable, therefore One Identity does neither recommend nor officially support such scenarios. If you can avoid it, do not store log files on NFS. If the NFS connection is stable and reliable, syslog-ng PE can read and write files on mounted NFS partitions as a normal file source or destination. Read this section carefully before using syslog-ng PE and NFS-mounted log files.

Risks

If there is any issue with the NFS connection (for example, connection loss, the NFS server stops), syslog-ng PE can stop working. These NFS issues can be related to the operating system, and can also vary depending on its patch level and kernel version. The possible effects include the following:

  • syslog-ng PE freezes, does not respond, does not process logs, is unable to stop or reload, and you can stop it only using the kill -9 command

  • syslog-ng PE is not able to start, and hangs during startup

  • Message loss or message duplication

  • Message becomes corrupt (it is not lost, but the message or some parts of it contain garbage)

  • When using the logstore() destination, the logstore file becomes corrupt

  • On some RHEL-based systems (possibly depending on the kernel version too), NFS returns NULL characters when reading a file that another process is writing at the very same moment.

Limitations of using syslog-ng PE with NFS

  • Do not use the logstore() destination to store files on an NFS-mounted partition

  • To use wildcards in the file source if your log files are on an NFS file system, set the force-directory-polling() option to yes to detect newly created files. Note that wildcard file sources are available only in syslog-ng PE version 6.0.3 and newer versions of the 6.x branch, and are not yet available in syslog-ng PE version 7.

  • Since One Identity does not officially support scenarios where you use syslog-ng PE together with NFS, One Identity will handle support requests and bugs related to such scenarios only if you can reproduce the issue independently from NFS.

Recommendations for using NFS with syslog-ng PE

If you cannot avoid using NFS with syslog-ng PE note the following points.

  • USE at least NFS v4 (or newer if available)

  • USE the soft mount option (-o soft) to mount the partition

  • USE the TCP mount option (-o tcp) to mount the partition

  • DO NOT install syslog-ng PE on an NFS-mounted partition

  • DO NOT store the runtime files (for example, the configuration or the persist file) of syslog-ng PE on an NFS-mounted partition

  • DO NOT use logstore on an NFS-mounted partition, it can easily become corrupted

Chapter 3. Installing syslog-ng

This chapter explains how to install syslog-ng Premium Edition on the supported platforms using the precompiled binary files.

  • The syslog-ng PE application features a unified installer package with identical look on every supported Linux and UNIX platforms. The generic installer, as well as installing platform-specific (for example, RPM) is described in the following sections.

  • For details on installing the syslog-ng Agent for Windows application, see Administration Guide for syslog-ng Agent for Windows.

  • If you want to manage your syslog-ng PE hosts using Puppet, see Procedure 3.10, “Managing syslog-ng PE from Puppet”.

The syslog-ng PE binaries include all required libraries and dependencies of syslog-ng PE, only the ncurses library is required as an external dependency (syslog-ng PE itself does not use the ncurses library, it is required only during the installation). The components are installed into the /opt/syslog-ng directory. It can automatically re-use existing configuration and license files, and also generate a simple configuration automatically into the /opt/syslog-ng/etc/syslog-ng.conf file.

 NOTE:

There are two versions of every binary release. The one with the compact suffix does not include SQL support. If you are installing syslog-ng PE in client or relay mode, or you do not use the sql() source or destination, use the compact binaries. That way no unnecessary components are installed to your system.

The syslog-ng PE application can be installed interactively following the on-screen instructions as described in the section called “Installing syslog-ng using the .run installer”, and also without user interaction using the silent installation option — see the section called “Installing syslog-ng PE without user-interaction”.

Prerequisites to installing syslog-ng PE

  • The binary installer packages of syslog-ng Premium Edition include every required dependency for most platforms, only the ncurses library is required as an external dependency (syslog-ng PE itself does not use the ncurses library, it is required only during the installation).

    		 NOTE:

    There are two versions of every binary release. The one with the compact suffix does not include SQL support. If you are installing syslog-ng PE in client or relay mode, or you do not use the sql() source or destination, use the compact binaries. That way no unnecessary components are installed to your system.

  • For Java-based destinations (for example, Elasticsearch, Apache Kafka, HDFS), Java must be installed on the host where you use such destinations. Typically, this is the host where you are running syslog-ng PE in server mode.

  • DO NOT install syslog-ng PE on an NFS-mounted partition.

  • DO NOT store the runtime files (for example, the configuration or the persist file) of syslog-ng PE on an NFS-mounted partition.

Security-enhanced Linux: grsecurity, SELinux

Security-enhanced Linux solutions such as grsecurity or SELinux can interfere with the operation of syslog-ng PE. The syslog-ng PE application supports these security enhancements as follows:

  • grsecurity: Version syslog-ng PE 5 F2 and later can be run on hosts using grsecurity, with the following limitations: using the Oracle SQL source and destination is not supported.

  • SELinux: Version syslog-ng PE 5 F2 and later properly supports SELinux on Red Hat Enterprise Linux 6.5 and newer platforms. The CentOS platforms corresponding to the supported RHEL versions are supported as well. For details, see Procedure 3.4, “Using syslog-ng PE on SELinux”.

Installing syslog-ng using the .run installer

 Caution:

If you already had syslog-ng Open Source Edition (OSE) installed on the host, and are upgrading to syslog-ng Premium Edition, make sure that the ${SYSLOGNG_OPTIONS} environmental variable does not contain a -p <path-to-pid-file> option. If it does, remove this option from the environmental variable, because it can prevent syslog-ng PE from stopping properly. Typically, the environmental variable is set in the files /etc/default/syslog-ng or /etc/sysconfig/syslog-ng, depending on the operating system you use.

This section describes how to install the syslog-ng PE application interactively using the binary installer. The installer has a simple interface: use the TAB or the arrow keys of your keyboard to navigate between the options, and Enter to select an option.

 NOTE:

The installer stops the running syslogd application if it is running, but its components are not removed. The /etc/init.d/sysklogd init script is automatically renamed to /etc/init.d/sysklogd.backup. Rename this file to its original name if you want to remove syslog-ng or restart the syslogd package.

Procedure 3.1. Installing syslog-ng PE in client or relay mode

Purpose: 

Complete the following steps to install syslog-ng Premium Edition on clients or relays. For details on the different operation modes of syslog-ng PE, see the section called “Modes of operation”.

Steps: 

NOTE:

The native logrotation tools do not send a SIGHUP to syslog-ng after rotating the log files, causing syslog-ng to write into files already rotated. To solve this problem, the syslog-ng init script links the /var/run/syslog.pid file to syslog-ng's pid. Also, on Linux, the install.sh script symlinks the initscript of the original syslog daemon to syslog-ng's initscript.

  1. Login to MyDownloads and download the syslog-ng PE installer package.

  2. Enable the executable attribute for the installer using the chmod +x syslog-ng-<edition>-<version>-<OS>-<platform>.run, then start the installer as root using the ./syslog-ng-<edition>-<version>-<OS>-<platform>.run command. (Note that the exact name of the file depends on the operating system and platform.) Wait until the package is uncompressed and the welcome screen appears, then select Continue.

    Figure 3.1. The welcome screen

    The welcome screen

  3. Accepting the EULA: You can install syslog-ng PE only if you understand and accept the terms of the End-User License Agreement (EULA). The full text of the EULA can be displayed during installation by selecting the Show EULA option, and is also available at Software Transaction, License and End User License Agreements. Select Accept to accept the EULA and continue the installation.

    If you do not accept the terms of the EULA for some reason, select Reject to cancel installing syslog-ng PE.

  4. Detecting platform and operating system: The installer attempts to automatically detect your oprating system and platform. If the displayed information is correct, select Yes. Otherwise select Exit to abort the installation, and verify that your platform is supported. For a list of supported platforms, see the section called “Supported platforms”. If your platform is supported but not detected correctly, contact our Support Team.

    Figure 3.2. Platform detection

    Platform detection

  5. Installation path: Enter the path to install syslog-ng PE to. This is useful if you intend to install syslog-ng PE without registering it as a service, or if it cannot be installed to the default location because of policy compliance reasons. If no path is given, syslog-ng PE is installed to the default folder.

    Figure 3.3. Installation path

    Installation path

    		NOTE:

    When installing syslog-ng PE to an alternative path on AIX, HP-UX, or Solaris platforms, set the CHARSETALIASDIR environmental variable to the lib subdirectory of the installation path. That way syslog-ng PE can find the charset.alias file.

  6. Registering as syslog service: Select Register to register syslog-ng PE as the syslog service. This will stop and disable the default syslog service of the system.

    Figure 3.4. Registering as syslog service

    Registering as syslog service

  7. Locating the license: Since you are installing syslog-ng PE in client or relay mode, simply select OK. For details on the different operation modes of syslog-ng PE, see the section called “Modes of operation”.

  8. Upgrading: The syslog-ng PE installer can automatically detect if you have previously installed a version of syslog-ng PE on your system. To use the configuration file of this previous installation, select Yes. To ignore the old configuration file and create a new one, select No.

    Note that if you decide to use your existing configuration file, the installer automatically checks it for syntax error and displays a list of warnings and errors if it finds any problems.

    Figure 3.5. Upgrading syslog-ng

    Upgrading syslog-ng

  9. Generating a new configuration file: The installer displays some questions to generate a new configuration file.

    1. Remote sources: Select Yes to accept log messages from the network. TCP, UDP, and SYSLOG messages on every interface will be automatically accepted.

      Figure 3.6. Accepting remote messages

      Accepting remote messages

    2. Remote destinations: Enter the IP address or hostname of your log server or relay and select OK.

      Figure 3.7. Forwarding messages to the log server

      Forwarding messages to the log server

    		NOTE:

    Accepting remote messages and forwarding them to a log server means that syslog-ng PE will start in relay mode.

  10. After the installation is finished, add the /opt/syslog-ng/bin and /opt/syslog-ng/sbin directories to your search PATH environment variable. That way you can use syslog-ng PE and its related tools without having to specify the full pathname. Add the following line to your shell profile:

    PATH=/opt/syslog-ng/bin:$PATH

  11. Optional step for SELinux-enabled systems: Complete Procedure 3.4, “Using syslog-ng PE on SELinux”.

Procedure 3.2. Installing syslog-ng PE in server mode

Purpose: 

Complete the following steps to install syslog-ng PE on log servers. For details on the different operation modes of syslog-ng PE, see the section called “Modes of operation”.

Steps: 

NOTE:

The native logrotation tools do not send a SIGHUP to syslog-ng after rotating the log files, causing syslog-ng to write into files already rotated. To solve this problem, the syslog-ng init script links the /var/run/syslog.pid file to syslog-ng's pid. Also, on Linux, the install.sh script symlinks the initscript of the original syslog daemon to syslog-ng's initscript.

  1. Login to MyDownloads and download the syslog-ng PE installer package and your syslog-ng Premium Edition license file (license.txt). The license will be required to run syslog-ng PE in server mode (see the section called “Server mode”) and is needed when you are installing syslog-ng PE on your central log server.

  2. Enable the executable attribute for the installer using the chmod +x syslog-ng-<edition>-<version>-<OS>-<platform>.run, then start the installer as root using the ./syslog-ng-<edition>-<version>-<OS>-<platform>.run command. (Note that the exact name of the file depends on the operating system and platform.) Wait until the package is uncompressed and the welcome screen appears, then select Continue.

    Figure 3.8. The welcome screen

    The welcome screen

  3. Accepting the EULA: You can install syslog-ng PE only if you understand and accept the terms of the End-User License Agreement (EULA). The full text of the EULA can be displayed during installation by selecting the Show EULA option, and is also available at Software Transaction, License and End User License Agreements. Select Accept to accept the EULA and continue the installation.

    If you do not accept the terms of the EULA for some reason, select Reject to cancel installing syslog-ng PE.

  4. Detecting platform and operating system: The installer attempts to automatically detect your oprating system and platform. If the displayed information is correct, select Yes. Otherwise select Exit to abort the installation, and verify that your platform is supported. For a list of supported platforms, see the section called “Supported platforms”. If your platform is supported but not detected correctly, contact our Support Team.

    Figure 3.9. Platform detection

    Platform detection

  5. Installation path: Enter the path to install syslog-ng PE to. This is useful if you intend to install syslog-ng PE without registering it as a service, or if it cannot be installed to the default location because of policy compliance reasons. If no path is given, syslog-ng PE is installed to the default folder.

    Figure 3.10. Installation path

    Installation path

    		NOTE:

    When installing syslog-ng PE to an alternative path on AIX, HP-UX, or Solaris platforms, set the CHARSETALIASDIR environmental variable to the lib subdirectory of the installation path. That way syslog-ng PE can find the charset.alias file.

  6. Registering as syslog service: Select Register to register syslog-ng PE as the syslog service. This will stop and disable the default syslog service of the system.

    Figure 3.11. Registering as syslog service

    Registering as syslog service

  7. Locating the license: Enter the path to your license file (license.txt) and select OK. Typically this is required only for your central log server.

    If you are upgrading an existing configuration that already has a license file, the installer automatically detects it.

    Figure 3.12. Platform detection

    Platform detection

  8. Upgrading: The syslog-ng PE installer can automatically detect if you have previously installed a version of syslog-ng PE on your system. To use the configuration file of this previous installation, select Yes. To ignore the old configuration file and create a new one, select No.

    Note that if you decide to use your existing configuration file, the installer automatically checks it for syntax error and displays a list of warnings and errors if it finds any problems.

    Figure 3.13. Upgrading syslog-ng

    Upgrading syslog-ng

  9. Generating a new configuration file: The installer displays some questions to generate a new configuration file.

    1. Remote sources: Select Yes to accept log messages from the network. TCP, UDP, and SYSLOG messages on every interface will be automatically accepted.

      Figure 3.14. Accepting remote messages

      Accepting remote messages

    2. Remote destinations: Enter the IP address or hostname of your log server or relay and select OK.

      Figure 3.15. Forwarding messages to the log server

      Forwarding messages to the log server

    		NOTE:

    Accepting remote messages and forwarding them to a log server means that syslog-ng PE will start in relay mode.

  10. After the installation is finished, add the /opt/syslog-ng/bin and /opt/syslog-ng/sbin directories to your search PATH environment variable. That way you can use syslog-ng PE and its related tools without having to specify the full pathname. Add the following line to your shell profile:

    PATH=/opt/syslog-ng/bin:$PATH

  11. Optional step for SELinux-enabled systems: Complete Procedure 3.4, “Using syslog-ng PE on SELinux”.

Installing syslog-ng PE without user-interaction

The syslog-ng PE application can be installed in silent mode without any user-interaction by specifying the required parameters from the command line. Answers to every question of the installer can be set in advance using command-line parameters.

./syslog-ng-premium-edition-<version>.run -- --silent [options]

 Caution:

The -- characters between the executable and the parameters are mandatory, like in the following example: ./syslog-ng-premium-edition-3.0.1b-solaris-10-sparc-client.run -- --silent --accept-eula -l /var/tmp/license.txt

To display the list of parameters, execute the ./syslog-ng-premium-edition-<version>.run -- --h command. Currently the following options are available:

  • --accept-eula or -a: Accept the EULA.

  • --license-file <file> or -l <file>: Path to the license file.

  • --upgrade | -u: Perform automatic upgrade — use the configuration file from an existing installation.

  • --remote <destination host>: Send logs to the specified remote server. Not available when performing an upgrade.

  • --network: Accept messages from the network. Not available when performing an upgrade.

  • --configuration <file>: Use the specified configuration file.

  • --list-installed: List information about all installed syslog-ngs.

  • --path <path>: Set installation path.

  • --register: Force service registration.

  • --no-register: Prevent service registration.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating