Chat now with support
Chat with Support

syslog-ng Open Source Edition 3.22 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local e-mail messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP collectd: sending metrics to collectd elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka (Java implementation) kafka: Publishing messages to Apache Kafka (C implementation, using the librdkafka client) loggly: Using Loggly logmatic: Using Logmatic.io mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (e-mail) from logs snmp: Sending SNMP traps Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng: Forwarding messages and tags to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages db-parser: Process message content with a pattern database (patterndb) Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License

Commercial version of syslog-ng

The syslog-ng application has a commercial version available, called syslog-ng Premium Edition (syslog-ng PE). The commercial version comes with well-tested features from its open source foundation, a number of extra features, enterprise-level support, as well as a ready-to-use log management appliance built on the strengths of syslog-ng Premium Edition.

Exclusive features related to compliance

Collecting and analyzing log messages is required directly or indirectly by several regulations, frameworks, and standards, including the Sarbanes-Oxley Act (SOX), the Health Insurance and Portability Act (HIPAA), and the Payment Card Industry Data Security Standard (PCI-DSS). syslog-ng PE provides a set of features that help you comply with regulations that require the central collection of log messages in a tamperproof way:

  • Logstore files enable you to store log messages securely in encrypted, compressed and timestamped binary files. From a compliance point of view, this serves a double purpose. Encryption guarantees the integrity of log messages so you can be sure that they have not been manipulated. Timestamping provides verifiable proof about the exact time when log messages arrived.

  • Reliable Log Transfer Protocol (RLTP) is a proprietary transport protocol that prevents message loss during connection breaks. When using this protocol, the sender detects which messages the receiver has successfully received (based on the acknowledgements returned by the receiver after having processed messages). If messages are lost during transfer, the sender resends the missing messages, starting from the last successfully received message. Therefore, messages are not duplicated at the receiving end in case of a connection break.

Wide range of supported platforms with binary installers

syslog-ng Premium Edition comes with tested binary files that are available for a wide array of server platforms, reducing the time required for installation and maintenance. Support for a wide range of operating system and hardware platforms also make syslog-ng PE an ideal choice to collect logs in massively heterogeneous environments.

Enterprise-level support services

As all commercial software, syslog-ng PE also comes with various enterprise-level support packages, which means that you get immediate and pro-active assistance (24x7 if you choose a top-tier package), dedicated to resolving your issue as soon as possible when you experience problems.

For more information about syslog-ng Premium Edition, see The syslog-ng Premium Edition Administrator Guide.

syslog-ng Store Box, ready-to-use log management appliance

syslog-ng Store Box (SSB) is a log management appliance that is built on syslog-ng Premium Edition. It is a turnkey solution to manage your log data, meaning that no software installation is necessary. As SSB is available both as a virtual machine and a physical appliance, it is also easily scalable.

SSB provides a number of features that can add value for your use cases:

  • A web GUI that makes searching logs, as well as configuring and managing SSB itself easy:

    • The search interface allows you to use wildcards and Boolean operators to perform complex searches, and drill down on the results. You can gain a quick overview and pinpoint problems fast by generating ad-hoc charts from the distribution of the log messages.

      In addition, you can easily create customized reports from the charts and statistics you create on the search interface to demonstrate compliance with standards and regulations such as PCI-DSS, ISO 27001, SOX and HIPAA.

    • Configuring SSB is done through the user interface. All of the flexible filtering, classification and routing features in the syslog-ng Open Source Edition and syslog-ng Premium Edition can be configured with it. Access and authentication policies can be set to integrate with Microsoft Active Directory, LDAP and Radius servers. The web interface is accessible through a network interface dedicated to management traffic. This management interface is also used for backups, sending alerts, and other administrative traffic.

  • High availability support to ensure continuous log collection in business-critical environments.

For further details about syslog-ng Store Box, see The syslog-ng Store Box Administrator Guide.

Upgrading from syslog-ng OSE to syslog-ng PE

If you wish to upgrade from syslog-ng OSE to syslog-ng PE, read the blog post Upgrading from syslog-ng OSE to syslog-ng PE for instructions and tips.

Installing syslog-ng

This chapter explains how to install syslog-ng Open Source Edition on various platforms.

  • You can install syslog-ng OSE on many platforms using the package manager and official repositories of the platform. For a list of third-party packages available for various Linux, UNIX, and other platforms, see the Downloads page.

  • For instructions on compiling syslog-ng Open Source Edition from the source code, see Compiling syslog-ng from source.

  • You can use a syslog-ng docker image.

    For detailed information on how to run your central log server in Docker and other Docker-related syslog-ng use cases, see the Logging in Docker using syslog-ng white paper.

Compiling syslog-ng from source

Purpose:

To compile syslog-ng Open Source Edition (OSE) from the source code, complete the following steps. Alternatively, you can use precompiled binary packages on several platforms. For a list of third-party packages available for various Linux, UNIX, and other platforms, see the Downloads page.

Steps:

  1. Download the latest version of syslog-ng OSE from GitHub. The source code is available as a tar.gz archive file.

  2. Install the following packages that are required to compile syslog-ng. These packages are available for most UNIX/Linux systems. Alternatively, you can also download the sources and compile them.

    • A version of the gcc C compiler that properly supports Thread Local Storage (TLS), for example, version 4.5.

    • The GNU flex lexical analyser generator, available here.

    • The bison parser generator, available here.

    • The development files of the glib library, available here.

    • The development files of the Autoconf Archive package, available here.

    • The syslog-ng OSE application now uses PCRE-type regular expressions by default. It requires the libpcre library package, available here.

    • If you want to use the Java-based modules of syslog-ng OSE (for example, the Elasticsearch, HDFS, or Kafka destinations), you must compile syslog-ng OSE with Java support.

      • Download and install the Java Runtime Environment (JRE), 1.7 (or newer). You can use OpenJDK or Oracle JDK, other implementations are not tested.

      • Install gradle version 2.2.1 or newer.

      • Set LD_LIBRARY_PATH to include the libjvm.so file, for example:LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH

        Note that many platforms have a simplified links for Java libraries. Use the simplified path if available. If you use a startup script to start syslog-ng OSE set LD_LIBRARY_PATH in the script as well.

      • If you are behind an HTTP proxy, create a gradle.properties under the modules/java-modules/ directory. Set the proxy parameters in the file. For details, see The Gradle User Guide.

  3. If you want to post log messages as HTTP requests using the http() destination, install the development files of the libcurl library. This library is not needed if you use the --disable-http compile option. Alternatively, you can use a Java-based implementation of the HTTP destination.

  4. If you want to use the spoof-source function of syslog-ng, install the development files of the libnet library, available here.

  5. If you want to send e-mails using the smtp() destination, install the development files of the libesmtp library. This library is not needed if you use the --disable-smtp compile option.

  6. If you want to send SNMP traps using the snmp() destination, install the development files of the Net-SNMP library libsnmp-dev. This library is not needed if you use the --disable-snmp compile option.

  7. If you want to use the /etc/hosts.deny and /etc/hosts.allow for TCP access, install the development files of the libwrap (also called TCP-wrappers) library, available here.

  8. Enter the new directory and issue the following commands. (If the ./configure file does not exist, for example, because you cloned the repository from GitHub instead of using a release tarball, execute the ./autogen.sh command.)

    $ ./configure
$ make
$ make install

  9. Uncompress the syslog-ng archive using the

    tar xvfz syslog-ng-x.xx.tar.gz

    or the

    unzip -c syslog-ng-x.xx.tar.gz | tar xvf -

    command. A new directory containing the source code of syslog-ng will be created.

  10. Enter the new directory and issue the following commands:

    $ ./configure
$ make
$ make install

    These commands will build syslog-ng using its default options.

    		 NOTE:

    • On Solaris, use gmake (GNU make) instead of make.

    • To build syslog-ng OSE with less verbose output, use the make V=0 command. This results in shorter, less verbose output, making warnings and other anomalies easier to notice. Note that silent-rules support is only available in recent automake versions.

  11. If needed, use the following options to change how syslog-ng is compiled using the following command syntax:

    $ ./configure --compile-time-option-name

    		 NOTE:

    You can also use --disable options, to explicitly disable a feature and override autodetection. For example, to disable the TCP-wrapper support, use the --disable-tcp-wrapper option. For the list of available compiling options, see Compiling options of syslog-ng OSE.

    		 Caution:

    The default linking mode of syslog-ng is dynamic. This means that syslog-ng might not be able to start up if the /usr directory is on NFS. On platforms where syslog-ng is used as a system logger, the --enable-mixed-linking is preferred.

Compiling options of syslog-ng OSE

When compiling syslog-ng OSE from source, you can use the following compiling options.

  • --enable-all-modules This option will turn on or off all modules and most features when enabled, unless a feature is explicitly disabled, or not detected automatically. Currently, this means that you must explicitly enable the pacct() source, since it is not detected automatically (all other modules are compiled automatically if the required libraries are available).

    This also means that the Sun Streams source is enabled on every platform, not only on Solaris, causing a compile error. Use --enable-all-modules together with --disable-sun-streams.

  • --disable-http Disable support for the http() destination that is based on libcurl.

  • --disable-python Disable support for Python-based modules.

  • --disable-json Disable JSON support. It also disables json-parser, and the format-cim and format-json template functions. Also, it disables JSON support even if the json-c library is installed and detected (see --enable-json).

  • --disable-smtp Disable SMTP support. By default, SMTP support is enabled if the libesmtp library is detected.

  • --disable-snmp Disable SNMP support. By default, SNMP support is enabled if the libsnmp-dev library is detected.

  • --enable-amqp Enable the amqp destination (enabled by default). The source of the RabbitMQ client is included in the source code package of syslog-ng OSE. To use an external client instead, use the --with-librabbitmq-client=system compiling option. For details on using this destination, see amqp: Publishing messages using AMQP.

  • --enable-debug Include debug information.

  • --enable-dynamic-linking Compile syslog-ng as a completely dynamic binary. If not specified syslog-ng uses mixed linking (--enable-mixed-linking): it links dynamically to system libraries and statically to everything else.

  • --enable-geoip Enable GEOIP support, required for the geoip2 template function and the geoip2-parser (enabled automatically if the libmaxminddb library is detected).

  • --enable-ipv6 Enable IPv6 support.

  • --enable-java Enable support for Java-based modules. For other requirements, see the description of the Java-based module (for example, Prerequisites) that you want to use.

  • --enable-java-modules Compile the Gradle projects of every Java module available in modules/java-modules.

  • --enable-json Enables JSON support (by default, it uses the json-c library included in the source code package of syslog-ng OSE). JSON support is required for json-parser, and the format-cim and format-json template functions.

  • --enable-linux-caps Enable support for capabilities on Linux. For details, see The syslog-ng manual page.

  • --enable-mongodb Enable the mongodb destination (enabled by default). The source of the MongoDB client is included in the source code package of syslog-ng OSE. To use an external MongoDB client instead, use the --with-libmongo-client=system compiling option. For details on using this destination, see mongodb: Storing messages in a MongoDB database.

  • --enable-pacct Enable using the pacct() driver to collect process-accounting logs on Linux systems.

  • --enable-python Enable support for Python-based modules.

  • --enable-redis Enable the redis destination (enabled by default). The source of the libhiredis client (0.11 or newer) must be available. To specify the location of the library, use the --with-libhiredis=<path-to-libhiredis> compiling option. For details on using this destination, see redis: Storing name-value pairs in Redis.

  • --enable-riemann Enable the riemann destination (enabled by default). The source of the libriemann client must be available. For details on using this destination, see riemann: Monitoring your data with Riemann.

  • --enable-snmp-dest Enable SNMP support even if not detected (autodetected by default).

  • --enable-spoof-source Enable spoof_source feature (disabled by default).

  • --enable-sql Enables the sql() destination (enabled automatically if the libdbi library version 0.9 or newer is installed and detected).

  • --enable-ssl Enable SSL support, required for encrypted message transfer, as well as template functions that calculate hashes and UUIDs (enabled automatically if the libopenssl library is detected).

  • --enable-sun-door Enable Sun door support even if not detected (autodetected by default).

  • --enable-sun-streams Enable Sun STREAMS support even if not detected (autodetected by default).

  • --enable-systemd Enable systemd support on Linux platforms (autodetected by default) (enabled automatically if the libsystemd-daemon library is detected).

  • --enable-tcp-wrapper Enable using /etc/hosts.deny and /etc/hosts.allow for TCP access (enabled automatically if the libwrap libraries are detected).

  • --with-embedded-crypto If this option is set, the crypto library is linked directly into libsyslog-ng: the sources of libsyslog-ng-crypto will be appended to the libsyslog-ng sources, and -crypto is not built.

  • --with-ivykis Specifies which ivykis implementation to use (default value: internal). The source of ivykis is included in the source code package of syslog-ng OSE and is used by default. To use an external implementation instead, use the --with-ivykis=system compiling option.

  • --with-libcurl Specifies the path to the libcurl library. For details on using this destination, see http: Posting messages over HTTP without Java.

  • --with-libhiredis Specifies the path to the libhiredis library (0.11 or newer). For details on using this destination, see redis: Storing name-value pairs in Redis.

  • --with-libmongo-client Specifies which MongoDB client to use (default value: internal). The source of the mongodb client is included in the source code package of syslog-ng OSE. To use an external MongoDB client instead, use the --with-libmongo-client=system compiling option. For details on using this destination, see mongodb: Storing messages in a MongoDB database.

  • --with-librabbitmq-client Specifies which RabbitMQ client to use (default value: internal). The source of the rabbitmq client is included in the source code package of syslog-ng OSE and is used by default. To use an external client instead, use the --with-librabbitmq-client=system compiling option. For details on using this destination, see amqp: Publishing messages using AMQP.

  • --with-module-dir Specifies a single directory where the syslog-ng OSE Makefile will install the modules.

  • --module-install-dir Specifies syslog-ng OSE's module installation directory (normally $prefix/lib/syslog-ng). All Java-based SCLs use this option.

  • --with-module-path Specifies a colon-separated (:) list of directories, where the syslog-ng OSE binary will search for modules.

  • --with-net-snmp Specifies the path to the libsnmp-dev library, required for the snmp() destination.

  • --with-python Specifies which Python version to use, for example, --with-python=2.7

  • --with-timezone-dir Specifies the directory where syslog-ng looks for the timezone files to resolve the time-zone() and local-time-zone() options. If not specified, the /opt/syslog-ng/share/zoneinfo/ and /usr/share/zoneinfo/ directories are checked, respectively. Note that HP-UX uses a unique file format (tztab) to describe the timezone information, but that format is currently not supported in syslog-ng. As a workaround, copy the zoneinfo files from another, non-HP-UX system to the /opt/syslog-ng/share/zoneinfo/ directory of your HP-UX system.

  • --without-compile-date Removes the compilation date from the binary. For example, as openSUSE checks if recompilation changes the binary to detect if dependent packages need to be rebuilt or not, and including the date changes the binary every time.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating