Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.24 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng PE The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files google-pubsub: collecting messages from the Google Pub/Sub messaging service wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mssql, oracle, sql: collecting messages from an SQL database network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files google_pubsub: Sending logs to the Google Cloud Pub/Sub messaging service hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel: Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql: Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

Risks of using syslog-ng PE with NFS or CIFS (or SMB) file system

This section describes the risks of using syslog-ng PE with NFS or CIFS file system.

NOTE: All information referring to the CIFS network file system in this section automatically applies to the SMB network file system as well. As a result, even if the content of this section only refers to the CIFS file system, keep in mind that the same information also applies to SMB file systems as well.

Caution:

Hazard of data loss!

The syslog-ng PE application does not support storing the disk-buffer files on any kind of network share (that is, NFS, CIFS, and so on). To avoid crashes, data corruption, or data loss, One Identity does not recommend storing your disk-buffer files on network share.

If there is any issue with the NFS or CIFS connection (for example, connection loss, the NFS or CIFS server stops), syslog-ng PE can stop working. These NFS-related or CIFS-related issues can be related to the operating system, and can also vary depending on the operating system's patch level and kernel version. The possible effects include the following:

  • The syslog-ng PE application freezes, does not respond, does not process logs, is unable to stop or reload, and you can stop it only using the kill -9 command.

  • The syslog-ng PE application cannot start, and hangs during startup.

  • Message loss or message duplication.

  • Message becomes corrupt (it is not lost, but the message or some parts of it contain garbage).

  • When using the logstore() destination, the logstore file becomes corrupt.

  • On some RHEL-based systems (possibly depending on the kernel version too), NFS returns NULL characters when reading a file that another process is writing at the very same moment.

Recommendations for using syslog-ng PE with NFS or CIFS (or SMB) file system

This section lists the recommendations for using syslog-ng Premium Edition (syslog-ng PE) with NFS or CIFS file system.

NOTE: All information referring to the CIFS network file system in this section automatically applies to the SMB network file system as well. As a result, even if the content of this section only refers to the CIFS file system, keep in mind that the same information also applies to SMB file systems as well.

Caution:

When you use NFS with syslog-ng PE, One Identity recommends the following:

  • DO NOT install syslog-ng PE on an NFS-mounted partition.

  • DO NOT store the runtime files (for example, the configuration or the persist file) of syslog-ng PE on an NFS-mounted partition.

  • DO NOT use a logstore on an NFS-mounted partition. It can easily become corrupted.

Caution:

When you use CIFS with syslog-ng PE, One Identity recommends the following:

  • DO NOT install syslog-ng PE on a CIFS-mounted partition.

  • DO NOT store the runtime files (for example, the configuration or the persist file) of syslog-ng PE on a CIFS-mounted partition.

  • DO NOT use a logstore on a CIFS-mounted partition. It can easily become corrupted.

If you cannot avoid using NFS with syslog-ng PE, note the following points.

  • USE at least NFS v4 (or newer if available).

  • USE the soft mount option (-o soft) to mount the partition.

  • USE the TCP mount option (-o tcp) to mount the partition.

Installing syslog-ng PE

This chapter explains how to install syslog-ng Premium Edition on the supported platforms using the precompiled binary files.

  • The syslog-ng PE application features a unified installer package with identical look on every supported Linux and UNIX platforms. The generic installer, as well as platform-specific installers, for example, RPM, is described in the following sections.

The syslog-ng PE binaries include all required libraries and dependencies of syslog-ng PE. Only the ncurses library is required as an external dependency (syslog-ng PE itself does not use the ncurses library, it is required only during the installation). The components are installed into the /opt/syslog-ng directory. It can automatically reuse existing configuration and license files, and also generate a simple configuration automatically into the /opt/syslog-ng/etc/syslog-ng.conf file.

NOTE: There are two versions of every binary release. The one with the compact suffix does not include SQL support. If you are installing syslog-ng PE in client or relay mode, or you do not use the sql() source or destination, use the compact binaries. That way no unnecessary components are installed to your system.

The syslog-ng PE application can be installed interactively following the on-screen instructions as described in Installing syslog-ng using the .run installer, and also without user interaction using the silent installation option — see Installing syslog-ng PE without user-interaction.

NOTE: When setting up a virtual environment, carefully consider the configuration aspects such as CPU, memory availability, I/O subsystem, and network infrastructure to ensure the virtual layer has the necessary resources available. Please consult One Identity's Product Support Policies for more information on environment virtualization.

Prerequisites to installing syslog-ng PE

  • The binary installer packages of syslog-ng Premium Edition (syslog-ng PE) include every required dependency for most platforms, only the ncurses library is required as an external dependency (syslog-ng PE itself does not use the ncurses library, it is required only during the installation).

    NOTE: There are two versions of every binary release. The one with the compact suffix does not include SQL support. If you are installing syslog-ng PE in client or relay mode, or you do not use the sql() source or destination, use the compact binaries. That way no unnecessary components are installed to your system.

  • For Java-based destinations (for example, Elasticsearch, Apache Kafka, HDFS), Java must be installed on the host where you use such destinations. Typically, this is the host where you are running syslog-ng PE in server mode.

  • DO NOT install syslog-ng PE on an NFS-mounted partition.

  • DO NOT store the runtime files (for example, the configuration or the persist file) of syslog-ng PE on an NFS-mounted partition.

  • Supported OpenSSL versions

    The following list contains information about the supported OpenSSL versions in each syslog-ng PE application version.

    Linux glibc 2.11

    syslog-ng PE version supported Open SSL version
    7.0.1 OpenSSL 1.0.2j
    7.0.2 OpenSSL 1.0.2j
    7.0.3 OpenSSL 1.0.2j
    7.0.4 OpenSSL 1.0.2j
    7.0.5 OpenSSL 1.0.2j
    7.0.6 OpenSSL 1.0.2m
    7.0.7 OpenSSL 1.0.2m
    7.0.8 OpenSSL 1.0.2m
    7.0.9 OpenSSL 1.0.2o
    7.0.10 OpenSSL 1.0.2o
    7.0.11 OpenSSL 1.0.2p
    7.0.12 OpenSSL 1.0.2q
    7.0.13 OpenSSL 1.0.2q
    7.0.14 OpenSSL 1.0.2r
    7.0.15 OpenSSL 1.0.2s
    7.0.16 OpenSSL 1.0.2s
    7.0.17 OpenSSL 1.0.2t
    7.0.18 OpenSSL 1.0.2t
    7.0.19 OpenSSL 1.1.1d
    7.0.20 OpenSSL 1.1.1g
    7.0.21 OpenSSL 1.1.1g
    7.0.22 OpenSSL 1.1.1g
    7.0.23 OpenSSL 1.1.1h
    7.0.24 OpenSSL 1.1.1j
Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating