Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.20 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel: Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql: Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

Enabling the reliable disk-buffer option

The following destination drivers can use the disk-buffer option: elasticsearch2(), file(), hdfs(), kafka(), mongodb(), program(), riemann(), smtp(),sql(), unix-dgram(), and unix-stream(). The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method. (The other destinations or protocols do not provide the necessary feedback mechanisms required for the disk-buffer option.)

To enable using the reliable disk-buffer option, use the disk-buffer(reliable(yes)) parameter in the destination. Use the reliable disk-buffer option if you do not want to lose logs in case of reload/restart, an unreachable destination or syslog-ng PE crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. The filename of the reliable disk-buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.rqf.

Example: Example for using the reliable disk-buffer option
destination d_BSD {
    network(
            "127.0.0.1"
            port(3333)
            disk-buffer(
                mem-buf-size(10000)
                disk-buf-size(2000000)
                reliable(yes)
            )
        );
}; 

For more details on the differences between the normal and the reliable disk-buffer options, see About disk queue files.

Enabling the normal disk-buffer option

The following destination drivers can use the disk-buffer option: elasticsearch2(), file(), hdfs(), kafka(), mongodb(), program(), riemann(), smtp(),sql(), unix-dgram(), and unix-stream(). The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method. (The other destinations or protocols do not provide the necessary feedback mechanisms required for the disk-buffer option.)

To enable the normal disk-buffer option, use the disk-buffer(reliable(no)) parameter in the destination. Use the normal disk-buffer option if you want a solution that is faster than the reliable disk-buffer option. In this case, the process will be less reliable and it is possible to lose logs in case of syslog-ng PE crash. The filename of the normal disk-buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.qf.

Example: Example for using the normal disk-buffer option

When using the plugin for the disk-buffer file

destination d_BSD {
    network(
            "127.0.0.1"
            port(3333)
            disk-buffer(
                mem-buf-length(10000)
                disk-buf-size(2000000)
                reliable(no)
            )
        );
        }; 

For more details on the differences between the normal and the reliable disk-buffer options, see About disk queue files.

How to get information about disk-buffer files

This section describes how to get information about disk-buffer files used in syslog-ng Premium Edition (syslog-ng PE).

NOTE: Consider the following while reading this section:

  • This section uses the default installation path /opt/syslog-ng in the commands and syslog-ng PE files.
  • The syslog-ng PE persist file format is different in syslog-ng PE 6 and syslog-ng PE 7, so the commands may differ for the two versions.
Topics:

Useful information about disk-buffers

This section describes useful information about disk-buffers used in syslog-ng Premium Edition(syslog-ng PE).

The following list contains useful information about disk-buffers:

  • You can configure disk-buffer() for a remote destination in the destination() statement.

    For more information, see [%=System.LinkedTitle%] > [%=System.LinkedTitle%] > [%=System.LinkedTitle%].

  • By default, syslog-ng PE creates disk-buffer files under /opt/syslog-ng/var directory, unless dir() option is set in disk-buffer().
  • The filenames are generated automatically by syslog-ng PE with the extensions .qf for normal disk-buffer and .rqf for reliable disk-buffer.
  • The disk-buffer file stores processed log messages in the format in which they would have been sent out to the destination, but doesn't store information about the destination.
Related Documents