Chat now with support
Chat with Support

syslog-ng Open Source Edition 3.38 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files kubernetes: Collecting and parsing the Kubernetes CRI (Container Runtime Interface) format linux-audit: Collecting messages from Linux audit logs mqtt: receiving messages from an MQTT broker network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local email messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP collectd: sending metrics to collectd discord: Sending alerts and notifications to Discord elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka (Java implementation) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) loggly: Using Loggly logmatic: Using Logmatic.io mongodb(): Storing messages in a MongoDB database mqtt() destination: sending messages from a local network to an MQTT broker network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (email) from logs snmp: Sending SNMP traps Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP Sumo Logic destinations: sumologic-http() and sumologic-syslog() syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages
Parsing syslog messages Parsing messages with comma-separated and similar values Parsing key=value pairs JSON parser XML parser Parsing dates and timestamps Python parser Parsing tags Apache access log parser Linux audit parser Cisco parser Parsing enterprise-wide message model (EWMM) messages iptables parser Netskope parser panos-parser(): parsing PAN-OS log messages Sudo parser MariaDB parser Websense parser Fortigate parser Check Point Log Exporter parser Regular expression (regexp) parser db-parser: Process message content with a pattern database (patterndb)
Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License The syslog-ng Open Source Edition Documentation License Glossary

Printing the content of disk-buffer files

This section describes printing the content of the disk-buffer files used in syslog-ng Open Source Edition(syslog-ng OSE).

Command syntax

The command syntax for printing the content of the disk-buffer files used in syslog-ng OSE looks like the following:

dqtool cat DISK-BUFFER_FILE
Short example output for printed content
Example: short output that shows the printed content of the disk-buffer files used in syslog-ng OSE

The following short output example shows the printed content of the disk-buffer files used in syslog-ng OSE:

dqtool cat /opt/syslog-ng/var/syslog-ng-00000.rqf

Reliable disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.rqf', queue_length='2952', size='-437712'
Jul 31 12:33:48.226 10.21.10.10 <382019-07-31T12:33:36 localhost prg00000[1234]: seq: 0000000838, thread: 0000, runid: 1564569216, stamp: 2019-07-31T12:33:36 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
...

Orphan disk-buffer files

This section describes orphan disk-buffer files used in syslog-ng Open Source Edition(syslog-ng OSE).

In certain situations (for example, after modifying the disk-buffer configuration or losing the persist information), syslog-ng OSE creates a new disk-buffer file instead of using the already existing one. In these situations, the already existing disk-buffer file becomes a so-called orphan disk-buffer file.

NOTE: The syslog-ng OSE application does not store messages in orphan disk-buffer files or forward the messages stored in the disk-buffer file.

Discovering the new disk-buffer files (orphan disk-buffer files)

To discover orphan disk-buffer files, get the list of disk-buffer files from the persist file, then compare the list with the contents of the disk-buffer files' saving directory.

For more information about how you can get the list of disk-buffer files from the persist file, see Getting the list of disk-buffer files).

Example: difference between the list of disk-buffer files from the persist file and the content of the disk-buffer files' saving directory

The following examples show the difference between the list of disk-buffer files from the persist file and the content of the disk-buffer files' saving directory.

Disk-buffer file list from persist file: 

afsocket_dd_qfile(stream,10.21.10.112:514) = { "queue_file": "/opt/syslog-ng/var/syslog-ng-00001.rqf" }

Disk-buffer files' saving directory content: 

# ls -l /var/lib/syslog-ng/*qf
-rw------- 1 root root 2986780 Jul 31 12:30 /var/lib/syslog-ng/syslog-ng-00000.qf
-rw------- 1 root root 2000080 Jul 31 12:31 /opt/syslog-ng/var/syslog-ng-00000.rqf
-rw------- 1 root root    4096 Aug  1 11:09 /opt/syslog-ng/var/syslog-ng-00001.rqf

The disk-buffer files syslog-ng-00000.qf and syslog-ng-00000.rqf don't exist in the persist file. These two files are the orphan disk-buffer files.

For more information about orphan disk-buffer files and how to process the messages in orphan disk-buffer files using a separate syslog-ng OSE instance, see How to process messages from an orphan disk-buffer file using a separate syslog-ng OSE instance.

How to process messages from an orphan disk-buffer file using a separate syslog-ng OSE instance

This section describes how to read messages from an orphan disk-buffer file by using a separate syslog-ng Open Source Edition (syslog-ng OSE) process running parallel to the already running syslog-ng OSE instance.

Orphan disk-buffer files

In certain situations (for example, after modifying the disk-buffer configuration or losing the persist information), syslog-ng OSE creates a new disk-buffer file instead of using the already existing one. In these situations, the already existing disk-buffer file becomes a so-called orphan disk-buffer file.

NOTE: The syslog-ng OSE application does not store messages in orphan disk-buffer files or forward the messages stored in the disk-buffer file.

Processing the messages from an orphan disk-buffer file by using a separate syslog-ng OSE instance

When syslog-ng OSE creates orphan disk-buffer files, you can start a separate syslog-ng OSE instance parallel to the syslog-ng OSE instance already running, and use the following resolution process to process the messages in the orphan disk-buffer file.

Caution:

Before starting a separate syslog-ng OSE instance to process the messages from the orphan disk-buffer file, consider the following:

  • During the resolution process, a separate syslog-ng OSE instance will be started with its temporary files beside the syslog-ng OSE instance already running.
  • An incorrect startup command and incorrect configurations may cause issues for the syslog-ng OSE instance already running.
  • The disk-buffer file stores processed log messages in the format in which they would have been sent out to the destination.
  • The disk-buffer file doesn't store information about the destination.

To process the messages from an orphan disk-buffer file using a separate syslog-ng OSE instance,

  1. Identify the orphan disk-buffer files and make a record of them. For more information, see How to get information about disk-buffer files.

    It is important to know the type of the disk-buffer file. Disk-buffer file types can be normal (.qf) or reliable (.rqf).

    In the examples during this process, the /opt/syslog-ng/var/syslog-ng-00005.rqf orphan reliable disk-buffer file is used.

  2. Determine the destination of the logs. The content of the disk-buffer may help you determine the logs' destination. For more information, see How to get information about disk-buffer files.

    In the examples during this process, the destination 10.21.10.20 is used with the standard network() port 514.

  3. Create a directory for the temporary instance. In the examples during this process, the /tmp/qdisk directory is used.

    mkdir /tmp/qdisk

    		 Caution:

    Make sure that there is sufficient disk space in the directory. The minimum recommended disk space in the directory is equal to the size of the orphan disk-buffer file.

    If you want to use a different temporary directory (that is, other than /tmp/qdisk), create a symbolic link between /tmp/qdisk and the temporary directory you want to use with ln -s /path/to/tempdir /tmp/qdisk. This will allow you to use the commands in this resolution process.

    If you will not use a different temporary directory, use the /tmp/qdisk temporary directory in the example commands and file names.

  4. Create the configuration file /tmp/qdisk/qdisk.conf for the temporary instance with the following content.

    Example: creating the /tmp/qdisk/qdisk.conf configuration file for the temporary instance
    @version:7.0
@include "scl.conf"

options {
  keep-hostname(yes);
  keep-timestamp(yes);
};

destination d_destination {
#    ADD YOUR DESTINATION HERE

};

log {
  destination(d_destination);
};

  5. Add your destination statement with disk-buffer() to the configuration file. You can copy the destination statement from your running syslog-ng OSE configuration.

    		 Caution:

    Add the dir() option and set the disk-buffer file's destination directory to the temporary directory (that is, /tmp/qdisk) in your destination statement.

    Example: adding the destination statement with disk-buffer() to the configuration file
    network("10.21.10.20"
    disk-buffer(
        disk-buf-size(1048576)
        reliable(yes)
        dir(/tmp/qdisk/)
);

  6. Start the temporary syslog-ng OSE instance in the foreground.

    syslog-ng -Fe -f /tmp/qdisk/qdisk.conf -R /tmp/qdisk/qdisk.persist -c /tmp/qdisk/qdisk.ctl

    The syslog-ng OSE application will log to the console, so you will see any potential error that may occur during startup.

    The following example output displays that an empty disk-buffer file has been created and the connection to the remote destination has been established.

    Example: output displaying newly created empty disk-buffer file and connection established to remote destination
    Follow-mode file source not found, deferring open; filename='/no_such_file_or.dir'
Reliable disk-buffer state saved; filename='/tmp/qdisk/syslog-ng-00000.rqf', qdisk_length='0'
No server license found, running in client mode;
syslog-ng starting up; version='7.0.20', cfg-fingerprint='eaa03b9efb88b87d7c1b0ce7efd042ed8ac0c013', cfg-nonce-ndx='0', cfg-signature='c0327a7f7e6418ce0399a75089377dfb662bb072'
FIPS information; FIPS-mode='disabled'
Syslog connection established; fd='7', server='AF_INET(10.21.10.20:514)', local='AF_INET(0.0.0.0:0)'

  7. To stop syslog-ng OSE, press CTRL+C.

  8. Overwrite the empty disk-buffer file with the orphan disk-buffer file.

    mv /opt/syslog-ng/var/syslog-ng-00005.rqf /tmp/qdisk/syslog-ng-00000.rqf

  9. Start syslog-ng OSE using the command used in Start the temporary syslog-ng OSE instance in the foreground step.

    syslog-ng -Fe -f /tmp/qdisk/qdisk.conf -R /tmp/qdisk/qdisk.persist -c /tmp/qdisk/qdisk.ctl

  10. Open another terminal and check the progress by using one of the following methods.

    • Checking the number of stored logs in the disk-buffer (that is, the last number from the output).

      /opt/syslog-ng/sbin/syslog-ng-ctl stats -c /tmp/qdisk/qdisk.ctl | grep 'dst.*queued'

    • Checking the status of the disk-buffer file.

      dqtool info /tmp/qdisk/syslog-ng-00000.rqf

      An empty disk-buffer file will look similar to this:

      Example: empty disk-buffer file status message

      When checking the status of the disk-buffer files, the terminal will display a similar status message for an empty disk-buffer file:

      Reliable disk-buffer state loaded; filename='/tmp/qdisk/syslog-ng-00000.rqf', queue_length='0', size='0'

  11. Press CTRL+C to stop syslog-ng OSE.

  12. Check the state of the orphan disk-buffer file. For more information, see How to get information about disk-buffer files.

  13. If you have more than one orphan disk-buffer file, repeat the steps following the syslog-ng OSE stop (that is, the steps beginning from overwriting the empty disk-buffer file with the orphan disk-buffer file) for each orphan disk-buffer file.

  14. Remove the temporary directory.

    Example: command for removing the temporary directory

    The following command removes the /mp/qdisk temporary directory:

    rm -rf /tmp/qdisk

Enabling memory buffering

To enable memory buffering, use the log-fifo-size() parameter in the destination. All destination drivers can use memory buffering. Use memory buffering if you want to send logs to destinations where disk-based buffering is not available. Or if you want the fastest solution, and if syslog-ng OSE crash or network downtime is never expected. In these cases, losing logs is possible. This solution does not use disk-based buffering, logs are stored only in the memory.

Example: Example for using memory buffering
destination d_BSD {
    network("127.0.0.1"
        port(3333)
        log-fifo-size(10000)
    );
};
Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating