Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.34 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng PE The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files google-pubsub: collecting messages from the Google Pub/Sub messaging service wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mssql, oracle, sql: collecting messages from an SQL database network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6,udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2>: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files google_bigquery(): Sending logs to a Google BigQuery table google_bigquery_managedaccount(): Sending logs to a Google BigQuery table authenticated by Google Cloud managed service account google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service google_pubsub-managedaccount(): Sending logs to the Google Cloud Pub/Sub messaging service authenticated by Google Cloud managed service account hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka(): Publishing messages to Apache Kafka (Java implementation) (DEPRECATED) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel(): Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql(): Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transport Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

Things to consider when forwarding messages between syslog-ng PE hosts

When you send your log messages from a syslog-ng PE client through the network to a syslog-ng PE server, you can use different protocols and options. Every combination has its advantages and disadvantages. The most important thing is to use matching protocols and options, so the server handles the incoming log messages properly.

In syslog-ng PE you can change many aspects of the network communication. First of all, there is the structure of the messages itself. Currently, syslog-ng PE supports two standard syslog protocols: the BSD (RFC3164) and the syslog (RFC5424) message format.

These RFCs describe the format and the structure of the log message, and add a (lightweight) framing around the messages. You can set this framing/structure by selecting the appropriate driver in syslog-ng PE. There are two drivers you can use: the network() driver and the syslog() driver. The syslog() driver is for the syslog (RFC5424) protocol and the network() driver is for the BSD (RFC3164) protocol.

The tcp() and udp() drivers are now deprecated, they are essentially equivalent with the network(transport(tcp)) and network(transport(udp)) drivers.

In addition to selecting the driver to use, both drivers allow you to use different transport-layer protocols: TCP and UDP, and optionally also higher-level transport protocols: TLS (over TCP, and ALTP (optionally using TLS). To complicate things a bit more, you can configure the network() driver (corresponding to the BSD (RFC3164) protocol) to send the messages in the syslog (RFC5424) format (but without the framing used in RFC5424) using the flag(syslog-protocol) option.

Because some combination of drivers and options are invalid, you can use the following drivers and options as sources and as destinations:

  1. syslog(transport(tcp))

  2. syslog(transport(udp))

  3. syslog(transport(altp))

  4. syslog(transport(tls))

  5. syslog(transport(altp(tls-required(yes)))

  6. network(transport(tcp))

  7. network(transport(udp))

  8. network(transport(altp))

  9. network(transport(tls))

  10. network(transport(altp(tls-required(yes)))

  11. network(transport(tcp) flag(syslog-protocol))

  12. network(transport(udp) flag(syslog-protocol))

  13. network(transport(altp)flag(syslog-protocol))

  14. network(transport(tls) flag(syslog-protocol))

  15. network(transport(altp(tls-required(yes)) flag(syslog-protocol))

If you use the same driver and options in the destination of your syslog-ng PE client and the source of your syslog-ng PE server, everything should work as expected. Unfortunately there are some other combinations, that seem to work, but result in losing parts of the messages. The following table show the combinations:

Table 6: Source-destination driver combinations
Source \ Destination syslog/tcp syslog/udp syslog/tls network/tcp network/udp network/tls network/tcp/flag network/udp/flag network/tls/flag
syslog/tcp ✔ - - ! - - ! - -
syslog/udp - ✔ - - ! - - ! -
syslog/tls - - ✔ - - ! - - !
network/tcp - - - ✔ - - ✔? - -
network/udp - ✔? - - ✔ - - ✔? -
network/tls - - - - - ✔ - - ✔?
network/tcp/flag ! - - ! - - ✔ - -
network/udp/flag - ! - - ! - - ✔ -
network/tls/flag - - ! - - ! - - ✔

  • - This method does not work. The logs will not get to the server.

  • ✔ This method works.

  • ! This method has some visible drawbacks. The logs go through, but some of the values are missing/misplaced/and so on.

  • ✔? This method seems to work, but it is not recommended because this can change in a future release.

NOTE: To receive UDP messages at very high message rate, you can use the udp-balancer() source. For details, see udp-balancer: Receiving UDP messages at very high rate.

Using syslog-ng PE with NFS or CIFS (or SMB) file system for log files

This section and its subsections describe using syslog-ng Premium Edition (syslog-ng PE) with NFS or CIFS file system for log files.

NOTE: All information referring to the CIFS network file system in this section automatically applies to the SMB network file system as well. As a result, even if the content of this section only refers to the CIFS file system, keep in mind that the same information also applies to SMB file systems as well.

If your NFS or CIFS connection is not stable, using the NFS or CIFS network file system can lead to problems. As a result, One Identity does neither recommend, nor officially support such scenarios. If you can avoid it, do not store log files on NFS or CIFS. If the NFS or CIFS connection is stable and reliable, syslog-ng Premium Edition (syslog-ng PE) can read and write files on mounted NFS or CIFS partitions as a normal file source or destination. Read this section carefully before using syslog-ng PE and NFS-mounted or CIFS-mounted log files.

Limitations of using syslog-ng PE with NFS or CIFS (or SMB) file system

This section lists the limitations of using syslog-ng PE with NFS or CIFS file system.

NOTE: All information referring to the CIFS network file system in this section automatically applies to the SMB network file system as well. As a result, even if the content of this section only refers to the CIFS file system, keep in mind that the same information also applies to SMB file systems as well.

Using syslog-ng PE with NFS or CIFS file system has the following limitations:

  		•  Caution:

    Hazard of data loss!

    The syslog-ng PE application does not support storing the disk-buffer files on any kind of network share (that is, NFS, CIFS, and so on). To avoid crashes, data corruption, or data loss, One Identity does not recommend storing your disk-buffer files on network share.

  • Do not use the logstore() destination to store files on an NFS-mounted or CIFS-mounted partition.

  • To use wildcards in the file source, set the force-directory_polling() option to yes to detect newly created files. Note that this option is available only in syslog-ng PE version 6.0.3 and newer versions of the 6.x branch, and is not yet available in syslog-ng PE version 7.

  • Since One Identity does not officially support scenarios where you use syslog-ng PE together with NFS or CIFS, One Identity will handle support requests and bugs related to such scenarios only if you can reproduce the issue independently from NFS or CIFS.

Risks of using syslog-ng PE with NFS or CIFS (or SMB) file system

This section describes the risks of using syslog-ng PE with NFS or CIFS file system.

NOTE: All information referring to the CIFS network file system in this section automatically applies to the SMB network file system as well. As a result, even if the content of this section only refers to the CIFS file system, keep in mind that the same information also applies to SMB file systems as well.

Caution:

Hazard of data loss!

The syslog-ng PE application does not support storing the disk-buffer files on any kind of network share (that is, NFS, CIFS, and so on). To avoid crashes, data corruption, or data loss, One Identity does not recommend storing your disk-buffer files on network share.

If there is any issue with the NFS or CIFS connection (for example, connection loss, the NFS or CIFS server stops), syslog-ng PE can stop working. These NFS-related or CIFS-related issues can be related to the operating system, and can also vary depending on the operating system's patch level and kernel version. The possible effects include the following:

  • The syslog-ng PE application freezes, does not respond, does not process logs, is unable to stop or reload, and you can stop it only using the kill -9 command.

  • The syslog-ng PE application cannot start, and hangs during startup.

  • Message loss or message duplication.

  • Message becomes corrupt (it is not lost, but the message or some parts of it contain garbage).

  • When using the logstore() destination, the logstore file becomes corrupt.

  • On some RHEL-based systems (possibly depending on the kernel version too), NFS returns NULL characters when reading a file that another process is writing at the very same moment.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating