syslog-ng Open Source Edition 3.21 - Administration Guide

Summary of contents Target audience and prerequisites Products covered in this guide Summary of changes Acknowledgments

What syslog-ng is What syslog-ng is not Why is syslog-ng needed? What is new in syslog-ng Open Source Edition 3.21? Who uses syslog-ng? Supported platforms

The philosophy of syslog-ng Logging with syslog-ng Modes of operation Global objects Timezones and daylight saving Product licensing High availability support The structure of a log message Message representation in syslog-ng OSE Structuring macros, metadata, and other value-pairs Things to consider when forwarding messages between syslog-ng OSE hosts Commercial version of syslog-ng

Compiling syslog-ng from source Compiling options of syslog-ng OSE Uninstalling syslog-ng OSE Configuring Microsoft SQL Server to accept logs from syslog-ng

Configuring syslog-ng on client hosts Configuring syslog-ng on server hosts Configuring syslog-ng relays

Location of the syslog-ng configuration file The configuration syntax in detail Notes about the configuration syntax Defining configuration objects inline Using channels in configuration objects Global and environmental variables Modules in syslog-ng OSE Managing complex syslog-ng configurations Python code in external files

How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local e-mail messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream

amqp: Publishing messages using AMQP collectd: sending metrics to collectd elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka (Java implementation) kafka: Publishing messages to Apache Kafka (C implementation) loggly: Using Loggly logmatic: Using Logmatic.io mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (e-mail) from logs Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng: Forwarding messages and tags to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover

Log paths Managing incoming and outgoing messages with flow-control Using disk-based and memory buffering Filters Dropping messages

Configuring global syslog-ng options Global options

Secure logging using TLS Encrypting log messages with TLS Mutual authentication using TLS Password-protected keys TLS options

Customize message format using macros and templates Modifying messages using rewrite rules Regular expressions

Parsing syslog messages Parsing messages with comma-separated and similar values Parsing key=value pairs JSON parser XML parser Parsing dates and timestamps Python parser Apache access log parser Linux audit parser Cisco parser Parsing enterprise-wide message model (EWMM) messages iptables parser Netskope parser Sudo parser Websense parser Check Point Log Exporter parser

Classifying log messages Using pattern databases Triggering actions for identified messages Creating pattern databases

Correlating messages using the grouping-by() parser

Adding metadata from an external file Looking up GeoIP data from IP addresses (DEPRECATED) Looking up GeoIP2 data from IP addresses

Metrics and counters of syslog-ng OSE Log statistics from the internal() source

Multithreading concepts of syslog-ng OSE Configuring multithreading Optimizing multithreaded performance

Possible causes of losing log messages Creating syslog-ng core files Collecting debugging information with strace, truss, or tusc Running a failure script Stopping syslog-ng Reporting bugs and finding help Recover data from orphaned diskbuffer files No local logs after specifying an unusual storage directory No logs after specifying an unusual port number Error messages

General recommendations Handling large message load Using name resolution in syslog-ng Collecting logs from chroot Configuring log rotation

Manual pages