This chapter discusses the technical concepts of syslog-ng.
Preface
Introduction to syslog-ng
What syslog-ng is
What syslog-ng is not
Why is syslog-ng needed?
What is new in syslog-ng Open Source Edition version 3.33?
Who uses syslog-ng?
Supported platforms
The concepts of syslog-ng
The philosophy of syslog-ng
Logging with syslog-ng
Modes of operation
Global objects
Timezones and daylight saving
Product licensing
High availability support
The structure of a log message
Message representation in syslog-ng OSE
Structuring macros, metadata, and other value-pairs
Things to consider when forwarding messages between syslog-ng OSE hosts
Commercial version of syslog-ng
Installing syslog-ng
Compiling syslog-ng from source
Compiling options of syslog-ng OSE
Uninstalling syslog-ng OSE
Configuring Microsoft SQL Server to accept logs from syslog-ng
The syslog-ng OSE quick-start guide
Configuring syslog-ng on client hosts
Configuring syslog-ng on server hosts
Configuring syslog-ng relays
Managing and checking syslog-ng OSE service on Linux
The syslog-ng OSE configuration file
Location of the syslog-ng configuration file
The configuration syntax in detail
Notes about the configuration syntax
Defining configuration objects inline
Using channels in configuration objects
Global and environmental variables
Modules in syslog-ng Open Source Edition (syslog-ng OSE)
Managing complex syslog-ng configurations
source: Read, receive, and collect log messages
Including configuration files
Reusing configuration blocks
Generating configuration blocks from a script
Python code in external files
Logging from your Python code
How sources work
default-network-drivers: Receive and parse common syslog messages
internal: Collecting internal messages
file: Collecting messages from text files
wildcard-file: Collecting messages from multiple text files
linux-audit: Collecting messages from Linux audit logs
network: Collecting messages using the RFC3164 protocol (network() driver)
nodejs: Receiving JSON messages from nodejs applications
mbox: Converting local email messages to log messages
osquery: Collect and parse osquery result logs
pipe: Collecting messages from named pipes
pacct: Collecting process accounting logs on Linux
program: Receiving messages from external applications
python: writing server-style Python sources
python-fetcher: writing fetcher-style Python sources
snmptrap: Read Net-SNMP traps
sun-streams: Collecting messages on Sun Solaris
syslog: Collecting messages using the IETF syslog protocol (syslog() driver)
system: Collecting the system-specific log messages of a platform
systemd-journal: Collecting messages from the systemd-journal system log storage
systemd-syslog: Collecting systemd messages using a socket
tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE
unix-stream, unix-dgram: Collecting messages from UNIX domain sockets
stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP
collectd: sending metrics to collectd
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED)
log: Filter and route log messages using log paths, flags, and filters
Prerequisites
How syslog-ng OSE interacts with Elasticsearch
Client modes
Search Guard and syslog-ng OSE
Elasticsearch2 destination options (DEPRECATED)
Example use cases of sending logs to Elasticsearch using syslog-ng
elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API
file: Storing messages in plain-text files
graphite: Sending metrics to Graphite
Sending logs to Graylog
hdfs: Storing messages on the Hadoop Distributed File System (HDFS)
Prerequisites
How syslog-ng OSE interacts with HDFS
Storing messages with MapR-FS
Kerberos authentication with syslog-ng hdfs() destination
HDFS destination options
Posting messages over HTTP
http: Posting messages over HTTP without Java
Batch mode and load balancing
HTTP destination options
The Azure auth header plugin
The Python HTTP header plugin
kafka: Publishing messages to Apache Kafka (Java implementation)
kafka(): Publishing messages to Apache Kafka (C implementation, using the librdkafka client)
Shifting from Java implementation to C implementation
Before you begin
Flow control in syslog-ng PE and the Kafka client
Options of the kafka() destination's C implementation
loggly: Using Loggly
logmatic: Using Logmatic.io
mongodb: Storing messages in a MongoDB database
mqtt() destination: sending messages from a local network to an MQTT broker
Prerequisites to using the mqtt() destination
Limitations to using the mqtt() destination
Options of the mqtt() destination
Possible error messages you may encounter while using the mqtt() destination
network: Sending messages to a remote log server using the RFC3164 protocol (network() driver)
osquery: Sending log messages to osquery's syslog table
pipe: Sending messages to named pipes
program: Sending messages to external applications
pseudofile()
python: writing custom Python destinations
redis: Storing name-value pairs in Redis
riemann: Monitoring your data with Riemann
slack: Sending alerts and notifications to a Slack channel
smtp: Generating SMTP messages (email) from logs
snmp: Sending SNMP traps
Splunk: Sending log messages to Splunk
sql: Storing messages in an SQL database
Using the sql() driver with an Oracle database
Using the sql() driver with a Microsoft SQL database
The way syslog-ng interacts with the database
sql() destination options
stomp: Publishing messages using STOMP
Sumo Logic destinations: sumologic-http() and sumologic-syslog()
syslog: Sending messages to a remote logserver using the IETF-syslog protocol
syslog-ng(): Forward logs to another syslog-ng node
tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)
Telegram: Sending messages to Telegram
unix-stream, unix-dgram: Sending messages to UNIX domain sockets
usertty: Sending messages to a user terminal: usertty() destination
Write your own custom destination in Java or Python
Client-side failover
Log paths
Managing incoming and outgoing messages with flow-control
Using disk-based and memory buffering
Global options of syslog-ng OSE
TLS-encrypted message transfer
Enabling reliable disk-based buffering
Enabling normal disk-based buffering
Enabling memory buffering
About disk queue files
Filters
Using filters
Combining filters with boolean operators
Comparing macro values in filters
Using wildcards, special characters, and regular expressions in filters
Tagging messages
Filter functions
Dropping messages
Secure logging using TLS
Encrypting log messages with TLS
Mutual authentication using TLS
Password-protected keys
TLS options
template and rewrite: Format, modify, and manipulate log messages
Customize message format using macros and templates
parser: Parse and segment structured messages
Formatting messages, filenames, directories, and tablenames
Templates and macros
Date-related macros
Hard versus soft macros
Macros of syslog-ng OSE
Using template functions
Template functions of syslog-ng OSE
Modifying the on-the-wire message format
Modifying messages using rewrite rules
Replacing message parts
Setting message fields to specific values
Setting severity with the set-severity() rewrite function
Setting the facility field with the set-facility() rewrite function
Setting the priority of a message with the set-pri() rewrite function
Unsetting message fields
Creating custom SDATA fields
Setting multiple message fields to specific values
map-value-pairs: Rename value-pairs to normalize logs
Conditional rewrites
Adding and deleting tags
Rewrite the timezone of a message
Anonymizing credit card numbers
Regular expressions
Parsing syslog messages
Parsing messages with comma-separated and similar values
Parsing key=value pairs
JSON parser
XML parser
Parsing dates and timestamps
Python parser
Parsing tags
Apache access log parser
Linux audit parser
Cisco parser
Parsing enterprise-wide message model (EWMM) messages
iptables parser
Netskope parser
panos-parser(): parsing PAN-OS log messages
Sudo parser
Websense parser
Check Point Log Exporter parser
db-parser: Process message content with a pattern database (patterndb)
Classifying log messages
Using pattern databases
Correlating log messages
Enriching log messages with external data
Using parser results in filters and templates
Downloading sample pattern databases
Correlating log messages using pattern databases
Triggering actions for identified messages
Creating pattern databases
Adding metadata from an external file
Looking up GeoIP data from IP addresses (DEPRECATED)
Looking up GeoIP2 data from IP addresses
Statistics of syslog-ng
Multithreading and scaling in syslog-ng OSE
Multithreading concepts of syslog-ng OSE
Configuring multithreading
Optimizing multithreaded performance
Troubleshooting syslog-ng
Possible causes of losing log messages
Creating syslog-ng core files
Collecting debugging information with strace, truss, or tusc
Running a failure script
Stopping syslog-ng
Reporting bugs and finding help
Recover data from orphaned diskbuffer files
No local logs after specifying an unusual storage directory
No logs after specifying an unusual port number
Error messages
SELinux prevents syslog-ng OSE from using the execmem access on a process
Best practices and examples
General recommendations
Handling large message load
Using name resolution in syslog-ng
Collecting logs from chroot
Configuring log rotation
Load balancing logs between multiple destinations
The syslog-ng manual pages
Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License
Glossary
The concepts of syslog-ng
The concepts of syslog-ng
The philosophy of syslog-ng
Typically, syslog-ng is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices — called syslog-ng clients — all run syslog-ng, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote syslog-ng server, which sorts and stores them.
Logging with syslog-ng
The syslog-ng application reads incoming messages and forwards them to the selected destinations. The syslog-ng application can receive messages from files, remote hosts, and other sources.
Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.
Sources and destinations are independent objects, log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations: messages arriving from a source are sent to every destination listed in the log path. A log path defined in syslog-ng is called a log statement.
Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path.
Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.
The route of a log message in syslog-ng
Purpose:
The following procedure illustrates the route of a log message from its source on the syslog-ng client to its final destination on the central syslog-ng server.
Figure 1: The route of a log message
Steps:
-
A device or application sends a log message to a source on the syslog-ng client. For example, an Apache web server running on Linux enters a message into the /var/log/apache file.
-
The syslog-ng client running on the web server reads the message from its /var/log/apache source.
-
The syslog-ng client processes the first log statement that includes the /var/log/apache source.
-
The syslog-ng client performs optional operations (message filtering, parsing, and rewriting) on the message, for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement, for example, to the remote syslog-ng server.
Caution: Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.
NOTE: The syslog-ng client sends a message to all matching destinations by default. As a result, a message may be sent to a destination more than once, if the destination is used in multiple log statements. To prevent such situations, use the final flag in the destination statements. For details, see Log statement flags.
-
The syslog-ng client processes the next log statement that includes the /var/log/apache source, repeating Steps 3-4.
-
The message sent by the syslog-ng client arrives from a source set in the syslog-ng server.
-
The syslog-ng server reads the message from its source and processes the first log statement that includes that source.
-
The syslog-ng server performs optional operations (message filtering, parsing, and rewriting) on the message, for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement.
Caution: Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.
-
The syslog-ng server processes the next log statement, repeating Steps 7-9.
NOTE: The syslog-ng application can stop reading messages from its sources if the destinations cannot process the sent messages. This feature is called flow-control and is detailed in Managing incoming and outgoing messages with flow-control.
