This chapter discusses the technical concepts of syslog-ng.
Preface
Introduction to syslog-ng
What syslog-ng is
What syslog-ng is not
Why is syslog-ng needed?
What is new in syslog-ng Premium Edition 7?
Who uses syslog-ng?
Supported platforms
The concepts of syslog-ng
The philosophy of syslog-ng
Logging with syslog-ng
Modes of operation
Global objects
Timezones and daylight saving
Versions and releases of syslog-ng Premium Edition
Licensing
GPL and LGPL licenses
High availability support
The structure of a log message
Message representation in syslog-ng PE
Structuring macros, metadata, and other value-pairs
Things to consider when forwarding messages between syslog-ng PE hosts
Using syslog-ng PE with NFS or CIFS (or SMB) file system for log files
Installing syslog-ng PE
Prerequisites to installing syslog-ng PE
Security-enhanced Linux: grsecurity, SELinux
Installing syslog-ng PE on RPM-based platforms (Red Hat, SUSE, AIX)
Using syslog-ng PE on SELinux
Installing syslog-ng PE on Debian-based platforms
Installing syslog-ng in Docker
Installing syslog-ng using the .run installer
The syslog-ng PE quick-start guide
Installing syslog-ng PE in client or relay mode
Installing syslog-ng PE in server mode
Installing syslog-ng PE without user-interaction
Upgrading syslog-ng PE
Upgrading from syslog-ng PE 7.0.x to version 7
Upgrading from syslog-ng PE 6.0.x to version 7
Upgrading syslog-ng PE to other package versions
Upgrading from syslog-ng PE to syslog-ng OSE
Upgrade from syslog-ng OSE to syslog-ng PE
Upgrading from complete syslog-ng PE to client setup version of syslog-ng PE
Upgrading the sql() source of syslog-ng PE
Uninstalling syslog-ng PE
Configuring Microsoft SQL Server to accept logs from syslog-ng
Configuring syslog-ng on client hosts
Configuring syslog-ng on server hosts
Configuring syslog-ng relays
Managing and checking syslog-ng PE service on Linux
The syslog-ng PE configuration file
Location of the syslog-ng configuration file
The configuration syntax in detail
Notes about the configuration syntax
Defining configuration objects inline
Using channels in configuration objects
Global and environmental variables
Logging configuration changes
Modules in syslog-ng Premium Edition (syslog-ng PE)
Managing complex syslog-ng configurations
Collecting log messages — sources and source drivers
Including configuration files
Reusing configuration blocks
Generating configuration blocks from a script
Python code in external files
Logging from your Python code
How sources work
default-network-drivers: Receive and parse common syslog messages
internal: Collecting internal messages
file: Collecting messages from text files
google-pubsub: collecting messages from the Google Pub/Sub messaging service
Sending and storing log messages — destinations and destination drivers
Prerequisites
Limitations
Supported platforms
Declaration
The Google Pub/Sub message format in syslog-ng PE
wildcard-file: Collecting messages from multiple text files
linux-audit: Collecting messages from Linux audit logs
mssql, oracle, sql: collecting messages from an SQL database
The contents of the Google Pub/Sub Message body on the syslog-ng Premium Edition (syslog-ng PE) side
The contents of the Google Pub/Sub Message attributes on the syslog-ng PE (syslog-ng PE) side
Processing incoming message contents in raw message format and in .JSON format
google-pubsub() source options
Preventing message duplication resulting from the At-Least-Once delivery behavior
Error messages you may encounter while using the google-pubsub() source
mssql(), oracle(), and sql() source options
Customizing mssql() queries
Configuring TLS encryption for MSSQL servers
Possible connection errors between the MSSQL server (2019) and syslog-ng PE 7 LTS
network: Collecting messages using the RFC3164 protocol (network() driver)
office365: Fetching logs from Office 365
Configuring Office 365 to permit fetching logs
office365() source options
Troubleshooting audit logging in Office 365
osquery: Collect and parse osquery result logs
pipe: Collecting messages from named pipes
program: Receiving messages from external applications
python: writing server-style Python sources
python-fetcher: writing fetcher-style Python sources
snmptrap: Read Net-SNMP traps
syslog: Collecting messages using the IETF syslog protocol (syslog() driver)
system: Collecting the system-specific log messages of a platform
systemd-journal: Collecting messages from the systemd-journal system log storage
systemd-syslog: Collecting systemd messages using a socket
tcp, tcp6,udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol
udp-balancer: Receiving UDP messages at very high rate
unix-stream, unix-dgram: Collecting messages from UNIX domain sockets
windowsevent: Collecting Windows event logs
elasticsearch2>: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED)
Routing messages: log paths, flags, and filters
Prerequisites
How syslog-ng PE interacts with Elasticsearch
Client modes
Elasticsearch2 destination options (DEPRECATED)
elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector
file: Storing messages in plain-text files
google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service
Limitations
Configuring the google_pubsub() destination
google_pubsub() destination options
Available endpoints for the google_pubsub() destination
Error messages you may encounter while using the google_pubsub() destination
google_pubsub-managedaccount(): Sending logs to the Google Cloud Pub/Sub messaging service authenticated by Google Cloud managed service account
Limitations
Configuring the google_pubsub_managedaccount() destination
google_pubsub_managedaccount() destination options
Available endpoints for the google_pubsub_managedaccount() destination
Error messages you can encounter while using the google_pubsub_managedaccount() destination
hdfs: Storing messages on the Hadoop Distributed File System (HDFS)
Prerequisites
How syslog-ng PE interacts with HDFS
Storing messages with MapR-FS
Kerberos authentication with syslog-ng hdfs() destination
HDFS destination options
http: Posting messages over HTTP
kafka(): Publishing messages to Apache Kafka (Java implementation) (DEPRECATED)
kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation)
kafka-c(): Prerequisites and limitations
kafka-c(): Shifting from the Java implementation to the C implementation
kafka-c(): Flow control in syslog-ng PE and the Kafka client
Options of the kafka-c() destination
logstore: Storing messages in encrypted files
mongodb: Storing messages in a MongoDB database
network: Sending messages to a remote log server using the RFC3164 protocol (network() driver)
pipe: Sending messages to named pipes
program: Sending messages to external applications
python: writing custom Python destinations
sentinel(): Sending logs to the Microsoft Azure Sentinel cloud
Configuring the sentinel() destination to send logs to the Microsoft Azure Sentinel cloud
snmp: Sending SNMP traps
smtp: Generating SMTP messages (email) from logs
splunk-hec: Sending messages to Splunk HTTP Event Collector
sql(): Storing messages in an SQL database
Getting the required credentials to configure syslog-ng PE as a Data Connector for Microsoft Azure Sentinel
Log types
sentinel() destination options
Using the sql() driver with an Oracle database
Using the sql() driver with a Microsoft SQL database
The way syslog-ng PE interacts with the database
sql() destination options
stackdriver: Sending logs to the Google Stackdriver cloud
syslog: Sending messages to a remote logserver using the IETF-syslog protocol
syslog-ng(): Forward logs to another syslog-ng node
tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)
unix-stream, unix-dgram: Sending messages to UNIX domain sockets
usertty: Sending messages to a user terminal — usertty() destination
Client-side failover
Log paths
Managing incoming and outgoing messages with flow-control
Using the disk-buffer option and memory buffering
Global options of syslog-ng PE
TLS-encrypted message transfer
Enabling the reliable disk-buffer option
Enabling the normal disk-buffer option
How to get information about disk-buffer files
Filters
Information about disk-buffer files
Getting the list of disk-buffer files
Getting the status information of disk-buffer files
Printing the content of disk-buffer files
Orphan disk-buffer files
How to process messages from an orphan disk-buffer file using a separate syslog-ng PE instance
How to empty disk-buffer files
Enabling memory buffering
About disk queue files
Using filters
Combining filters with boolean operators
Comparing macro values in filters
Using wildcards, special characters, and regular expressions in filters
Tagging messages
Filter functions
Dropping messages
Secure logging using TLS
Encrypting log messages with TLS
Mutual authentication using TLS
Password-protected keys
TLS options
Advanced Log Transport Protocol
Reliability and minimizing the loss of log messages
Introduction
Flow control, no disk-buffer option, no ALTP
Flow control, normal disk-buffer option, no ALTP
Flow control, reliable disk-buffer option, no ALTP
Flow control, reliable disk-buffer option, ALTP
Deciding which loss prevention mechanism to apply
Manipulating messages
Customizing message format using macros and templates
parser: Parse and segment structured messages
Formatting messages, filenames, directories, and tablenames
Templates and macros
Date-related macros
Hard versus soft macros
Macros of syslog-ng PE
Using template functions
Template functions of syslog-ng PE
Modifying the on-the-wire message format
Modifying messages using rewrite rules
Replacing message parts
Setting message fields to specific values
Unsetting message fields
Creating custom SDATA fields
Setting multiple message fields to specific values
Conditional rewrites
Anonymizing credit card numbers
Regular expressions
Parsing syslog messages
Parsing messages with comma-separated and similar values
Parsing key=value pairs
JSON parser
XML parser
Parsing dates and timestamps
Cisco Parser
Linux audit parser
Python parser
Parsing enterprise-wide message model (EWMM) messages
Sudo parser
iptables parser
Processing message content with a pattern database
Classifying log messages
Using pattern databases
Correlating log messages using pattern databases
Triggering actions for identified messages
Creating pattern databases
Correlating log messages
Enriching log messages with external data
Monitoring statistics and metrics of syslog-ng
Metrics and counters of syslog-ng PE
Log statistics from the internal() source
The monitoring() source
Multithreading and scaling in syslog-ng PE
Multithreading concepts of syslog-ng PE
Configuring multithreading
Optimizing multithreaded performance
Troubleshooting syslog-ng
Possible causes of losing log messages
Creating syslog-ng core files
Collecting debugging information with strace, truss, or tusc
Running a failure script
Stopping syslog-ng
Reporting bugs and finding help
Error messages
Best practices and examples
General recommendations
Handling large message load
Using name resolution in syslog-ng
Collecting logs from chroot
Configuring log rotation
Load balancing logs between multiple destinations
The syslog-ng manual pages
Glossary
The concepts of syslog-ng
The concepts of syslog-ng
The philosophy of syslog-ng
Typically, syslog-ng is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices — called syslog-ng clients — all run syslog-ng, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote syslog-ng server, which sorts and stores them.
NOTE: Due to complexity of deployment, configuration, and design, you may require assistance from One Identity Professional Services while introducing new or additional:
-
sources
-
destinations
-
log paths
-
significant increases in log volume.
One Identity Professional Services is equipped and trained to evaluate the needs of any organization, and to provide configuration and architectural recommendations that help our users get the most out of any syslog-ng PE version.
One Identity Professional Services offer assistance in planning and scoping for current needs, as well as recommendations for the future to ensure success.
Logging with syslog-ng
The syslog-ng application reads incoming messages and forwards them to the selected destinations. The syslog-ng application can receive messages from files, remote hosts, and other sources.
Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.
Sources and destinations are independent objects, log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations: messages arriving from a source are sent to every destination listed in the log path. A log path defined in syslog-ng is called a log statement.
Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path.
Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.
NOTE: Due to complexity of deployment, configuration, and design, you may require assistance from One Identity Professional Services while introducing new or additional:
-
sources
-
destinations
-
log paths
-
significant increases in log volume.
One Identity Professional Services is equipped and trained to evaluate the needs of any organization, and to provide configuration and architectural recommendations that help our users get the most out of any syslog-ng PE version.
One Identity Professional Services offer assistance in planning and scoping for current needs, as well as recommendations for the future to ensure success.
The route of a log message in syslog-ng
The following procedure illustrates the route of a log message from its source on the syslog-ng client to its final destination on the central syslog-ng server.
Figure 1: The route of a log message
-
A device or application sends a log message to a source on the syslog-ng client. For example, an Apache web server running on Linux enters a message into the /var/log/apache file.
-
The syslog-ng client running on the web server reads the message from its /var/log/apache source.
-
The syslog-ng client processes the first log statement that includes the /var/log/apache source.
-
The syslog-ng client performs optional operations (message filtering, parsing, and rewriting) on the message, for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement, for example, to the remote syslog-ng server.
Caution: Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.
NOTE: The syslog-ng client sends a message to all matching destinations by default. As a result, a message may be sent to a destination more than once, if the destination is used in multiple log statements. To prevent such situations, use the final flag in the destination statements. For details, see Log path flags.
-
The syslog-ng client processes the next log statement that includes the /var/log/apache source, repeating Steps 3-4.
-
The message sent by the syslog-ng client arrives from a source set in the syslog-ng server.
-
The syslog-ng server reads the message from its source and processes the first log statement that includes that source.
-
The syslog-ng server performs optional operations (message filtering, parsing, and rewriting) on the message, for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement.
Caution: Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.
-
The syslog-ng server processes the next log statement, repeating Steps 7-9.
NOTE: The syslog-ng application can stop reading messages from its sources if the destinations cannot process the sent messages. This feature is called flow-control and is detailed in Managing incoming and outgoing messages with flow-control.
