The syslog-ng Store Box (SSB) is a log server appliance that collects, stores and monitors log messages sent by network devices, applications and computers. SSB can receive traditional syslog messages, syslog messages that comply with the new Internet Engineering Task Force (IETF) standard (RFC 5424-5428), eventlog messages from Microsoft Windows hosts, as well as SNMP messages.
Figure 1: The philosophy of the syslog-ng Store Box
Clients can send messages to SSB using their own logging application if it supports the BSD-syslog (RFC 3164) or the IETF-syslog (RFC 5424-5428) protocol, or they can use the syslog-ng Premium Edition application to act as the log-forwarding agent of SSB.
The main purpose of SSB is to collect the logs from the clients and store them on its hard disk. The messages are stored in so-called logspaces. There are two types of logspaces: the first stores messages in traditional plain-text files, while the second one uses a binary format that can be compressed, encrypted, and timestamped.
You can also define multiple logspaces, remote logspaces, and configure filtered subsets of each logspace. A multiple logspace aggregates messages from multiple SSBs (located at different sites), allowing you to view and search the logs of several SSBs from a single web interface without having to log on to several different interfaces. Remote logspaces, on the other hand, enable you to access and search logspaces (including filtered logspaces) on other SSB appliances. Filtered logspaces allow the creation of a smaller, filtered subset of the logs contained in an existing local, remote or multiple logspace.
The syslog-ng application reads incoming messages and forwards them to the selected destinations. The syslog-ng application can receive messages from files, remote hosts, and other sources.
Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations. In the case of the clients, one of the destinations is the syslog-ng Store Box. The destinations on the SSB can be logspaces or remote servers, such as database servers or log analyzing engines.
Sources and destinations are independent objects, log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations: messages arriving to a source are sent to every destination listed in the log path. A log path defined in syslog-ng is called a log statement.
Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path.
SSB is configured by an administrator or auditor using a web browser.
The following procedure illustrates the route of a log message from its source on the syslog-ng client to the syslog-ng Store Box.
Figure 2: The route of a log message
-
A device or application sends a log message to a source on the syslog-ng client. For example, an Apache web server running on Linux enters a message into the /var/log/apache file.
-
The syslog-ng client running on the web server reads the message from its /var/log/apache source.
-
The syslog-ng client processes the first log statement that includes the /var/log/apache source.
-
The syslog-ng client performs optional operations on the message, for example, it rewrites parts of the message or compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement, for example, to the remote syslog-ng server.
After that, the syslog-ng client processes the next log statement that includes the /var/log/apache source, repeating Steps 3-4.
-
The message sent by the syslog-ng client arrives to a source set on the syslog-ng Store Box.
-
The syslog-ng Store Box reads the message from its source and processes the first log path that includes that source.
-
The syslog-ng Store Box processes the message and performs the following operations. Note that most of these operations are optional, but the order of the processing steps is fixed.
-
Parse the message as a syslog message (unless message parsing is explicitly disabled for the source).
-
Classify the message using a pattern database.
-
Modify the message using rewrite rules (before filtering).
-
Filter the messages, for example, based on sender hostname or message content. If the message does not match the configured filter, SSB will not send it to the destination.
-
Parse the text of the message (that is, the ${MESSAGE} part) using a key-value parser or the sudo parser.
-
Modify the message using rewrite rules (after filtering and other parsing).
-
SSB sends the message to the destinations set in the logpath. The destinations are local, optionally encrypted files on SSB, or remote servers, such as a database server.
-
SSB processes the next log statement, repeating Steps 6-8.
This section describes the internal message-processing model of syslog-ng, as well as the flow-control feature that can prevent message loss. To use flow-control, the flow-control option must be enabled for the particular log path.
The internal message-processing model of syslog-ng
-
The syslog-ng application checks the source for messages.
-
When a log message is found, syslog-ng reads the message.
-
The message is processed and put into the output buffer of the destination.
-
When the destination can accept the message, syslog-ng sends the message to the destination from the output buffer.
Flow-control
If the destination cannot send out messages, or not as fast as they arrive in the destination, the output buffer fills up. When the output buffer is full, the sources stop reading messages. This can prevent message loss.
If a message is successfully sent out from the destination, the source that sent that message starts reading logs again, until the destination buffer fills up.
Flow-control and multiple destinations
Using flow-control on a source has an important side-effect if the messages of the source are sent to multiple destinations. If flow-control is in use and one of the destinations cannot accept the messages, the other destinations do not receive any messages either, because syslog-ng stops reading the source. For example, if messages from a source are sent to a remote server and also stored locally in a file, and the network connection to the server becomes unavailable, neither the remote server nor the local file will receive any messages. This side-effect of the flow-control can be avoided by using the disk-based buffering feature of syslog-ng.
|
|
NOTE:
Creating separate log paths for the destinations that use the same flow-controlled source does not help avoiding the problem. |
The syslog-ng Store Box receives log messages securely over the network using the Transport Layer Security (TLS) protocol (TLS is an encryption protocol over the TCP/IP network protocol).
TLS uses certificates to authenticate and encrypt communication, as illustrated in the following figure:
Figure 3: Certificate-based authentication
The client sending the logs authenticates SSB by requesting its certificate and public key. Optionally, SSB can also request a certificate from the client, thus mutual authentication is also possible.
In order to use TLS encryption in syslog-ng, the following elements are required:
When using mutual authentication to verify the identity of the clients, the following elements are required:
-
A certificate must be available on the syslog-ng client. This certificate identifies the syslog-ng client.
-
The certificate of the Certificate Authority that issued the certificate of the syslog-ng client must be available on SSB.
Mutual authentication ensures that SSB accepts log messages only from authorized clients.
For details on configuring TLS communication in syslog-ng, see Configuring message sources.
