The syslog-ng Store Box (SSB) appliance stores log messages in binary or plain text log files called logspaces. You can define multiple logspaces, remote logspaces, and configure filtered subsets of each logspace.
Binary log files (logstores) correspond to the encrypted logstore() destination of syslog-ng. Logstores can be compressed, encrypted, and time stamped by an external Time Stamping Authority (TSA). To make the contents of the logstore searchable, you can create a separate indexer configuration for each logstore.
A multiple logspace aggregates messages from multiple SSBs (located at different sites), allowing you to view and search the logs of several SSBs from a single web interface without having to log on to several different interfaces.
Remote logspaces enable you to access and search logspaces (including filtered logspaces) on other SSB appliances.
Filtered logspaces allow you to create a smaller, filtered subset of the logs contained in an existing local, remote or multiple logspace. Assigning a user group to a filtered logspace enables fine-grained access control by creating a group that sees only a subset of the logs from a logspace.
Summary of multiple, remote, and filtered logspace types provides a summary and comparison of these three logspace types.
Table 7: Summary of multiple, remote, and filtered logspace types
| Multiple |
Multiple SSBs located at different sites |
Aggregate messages from multiple logspaces into a single logspace
Pre-filter log messages and share with only select user groups |
✔ |
✔ |
| Remote |
Remote SSB |
Access a logspace on another SSB |
✔ |
✔ |
| Filtered |
Local / multiple / remote SSB(s) |
Control access to a logspace at a granular level by granting access only to a subset of a logspace |
✔ |
N/A |
By default, SSB has the following logspaces:
Figure 138: Log > Logspaces — Default logspaces in SSB
-
local: An unencrypted, binary logspace for storing the log messages of SSB.
-
center: An unencrypted, binary logspace for storing the log messages sent by the clients.
Logspaces are stored locally on the hard disk of SSB. To access a logspace remotely, you can configure another SSB to view and search the logspace as a remote logspace, or you can make the logspace accessible as a network drive.
Logstores are logspaces with binary log files for storing log messages sent by the clients. Logstores can be compressed, encrypted, and time stamped by an external Time Stamping Authority (TSA). To make the contents of the logstore searchable, you can create a separate indexer configuration for each logstore.
The following limitations apply to logstores:
-
Indexing logstore files is currently limited: the indexer can handle only one file from a logstore for every day (syslog-ng Store Box (SSB) automatically starts a new log file for every day).
-
Logstore files consist of chunks. In rare cases, if the syslog-ng application running on SSB crashes for some reason, it is possible that a chunk becomes broken: it contains log messages, but the chunk was not finished completely. However, starting with SSB version 2 F1 the syslog-ng application running on SSB processes log messages into a journal file before writing them to the logstore file, reducing message loss even in the case of an unexpected crash.
Similarly, if the indexer application crashes for some reason, it may be possible that some parts of a logstore file are not indexed, and therefore the messages from this part of the file do not appear in search results. This does not mean that the messages are lost. Currently it is not possible to reindex a file.
These limitations will be addressed in future versions of SSB.
The indexer service saves the indexes for the fields that are selected and makes them searchable. Indexing fields consumes disk space and processing power.
This section lists the limitations of the indexer service, and provides instructions for configuring indexing for logstores.
Limitations:
-
Messages are tokenized based on the specified separator characters. Only the first 512 tokens are indexed in a message, the rest are ignored. This limitation does not affect other static fields (PROGRAM, HOST, and so on) or name-value pairs added by the pattern database or values coming from the SDATA part of incoming messages.
-
Whitespace characters (space, tabulator and so on) are always treated as delimiters.
-
Tokens that are shorter than 2 characters are not indexed.
-
Tokens are truncated to 59 characters. Therefore, tokens with at least 59 characters long common prefix will be handled as identical ones.
-
When indexing name-value pairs, the 59 characters limitation is applied to this format: "<name-of-nvpair>=<value-of-nvpair>". Do not use long name parts, in order to avoid the premature truncation of the value part.
-
The shortest timeframe for searching and creating statistics is 1 second. Smaller interval cannot be used.
-
The order of the tokens in a message is not preserved. Therefore, if one message contains 'first_token second_token' and another message contains 'second_token first_token' search expressions such as 'first_token second_token' will find both messages.
To configure the indexer service
-
Navigate to Log > Logspaces and select the logstore to index.
-
To enable automatic indexing of the logstore files, select the Enable option of the Indexer field.
-
To limit the number of hits when searching in the logstore, enter the maximum number of search result hits in the Maximum number of search results field.
To disable the limit, enter 0.
-
Enter the maximum amount of memory the indexer can use for the current logspace in the Memory limit field.
|
|
Caution:
Hazard of data loss. Increasing the Memory limit option too high (1280 MB) can cause message loss and degraded performance. The exact values that can cause problems depend on your configuration and environment.
Make sure that the sum of the memory of all indexed logspaces is smaller than the available memory in your syslog-ng Store Box (SSB) appliance. |
-
Configure the fields to be indexed in the Indexed fields.
NOTE: At least one field must be selected.
The following fields can be indexed: Facility, Priority, Program, Pid, Host, Tags, Name/value pairs, Message.
For the Name/value pairs field, select All to index all Name/value fields or enter the names to be indexed in the Only with the name field as comma-separated names.
If the indexing of the Message field is enabled, the current Delimiters are displayed. By default, the indexer uses the following delimiter characters to separate the message into words (tokens): & : ~ ? ! [ ] = , ; ( ) ' ".
If your messages contain segments that include one of these delimiters, and you want to search for these segments as a whole, remove the delimiter from the list. For example, if your log messages contain MAC addresses, and you want to be able to search for messages that contain a particular MAC address, delete the colon (:) character from the list of delimiters. Otherwise, the indexer will separate the MAC address into several tokens.
NOTE: It is not possible to search for the whitespace ( ) character in the MESSAGE part of the log message, since it is a hard-coded delimiter character.
.
in the Encryption certificate field.
.