Chat now with support
Chat with Support

syslog-ng Store Box 6.1.0 - Administration Guide

Preface Introduction The concepts of SSB The Welcome Wizard and the first login Basic settings User management and access control Managing SSB Configuring message sources Storing messages on SSB Forwarding messages from SSB Log paths: routing and processing messages Configuring syslog-ng options Searching log messages Searching the internal messages of SSB Classifying messages with pattern databases The SSB RPC API Monitoring SSB Troubleshooting SSB Security checklist for configuring SSB

Using logstores

Logstores are logspaces with binary log files for storing log messages sent by the clients. Logstores can be compressed, encrypted, and timestamped by an external Timestamping Authority (TSA). To make the contents of the logstore searchable, you can create a separate indexer configuration for each logstore.

The following limitations apply to logstores:

  • Indexing logstore files is currently limited: the indexer can handle only one file from a logstore for every day (SSB automatically starts a new log file for every day).

  • Logstore files consist of chunks. In rare cases, if the syslog-ng application running on SSB crashes for some reason, it is possible that a chunk becomes broken: it contains log messages, but the chunk was not finished completely. However, starting with SSB version 2 F1 the syslog-ng application running on SSB processes log messages into a journal file before writing them to the logstore file, reducing message loss even in the case of an unexpected crash.

    Similarly, if the indexer application crashes for some reason, it may be possible that some parts of a logstore file are not indexed, and therefore the messages from this part of the file do not appear in search results. This does not mean that the messages are lost. Currently it is not possible to reindex a file.

These limitations will be addressed in future versions of SSB.

Creating logstores

To create logstores

  1. Navigate to Log > Logspaces and click .

  2. Enter a name for the logspace into the top field. Use descriptive names that help you to identify the source easily. Note that the name of the logspace must begin with a number or a letter.

    Figure 98: Log > Logspaces — Creating a new logstore

  3. Select LogStore from the Type field.

  4. To encrypt the log files using public-key encryption, click in the Encryption certificate field.

    A pop-up window is displayed.

    Click Browse, select the certificate you want to use to encrypt the log files, then click Upload. Alternatively, you can paste the certificate into the Certificate field and click Upload.

    NOTE:

    To view encrypted log messages, you will need the private key of this certificate. For details on browsing encrypted logstores online on the SSB web interface, see Browsing encrypted logspaces. Encrypted log files can be displayed using the logcat command-line tool as well. The logcat application is currently available only for UNIX-based systems.

    One Identity recommends:

    • Using 2048-bit RSA keys (or stronger).

    • Using the SHA-256 hash algorithm (or stronger) when creating the public key fingerprint.

    NOTE:

    Each certificate or encryption-related setting described above only takes effect from the next day.

    However, if you use decryption private keys, you can search in the encrypted logstores immediately after the private keys are uploaded. For more information, see Assigning decryption keys to a logstore.

  5. By default, SSB requests a timestamp every ten minutes from the internal Timestamping Authority. Adjust the frequency of timestamping requests in the Timestamping frequency field if needed. For details on how to request timestamps from an external provider, see Timestamping configuration on SSB.

  6. Indexing is enabled by default. For detailed instructions on configuring indexing, see Configuring the indexer service.

  7. Logstore files are compressed by default. If you do not want to use compression, uncheck the Compressed logstore option.

  8. Select how to organize the log files of this logspace from the Filename template field.

    • To save every message received during a day into a single file, select All messages in one file.

    • To create a separate log file for every peer (IP address or hostname) that sends messages, select the Per host option. This option corresponds to using the ${HOST} macro of syslog-ng.

    • To create a separate log file for every application that sends messages, select the Per application option. This option corresponds to using the ${PROGRAM} macro of syslog-ng.

    • To create a separate log file for every application of every peer (IP address or hostname) that sends messages, select Per host and application option. This option corresponds to using the ${HOST}-${PROGRAM} macros of syslog-ng.

    • To specify a custom template for naming the log files, select the Custom option and enter the template into the appearing Template field.

      NOTE:

      Templates that generate an invalid path (for example, they use a filename longer than 246 characters or refer to a parent directory) will not work.

      For details on using filename templates, see The syslog-ng Premium Edition 7.0 Administrator Guide.

  9. To create automatic daily backups of the logspace to a remote server, create a backup policy and select it from the Backup policy field. For details on creating backup policies, see Data and configuration backups.

  10. To archive the logspace automatically daily, create an archiving policy and select it from the Archive/Cleanup policy field. For details on creating archiving policies, see Archiving and cleanup.

    Caution:

    Use archiving and cleanup policies to remove older logfiles from SSB, otherwise the hard disk of SSB may become full.

  11. To make the log files of this logspace available via the network, create a sharing policy and select it from the Sharing policy field. For details on creating sharing policies, see Accessing log files across the network.

  12. Set a size for the logspace in the Warning size field: SSB will send an alert if the size of this logspace exceeds the limit.

    Caution:

    Make sure that the Logspace exceeded warning size alert is enabled in Basic Settings > Alerting & Monitoring page, and that the mail and SNMP settings of the Basic Settings > Management page are correct. Otherwise, you will not receive any alert when the logspace exceeds the size limit. For details on alerting and monitoring, see also Configuring system monitoring on SSB.

  13. By default, members of the search group can view the stored messages online. Use the Access control option to control which usergroups can access the logspace. For details, see also Managing user rights and usergroups.

  14. Click Commit.

Configuring the indexer service

The indexer service saves the indexes for the fields that are selected and makes them searchable. Indexing fields consumes disk space and processing power.

This section lists the limitations of the indexer service, and provides instructions for configuring indexing for logstores.

Limitations:
  • Messages are tokenized based on the specified separator characters. Only the first 512 tokens are indexed in a message, the rest are ignored. This limitation does not affect other static fields (PROGRAM, HOST, and so on) or name-value pairs added by the pattern database or values coming from the SDATA part of incoming messages.

  • Whitespace characters (space, tabulator and so on) are always treated as delimiters.

  • Tokens that are shorter than 2 characters are not indexed.

  • Tokens are truncated to 59 characters. Therefore, tokens with at least 59 characters long common prefix will be handled as identical ones.

  • When indexing name-value pairs, the 59 characters limitation is applied to this format: "<name-of-nvpair>=<value-of-nvpair>". Do not use long name parts, in order to avoid the premature truncation of the value part.

  • The shortest timeframe for searching and creating statistics is 1 second. Smaller interval cannot be used.

  • The order of the tokens in a message is not preserved. Therefore, if one message contains 'first_token second_token' and another message contains 'second_token first_token' search expressions such as 'first_token second_token' will find both messages.

To configure the indexer service

  1. Navigate to Log > Logspaces and select the logstore to index.

  2. To enable automatic indexing of the logstore files, select the Enable option of the Indexer field.

  3. To limit the number of hits when searching in the logstore, enter the maximum number of search result hits in the Maximum number of search results field.

    To disable the limit, enter 0.

  4. Enter the maximum amount of memory the indexer can use for the current logspace in the Memory limit field.

    Caution:

    Hazard of data loss. Increasing the Memory limit option too high (1280 MB) can cause message loss and degraded performance. The exact values that can cause problems depend on your configuration and environment.

    Make sure that the sum of the memory of all indexed logspaces is smaller than the available memory in your SSB appliance.

  5. Configure the fields to be indexed in the Indexed fields.

    NOTE:

    At least one field must be selected.

    The following fields can be indexed: Facility, Priority, Program, Pid, Host, Tags, Name/value pairs, Message.

    For the Name/value pairs field, select All to index all Name/value fields or enter the names to be indexed in the Only with the name field as comma-separated names.

    If the indexing of the Message field is enabled, the current Delimiters are displayed. By default, the indexer uses the following delimiter characters to separate the message into words (tokens): & : ~ ? ! [ ] = , ; ( ) ' ".

    If your messages contain segments that include one of these delimiters, and you want to search for these segments as a whole, remove the delimiter from the list. For example, if your log messages contain MAC addresses, and you want to be able to search for messages that contain a particular MAC address, delete the colon (:) character from the list of delimiters. Otherwise, the indexer will separate the MAC address into several tokens.

    NOTE:

    It is not possible to search for the whitespace ( ) character in the MESSAGE part of the log message, since it is a hard-coded delimiter character.

Viewing encrypted logs with logcat

To access logstore files, you can:

To display the contents of a logstore file, use the logcat command supplied with syslog-ng. For example:

logcat /var/log/messages.lgs

To display the contents of encrypted log files, specify the private key of the certificate used to encrypt the file. For example:

logcat -k private.key /var/log/messages.lgs

The contents of the file are sent to the standard output, so it is possible to use grep and other tools to find particular log messages. For example:

logcat /var/log/messages.lgs |grep 192.168.1.1

Every record that is stored in the logstore has a unique record ID. The logcat application can quickly jump to a specified record using the -- seek option.

For files that are in use by syslog-ng, the last chunk that is open cannot be read. Chunks are closed when their size reaches the limit set in the chunk_size parameter, or when the time limit set in the chunk_time parameter expires and no new message arrives.

When the logstore file is encrypted, a hash is also generated for every chunk to verify the integrity of the chunk. The hashes of the chunks are chained together to prevent injecting chunks into the logstore file. The encryption algorithm used is aes128 in CBC mode, the hashing (HMAC) algorithm is hmac-sha1.

Caution:

If the syslog-ng Premium Edition application or the computer crashes, an unclosed chunk remains at the end of the file. This chunk is marked as broken, its data stays there but is not shown by logcat.

Related Documents