Chat now with support
Chat with Support

syslog-ng Store Box 5.0.3 - Administration Guide

Preface Introduction The concepts of SSB The Welcome Wizard and the first login Basic settings User management and access control Managing SSB Configuring message sources Storing messages on SSB Forwarding messages from SSB Log paths: routing and processing messages Configuring syslog-ng options Searching log messages Searching the internal messages of SSB Classifying messages with pattern databases The SSB RPC API Troubleshooting SSB Security checklist for configuring SSB About us Third-party contributions

Creating a fetch query manually

To create a fetch query, complete the following steps.

Caution:

The SSB application does not validate or limit the contents of customized queries. Consequently, queries performed with a user with write-access can potentially modify or even harm the database. Use customized queries with care, and only for your own responsibility.

The query must return message parts with the following column names:

  • uid:

    The uid column must contain a unique number. This number must increase monotonously. SSB will store the last read uid in $last_read_uid macro. To prevent rereading the whole table, filter records that are newer than the last read record by adding WHERE <column_name_containing_the_id> > $last_read_uid clause to the query. (Note that $last_read_uid will be substituted by SSB appropriately.)

    Add the clause ORDER BY <column_name_containing_the_id> at the end of the query to prevent redundant search results. .

  • datetime or date and time:

    SSB will use the content of the datetime column as the timestamp of the log message. The following column types are supported:

    • MySQL: timestamp, datetime, int

    • PostgreSQL: timestamp, int

    • Oracle: timestamp, int

    • MSSQL: datetime, int

    If the type is int, SSB will assume that it contains a UNIX timestamp.

    When using separate date and time columns, the date column must be date type, the time column must be time type.

  • message:

    The message field must contain the message to be logged.

  • host (optional):

  • program (optional):

For example:

SELECT "uniq_id_number" AS "uid", "bsd_datetime" AS "datetime", "message" FROM "test_table"

The host, program, and timezone parameters can be selected from columns or set as a fix value. The timezone must contain time-shifting value and not the name of the time zone. For example:

SELECT "myhost" AS "host", "myprogram" AS "program", "+01:00" AS "timezone", <further-parts-of-the-query>

NOTE:

The query must not contain any comments.

Example: SQL source fetch_query

The following queries records that are older than the last read record:

SELECT * FROM <table_name> WHERE uid > $last_read_uid ORDER BY uid LIMIT 3000

Or a more detailed example:

SELECT "uniq_id_number" AS "uid", "date_string" as "datetime", "message" as "message" FROM "test_mysql" WHERE "uniq_id_number" > $last_read_uid ORDER BY "uniq_id_number"
Query to fetch the last UID from the table

If you are using MSSQL database, or you encounter SQL errors or unexpected results, specify a custom query to find the last UID in the database.

The last UID of the table is necessary for finding the initial position in the database. By default, SSB will use the maximum value of the "uid" column from the query specified above for this purpose. However, if it does not seem to produce the required results, you can specify a custom query here

If the Read old records option is enabled for this database source, this field is not used.

Example: Query to fetch the last UID from the table

The following queries the last UID of the table:

SELECT max uid FROM <further-parts-of-the-query>

NOTE:

If you are using MSSQL or MySQL database, you also have to limit the number of results of the fetch query. for example: SELECT top x <further-parts-of-the-query>. This limit must be lower than the internal SSB limit, that is 3000. In case you set a limit larger than 3000, it will be ignored and can result in performance issues.

Storing messages on SSB

SSB stores log messages in binary or plain text log files called logspaces. You can define multiple logspaces, remote logspaces, and configure filtered subsets of each logspace.

Binary log files (logstores) correspond to the encrypted logstore() destination of syslog-ng. Logstores can be compressed, encrypted, and timestamped by an external Timestamping Authority (TSA). To make the contents of the logstore searchable, you can create a separate indexer configuration for each logstore.

A multiple logspace aggregates messages from multiple SSBs (located at different sites), allowing you to view and search the logs of several SSBs from a single web interface without having to log on to several different interfaces.

Remote logspaces enable you to access and search logspaces (including filtered logspaces) on other SSB appliances.

Filtered logspaces allow you to create a smaller, filtered subset of the logs contained in an existing local, remote or multiple logspace. Assigning a user group to a filtered logspace enables fine-grained access control by creating a group that sees only a subset of the logs from a logspace.

Summary of multiple, remote, and filtered logspace types provides a summary and comparison of these three logspace types.

Table 7: Summary of multiple, remote, and filtered logspace types
Logspace type Source Main use case Can be searched Can be filtered
Multiple Multiple SSBs located at different sites

Aggregate messages from multiple logspaces into a single logspace

Pre-filter log messages and share with only select user groups

Remote Remote SSB Access a logspace on another SSB
Filtered Local / multiple / remote SSB(s) Control access to a logspace at a granular level by granting access only to a subset of a logspace N/A

By default, SSB has the following logspaces:

Figure 101: Log > Logspaces — Default logspaces in SSB

  • local: An unencrypted, binary logspace for storing the log messages of SSB.

  • center:: An unencrypted, binary logspace for storing the log messages sent by the clients.

Logspaces are stored locally on the hard disk of SSB. To access a logspace remotely, you can configure another SSB to view and search the logspace as a remote logspace, or you can make the logspace accessible as a network drive.

Using logstores

Logstores are logspaces with binary log files for storing log messages sent by the clients. Logstores can be compressed, encrypted, and timestamped by an external Timestamping Authority (TSA). To make the contents of the logstore searchable, you can create a separate indexer configuration for each logstore.

The following limitations apply to logstores:

  • Indexing logstore files is currently limited: the indexer can handle only one file from a logstore for every day (SSB automatically starts a new log file for every day).

  • Logstore files consist of chunks. In rare cases, if the syslog-ng application running on SSB crashes for some reason, it is possible that a chunk becomes broken: it contains log messages, but the chunk was not finished completely. However, starting with SSB version 2 F1 the syslog-ng application running on SSB processes log messages into a journal file before writing them to the logstore file, reducing message loss even in the case of an unexpected crash.

    Similarly, if the indexer application crashes for some reason, it may be possible that some parts of a logstore file are not indexed, and therefore the messages from this part of the file do not appear in search results. This does not mean that the messages are lost. Currently it is not possible to reindex a file.

These limitations will be addressed in future versions of SSB.

Creating logstores

Steps:
  1. Navigate to Log > Logspaces and click .

  2. Enter a name for the logspace into the top field. Use descriptive names that help you to identify the source easily. Note that the name of the logspace must begin with a number or a letter.

    Figure 102: Log > Logspaces — Creating a new logstore

  3. Select LogStore from the Type field.

  4. To encrypt the log files using public-key encryption, click in the Encryption certificate field.

    A pop-up window is displayed.

    Click Browse, select the certificate you want to use to encrypt the log files, then click Upload. Alternatively, you can paste the certificate into the Certificate field and click Upload.

    NOTE:

    To view encrypted log messages, you will need the private key of this certificate. For details on browsing encrypted logstores online on the SSB web interface, see Browsing encrypted logspaces. Encrypted log files can be displayed using the logcat command-line tool as well. The logcat application is currently available only for UNIX-based systems.

    One Identity recommends:

    • Using 2048-bit RSA keys (or stronger).

    • Using the SHA-256 hash algorithm (or stronger) when creating the public key fingerprint.

    NOTE:

    Each certificate or encryption-related setting described above only takes effect from the next day.

    However, if you use decryption private keys, you can search in the encrypted logstores immediately after the private keys are uploaded. For more information, see Assigning decryption keys to a logstore.

  5. By default, SSB requests a timestamp every ten minutes from the internal Timestamping Authority. Adjust the frequency of timestamping requests in the Timestamping frequency field if needed. For details on how to request timestamps from an external provider, see Timestamping configuration on SSB.

  6. Indexing is enabled by default. For detailed instructions on configuring indexing, see Configuring the indexer service.

  7. Logstore files are compressed by default. If you do not want to use compression, uncheck the Compressed logstore option.

  8. Select how to organize the log files of this logspace from the Filename template field.

    • To save every message received during a day into a single file, select All messages in one file.

    • To create a separate log file for every peer (IP address or hostname) that sends messages, select the Per host option. This option corresponds to using the ${HOST} macro of syslog-ng.

    • To create a separate log file for every application that sends messages, select the Per application option. This option corresponds to using the ${PROGRAM} macro of syslog-ng.

    • To create a separate log file for every application of every peer (IP address or hostname) that sends messages, select Per host and application option. This option corresponds to using the ${HOST}-${PROGRAM} macros of syslog-ng.

    • To specify a custom template for naming the log files, select the Custom option and enter the template into the appearing Template field.

      NOTE:

      Templates that generate an invalid path (for example, they use a filename longer than 246 characters or refer to a parent directory) will not work.

      For details on using filename templates, see The syslog-ng Premium Edition 6 LTS Administrator Guide.

  9. To create automatic daily backups of the logspace to a remote server, create a backup policy and select it from the Backup policy field. For details on creating backup policies, see Data and configuration backups.

  10. To archive the logspace automatically daily, create an archiving policy and select it from the Archive/Cleanup policy field. For details on creating archiving policies, see Archiving and cleanup.

    Caution:

    Use archiving and cleanup policies to remove older logfiles from SSB, otherwise the hard disk of SSB may become full.

  11. To make the log files of this logspace available via the network, create a sharing policy and select it from the Sharing policy field. For details on creating sharing policies, see Accessing log files across the network.

  12. Set a size for the logspace in the Warning size field: SSB will send an alert if the size of this logspace exceeds the limit.

    Caution:

    Make sure that the Logspace exceeded warning size alert is enabled in Basic Settings > Alerting & Monitoring page, and that the mail and SNMP settings of the Basic Settings > Management page are correct. Otherwise, you will not receive any alert when the logspace exceeds the size limit. For details on alerting and monitoring, see also Configuring system monitoring on SSB.

  13. By default, members of the search group can view the stored messages online. Use the Access control option to control which usergroups can access the logspace. For details, see also Managing user rights and usergroups.

  14. Click Commit.

Related Documents