Chat now with support
Chat with Support

syslog-ng Store Box 6.9.0 - Administration Guide

Preface Introduction The concepts of SSB The Welcome Wizard and the first login Basic settings User management and access control Managing SSB Configuring message sources Storing messages on SSB Forwarding messages from SSB Log paths: routing and processing messages Configuring syslog-ng options Searching log messages Searching the internal messages of SSB Classifying messages with pattern databases The SSB RPC API Monitoring SSB Troubleshooting SSB Security checklist for configuring SSB Glossary

Configuring the Azure Sentinel destination: Performance-related settings

This section describes configuring the performance-related settings of the Azure Sentinel destination after configuring the authentication and workspace settings and the advanced message parameters.

For more information about adding a new Azure Sentinel destination, see Configuring the Azure Sentinel destination: adding a new Azure Sentinel destination.

NOTE: This section and the other Azure Sentinel-related sections in this documentation are based on Azure Sentinel messaging service concepts and terminology. If you do not use the Azure Sentinel messaging service on a regular basis, One Identity recommends that you read the Azure Sentinel quick-start documentation to familiarize yourself with the messaging service's concepts and terminology before you continue reading these sections.

To configure the performance-related settings of your Azure Sentinel destination

  1. Navigate to Log > Destination > <your-sentinel-destination> > Performance-related settings.

    Figure 158: Log > Destinations > <your-sentinel-destination> — Configuring the performance-related settings

  2. In the Number of workers field, set the number of worker threads that you want SSB to use when sending messages to the server.

    CAUTION: Hazard of data loss!

    When you use more than one worker threads together with disk-buffering enabled, consider that the syslog-ng PE application behind SSB creates a separate disk-buffer file for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk-buffer files.

    To avoid data loss, One Identity recommends that you do not decrease the number of workers when the disk-buffer files are in use.

    NOTE: Increasing the number of worker threads can drastically improve the performance of the destination.

  3. In the Timeout field, specify the timeout (in seconds) that you want SSB to wait for an operation to complete, and attempt to reconnect to the server if the configured timeout limit is exceeded.

    The default value of the Timeout setting is 0, which means that it is disabled by default.

  4. In the Batch lines field, specify how many lines you want SSB to flush to a destination in one batch.

    NOTE: SSB waits for the configured number of lines to accumulate, and when this number is reached, SSB sends the message lines to the destination in a single batch. For example, if you set Batch lines to 100, SSB waits for 100 message lines before sending them in one batch.

    NOTE: Consider the following when configuring the number of batch lines:

    • Increasing the number of batch lines increases throughput (because more messages are sent in a single batch), but also increases message latency.

    • If the Batch-timeout option is disabled, the syslog-ng PE application behind SSB flushes the messages if it has sent the number of messages specified in Batch lines, or the queue became empty. If you stop or reload the syslog-ng PE application behind SSB, or if in case of network sources, the connection with the client is closed, the syslog-ng PE behind SSB automatically sends the unsent messages to the destination.

    • If the Batch-timeout option is enabled and the queue becomes empty, SSB flushes the messages only if Batch timeout expires, or if the batch reaches the limit set in Batch lines.

    NOTE: Depending on your source configuration settings, your batch may not reach the Batch lines limit before your queue becomes empty, and SSB forwards your messages.

  5. In the Batch-bytes field, set the maximum size of payload in a batch (in bytes).

    NOTE: When configuring Batch-bytes, consider the following:

    • If the size of the messages reaches this value, the syslog-ng PE application behind SSB sends the batch to Azure Sentinel even if the number of messages is less than the value you configure in the Batch-bytes field.

    • If Batch-timeout is enabled and the queue becomes empty, SSB flushes the messages only if the value specified in Batch-timeout expires, or if the message batch reaches the limit set in the Batch-bytes field.

  6. (Optional) Select Batch-timeout, and in the Batch-timeout value field, specify the time SSB waits for Batch lines to accumulate in the output buffer.

    SSB sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, SSB sends messages to the destination once every Batch timeout milliseconds at most.

Forwarding log messages to Google Pub/Sub

This section describes how to forward messages from syslog-ng Store Box (SSB) to the Google Pub/Sub messaging service (Google Pub/Sub).

From version 6.7.0, SSB uses the syslog-ng Premium Edition (syslog-ng PE) application’s support to generate your own messaging Google Pub/Sub infrastructure with syslog-ng PE as a “Publisher” entity, utilizing the HTTP REST interface of the service. The Google Pub/Sub destination is an asynchronous messaging service connected to Google’s infrastructure.

For more information about Google Pub/Sub’s messaging service, see What Is Pub/Sub? in the Google Pub/Sub online documentation.

NOTE: This section and the other Google Pub/Sub-related sections in this documentation are based on Google Pub/Sub messaging service concepts and terminology. If you do not use the Google Pub/Sub messaging service on a regular basis, One Identity recommends that you read the Google Pub/Sub overview documentation in the Google Pub/Sub online documentation to familiarize yourself with the messaging service's concepts and terminology before you continue reading these sections.

Prerequisites

This section describes the prerequisites to forwarding messages from syslog-ng Store Box (SSB) to the Google Pub/Sub messaging service (Google Pub/Sub).

NOTE: This section and the other Google Pub/Sub-related sections in this documentation are based on Google Pub/Sub messaging service concepts and terminology. If you do not use the Google Pub/Sub messaging service on a regular basis, One Identity recommends that you read the Google Pub/Sub overview documentation in the Google Pub/Sub online documentation to familiarize yourself with the messaging service's concepts and terminology before you continue reading these sections.

To configure the Google Pub/Sub destination, you must have:

Limitations

This section describes the limitations to using syslog-ng Store Box (SSB) to forward messages to the Google Pub/Sub messaging service (Google Pub/Sub).

The current implementation of the Google Pub/Sub destination has the following limitations:

  • No message-based acknowledgement

    While Google Pub/Sub acknowledges the batch of received messages, it also sends individual acknowledgement IDs to each message. However, SSB currently does not track individual messages inside Google Pub/Sub. Under normal operational circumstances, the lack of tracking individual messages has no effect on message delivery, and even allows flow control to work properly. However, in case of an error, the only solution is to repeat the entire batch, which can lead to message duplication in case Google Pub/Sub acknowledged part of the previous batch in spite of indicating an overall error.

    NOTE: This behavior, called At-Least-Once delivery, means that if an error occurs, it is more acceptable to duplicate messages than to lose any of them.

    NOTE: The Google Pub/Sub destination can not fetch logs, only serve as a “Publisher” entity in the Google Pub/Sub service.

  • Messages with HTTP 400 response code will be dropped

    If the message sent to Google Pub/Sub is invalid, the Google Pub/Sub messaging service will reply with an HTTP 400 response code.

    The message can be invalid for either of these reasons:

    • A required argument is missing from the message.

    • The message size exceeds limits.

    • The message itself has an invalid format.

    In these cases, SSB cannot successfully send the messages to Google Pub/Sub. These messages would prevent SSB from sending further messages to the messaging service, therefore SSB must drop them.

    For further details on HTTP error codes of the Google Pub/Sub messaging service, see Cloud Pub/Sub > Documentation > Reference > Error Codes in the Google Pub/Sub Reference Documentation.

  • Proxy limitations

    If you use a proxy, consider that only HTTP proxies are supported.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating