Configuring the Google Pub/Sub destination: Authentication and workspace settings
This section describes how you can configure the authentication and workspace settings after adding a new Google Pub/Sub destination.
For information about configuring the advanced message parameters of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Advanced message parameters.
For information about configuring the performance-related settings of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Performance-related settings.
To configure the authentication and workspace settings for your Google Pub/Sub destination
-
Navigate to Log > Destination > <your-pubsub-destination> > Authentication and workspace settings.
Figure 175: Log > Destination > <your-pubsub-destination> — Configuring the authentication and workspace settings
-
In the Project field, enter the Google Pub/Sub project ID of your choice.
-
In the Topic field, enter the Google Pub/Sub topic of your choice.
For more information about setting up Google Pub/Sub projects and topics, see Building a Pub/Sub messaging system > Tutorial setup > Set up your Google Cloud project and Pub/Sub topic and subscriptions in the Google Pub/Sub online documentation.
-
In the Credentials field, upload your Google Pub/Sub service account credentials.
After you upload your Google Pub/Sub service account credentials JSON file, the web interface will look like this:
Figure 176: Log > Destination > <your-pubsub-destination> — Credentials uploaded
For more information about creating Google Pub/Sub credentials, see Building a Pub/Sub messaging system > Tutorial setup > Create service account credentials in the Google Pub/Sub online documentation.
-
In the OAuth 2.0 Scope field, enter the OAuth scope to be able to publish into the Google Pub/Sub messaging service.
For more information about Google Pub/Sub OAuth scopes, see OAuth 2.0 Scopes for Google APIs > Cloud Pub/Sub API or OAuth Scopes in the Google Identity online documentation
-
(Optional) Select Use proxy, and in the Proxy field, enter the HTTP proxy URL.
Configuring the Google Pub/Sub destination: Advanced message parameters
This section describes how you can configure the advanced message parameters for your Google Pub/Sub destination after you finish configuring the authentication and workspace settings.
For information about adding a new Google Pub/Sub destination on the SSB web interface, see Configuring the Google Pub/Sub destination: adding a new Google Pub/Sub destination.
For information about configuring the performance-related settings of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Performance-related settings.
To configure the advanced message parameters for your Google Pub/Sub destination
-
Navigate to Log > Destination > <your-pubsub-destination> > Advanced message parameters.
Figure 177: Log > Destination > <your-pubsub-destination> — Configuring the advanced message parameters for your Google Pub/Sub destination
-
In the Body field, customize the message format sent to the Google Pub/Sub messaging service by entering the message format you want.
Configuring the Google Pub/Sub destination: Performance-related settings
This section describes configuring the performance-related settings of your Google Pub/Sub destination after you finish configuring the authentication and workspace settings and the advanced message parameters for the Google Pub/Sub destination.
To configure the performance-related settings of your Google Pub/Sub destination
-
Navigate to Log > Destination > <your-pubsub-destination> > Performance-related settings.
Figure 178: Log > Destination > <your-pubsub-destination> — Configuring the authentication and workspace settings
-
In the Number of workers field, set the number of worker threads that you want SSB to use when sending messages to the server.
|
|
CAUTION: Hazard of data loss!
When you use more than one worker threads together with disk-buffering enabled, consider that the syslog-ng PE application behind SSB creates a separate disk-buffer file for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk-buffer files. To avoid data loss, One Identity recommends that you do not decrease the number of workers when the disk-buffer files are in use. |
NOTE: Increasing the number of worker threads can drastically improve the performance of the destination.
-
In the Timeout field, specify the timeout (in seconds) that you want SSB to wait for an operation to complete, and then attempt to reconnect the server if the configured timeout limit is exceeded.
The default value of the Timeout setting is 0, which means that it is disabled by default.
-
In the Batch lines field, specify how many lines you want SSB to flush to a destination in one batch.
For further information on the maximum number of Batch lines, see the Publish request limit at Cloud Pub/Sub > Documentation > Resources > Quotas and limits > Resource limits.
NOTE: SSB waits for the configured number of lines to accumulate, and when this number is reached, SSB sends the message lines to the destination in a single batch. For example, if you set Batch lines to 100, SSB waits for 100 message lines before sending them in one batch.
NOTE: Consider the following when configuring the number of batch lines:
-
Increasing the number of batch lines increases throughput (because more messages are sent in a single batch), but also increases message latency.
-
If the Batch-timeout option is disabled, the syslog-ng PE application behind SSB flushes the messages if it has sent the number of messages specified in Batch lines, or the queue became empty. If you stop or reload the syslog-ng PE application behind SSB, or if in case of network sources, the connection with the client is closed, the syslog-ng PE behind SSB automatically sends the unsent messages to the destination.
-
If the Batch-timeout option is enabled and the queue becomes empty, SSB flushes the messages only if Batch timeout expires, or if the batch reaches the limit set in Batch lines.
NOTE: Depending on your source configuration settings, your batch may not reach the Batch lines limit before your queue becomes empty, and SSB forwards your messages.
-
In the Batch-bytes field, set the maximum size of payload in a batch (in bytes).
For the maximum value of Batch-bytes, see the Publish request limit at Cloud Pub/Sub > Documentation > Resources > Quotas and limits > Resource limits.
NOTE: When configuring Batch-bytes, consider the following:
-
If the size of the messages reaches this value, the syslog-ng PE application behind SSB sends the batch the Google Pub/Sub messaging service even if the number of messages is less than the value you configure in the Batch-bytes field.
-
If Batch-timeout is enabled and the queue becomes empty, SSB flushes the messages only if Batch-timeout value expires, or if the message batch reaches the limit set in the Batch-bytes field.
-
(Optional) Select Batch-timeout, and in the Batch-timeout value field, specify the time SSB waits for Batch lines to accumulate in the output buffer.
SSB sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, SSB sends messages to the destination once every Batch timeout milliseconds at most.
Forwarding log messages to Splunk
This section describes how to forward messages from syslog-ng Store Box (SSB) to Splunk.
From version 6.6.0, SSB uses the syslog-ng Premium Edition (syslog-ng PE) application's support to post messages to a Splunk deployment in JSON format, using the HTTP Event Collector (HEC) over HTTP and Secure HTTP (HTTPS).
For further details about how the application forwards messages to Splunk, see splunk-hec: Sending messages to Splunk HTTP Event Collector in the syslog-ng PE Administration Guide.
