Forwarding log messages to Google Pub/Sub
This section describes how to forward messages from syslog-ng Store Box (SSB) to the Google Pub/Sub messaging service (Google Pub/Sub).
From version 6.7.0, SSB uses the syslog-ng Premium Edition (syslog-ng PE) application’s support to generate your own messaging Google Pub/Sub infrastructure with syslog-ng PE as a “Publisher” entity, utilizing the HTTP REST interface of the service. The Google Pub/Sub destination is an asynchronous messaging service connected to Google’s infrastructure.
For more information about Google Pub/Sub’s messaging service, see What Is Pub/Sub? in the Google Pub/Sub online documentation.
NOTE: This section and the other Google Pub/Sub-related sections in this documentation are based on Google Pub/Sub messaging service concepts and terminology. If you do not use the Google Pub/Sub messaging service on a regular basis, One Identity recommends that you read the Google Pub/Sub overview documentation in the Google Pub/Sub online documentation to familiarize yourself with the messaging service's concepts and terminology before you continue reading these sections.
Prerequisites
This section describes the prerequisites to forwarding messages from syslog-ng Store Box (SSB) to the Google Pub/Sub messaging service (Google Pub/Sub).
NOTE: This section and the other Google Pub/Sub-related sections in this documentation are based on Google Pub/Sub messaging service concepts and terminology. If you do not use the Google Pub/Sub messaging service on a regular basis, One Identity recommends that you read the Google Pub/Sub overview documentation in the Google Pub/Sub online documentation to familiarize yourself with the messaging service's concepts and terminology before you continue reading these sections.
To configure the Google Pub/Sub destination, you must have:
Limitations
This section describes the limitations to using syslog-ng Store Box (SSB) to forward messages to the Google Pub/Sub messaging service (Google Pub/Sub).
The current implementation of the Google Pub/Sub destination has the following limitations:
-
No message-based acknowledgement
While Google Pub/Sub acknowledges the batch of received messages, it also sends individual acknowledgement IDs to each message. However, SSB currently does not track individual messages inside Google Pub/Sub. Under normal operational circumstances, the lack of tracking individual messages has no effect on message delivery, and even allows flow control to work properly. However, in case of an error, the only solution is to repeat the entire batch, which can lead to message duplication in case Google Pub/Sub acknowledged part of the previous batch in spite of indicating an overall error.
NOTE: This behavior, called At-Least-Once delivery, means that if an error occurs, it is more acceptable to duplicate messages than to lose any of them.
NOTE: The Google Pub/Sub destination can not fetch logs, only serve as a “Publisher” entity in the Google Pub/Sub service.
-
Messages with HTTP 400 response code will be dropped
If the message sent to Google Pub/Sub is invalid, the Google Pub/Sub messaging service will reply with an HTTP 400 response code.
The message can be invalid for either of these reasons:
-
A required argument is missing from the message.
-
The message size exceeds limits.
-
The message itself has an invalid format.
In these cases, SSB cannot successfully send the messages to Google Pub/Sub. These messages would prevent SSB from sending further messages to the messaging service, therefore SSB must drop them.
For further details on HTTP error codes of the Google Pub/Sub messaging service, see Cloud Pub/Sub > Documentation > Reference > Error Codes in the Google Pub/Sub Reference Documentation.
-
Proxy limitations
If you use a proxy, consider that only HTTP proxies are supported.
Configuring the Google Pub/Sub destination: adding a new Google Pub/Sub destination
This section describes the first steps of configuring the Google Pub/Sub destination, that is, adding a new Google Pub/Sub destination on the syslog-ng Store Box (SSB) web interface.
For more information about configuring the authentication and workspace settings of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Authentication and workspace settings.
For more information about configuring the advanced message parameters of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Advanced message parameters.
For more information about configuring the performance-related settings of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Performance-related settings.
NOTE: This section and the other Google Pub/Sub-related sections in this documentation are based on Google Pub/Sub messaging service concepts and terminology. If you do not use the Google Pub/Sub messaging service on a regular basis, One Identity recommends that you read the Google Pub/Sub overview documentation in the Google Pub/Sub online documentation to familiarize yourself with the messaging service's concepts and terminology before you continue reading these sections.
To create your custom Google Pub/Sub destination on the SSB web interface
-
Navigate to Log > Destinations, and select 
to create a new destination.
-
Under Destination type, select Google Pub/Sub destination.
Figure 177: Log > Destination — Adding a new Google Pub/Sub destination
-
After creating your Google Pub/Sub destination, continue customizing it by configuring the following:
