Chat now with support
Chat with Support

syslog-ng Store Box 7.3.0 - Administration Guide

Preface Introduction The concepts of SSB The Welcome Wizard and the first login Basic settings User management and access control Managing SSB Configuring message sources Storing messages on SSB Forwarding messages from SSB Log paths: routing and processing messages Configuring syslog-ng options Searching log messages Searching the internal messages of SSB Classifying messages with pattern databases The SSB RPC API Monitoring SSB Troubleshooting SSB Security checklist for configuring SSB Glossary

Limitations

This section describes the limitations to using syslog-ng Store Box (SSB) to forward messages to the Google Pub/Sub messaging service (Google Pub/Sub).

The current implementation of the Google Pub/Sub destination has the following limitations:

  • No message-based acknowledgement

    While Google Pub/Sub acknowledges the batch of received messages, it also sends individual acknowledgement IDs to each message. However, SSB currently does not track individual messages inside Google Pub/Sub. Under normal operational circumstances, the lack of tracking individual messages has no effect on message delivery, and even allows flow control to work properly. However, in case of an error, the only solution is to repeat the entire batch, which can lead to message duplication in case Google Pub/Sub acknowledged part of the previous batch in spite of indicating an overall error.

    NOTE: This behavior, called At-Least-Once delivery, means that if an error occurs, it is more acceptable to duplicate messages than to lose any of them.

    NOTE: The Google Pub/Sub destination can not fetch logs, only serve as a “Publisher” entity in the Google Pub/Sub service.

  • Messages with HTTP 400 response code will be dropped

    If the message sent to Google Pub/Sub is invalid, the Google Pub/Sub messaging service will reply with an HTTP 400 response code.

    The message can be invalid for either of these reasons:

    • A required argument is missing from the message.

    • The message size exceeds limits.

    • The message itself has an invalid format.

    In these cases, SSB cannot successfully send the messages to Google Pub/Sub. These messages would prevent SSB from sending further messages to the messaging service, therefore SSB must drop them.

    For further details on HTTP error codes of the Google Pub/Sub messaging service, see Cloud Pub/Sub > Documentation > Reference > Error Codes in the Google Pub/Sub Reference Documentation.

  • Proxy limitations

    If you use a proxy, consider that only HTTP proxies are supported.

Configuring the Google Pub/Sub destination: adding a new Google Pub/Sub destination

This section describes the first steps of configuring the Google Pub/Sub destination, that is, adding a new Google Pub/Sub destination on the syslog-ng Store Box (SSB) web interface.

For more information about configuring the authentication and workspace settings of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Authentication and workspace settings.

For more information about configuring the advanced message parameters of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Advanced message parameters.

For more information about configuring the performance-related settings of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Performance-related settings.

NOTE: This section and the other Google Pub/Sub-related sections in this documentation are based on Google Pub/Sub messaging service concepts and terminology. If you do not use the Google Pub/Sub messaging service on a regular basis, One Identity recommends that you read the Google Pub/Sub overview documentation in the Google Pub/Sub online documentation to familiarize yourself with the messaging service's concepts and terminology before you continue reading these sections.

To create your custom Google Pub/Sub destination on the SSB web interface

  1. Navigate to Log > Destinations, and select to create a new destination.

  2. Under Destination type, select Google Pub/Sub destination.

    Figure 182: Log > Destination — Adding a new Google Pub/Sub destination

  3. After creating your Google Pub/Sub destination, continue customizing it by configuring the following:

Configuring the Google Pub/Sub destination: Authentication and workspace settings

This section describes how you can configure the authentication and workspace settings after adding a new Google Pub/Sub destination.

For information about configuring the advanced message parameters of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Advanced message parameters.

For information about configuring the performance-related settings of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Performance-related settings.

NOTE: This section and the other Google Pub/Sub-related sections in this documentation are based on Google Pub/Sub messaging service concepts and terminology. If you do not use the Google Pub/Sub messaging service on a regular basis, One Identity recommends that you read the Google Pub/Sub overview documentation in the Google Pub/Sub online documentation to familiarize yourself with the messaging service's concepts and terminology before you continue reading these sections.

To configure the authentication and workspace settings for your Google Pub/Sub destination

  1. Navigate to Log > Destination > <your-pubsub-destination> > Authentication and workspace settings.

    Figure 183: Log > Destination > <your-pubsub-destination> — Configuring the authentication and workspace settings

  2. In the Project field, enter the Google Pub/Sub project ID of your choice.

  3. In the Topic field, enter the Google Pub/Sub topic of your choice.

    For more information about setting up Google Pub/Sub projects and topics, see Building a Pub/Sub messaging system > Tutorial setup > Set up your Google Cloud project and Pub/Sub topic and subscriptions in the Google Pub/Sub online documentation.

  4. In the Credentials field, upload your Google Pub/Sub service account credentials.

    After you upload your Google Pub/Sub service account credentials JSON file, the web interface will look like this:

    Figure 184: Log > Destination > <your-pubsub-destination> — Credentials uploaded

    For more information about creating Google Pub/Sub credentials, see Building a Pub/Sub messaging system > Tutorial setup > Create service account credentials in the Google Pub/Sub online documentation.

  5. In the OAuth 2.0 Scope field, enter the OAuth scope to be able to publish into the Google Pub/Sub messaging service.

    For more information about Google Pub/Sub OAuth scopes, see OAuth 2.0 Scopes for Google APIs > Cloud Pub/Sub API or OAuth Scopes in the Google Identity online documentation.

  6. (Optional) Select Use proxy, and in the Proxy field, enter the HTTP proxy URL.

Configuring the Google Pub/Sub destination: Advanced message parameters

This section describes how you can configure the advanced message parameters for your Google Pub/Sub destination after you finish configuring the authentication and workspace settings.

For information about adding a new Google Pub/Sub destination on the SSB web interface, see Configuring the Google Pub/Sub destination: adding a new Google Pub/Sub destination.

For information about configuring the performance-related settings of your Google Pub/Sub destination, see Configuring the Google Pub/Sub destination: Performance-related settings.

NOTE: This section and the other Google Pub/Sub-related sections in this documentation are based on Google Pub/Sub messaging service concepts and terminology. If you do not use the Google Pub/Sub messaging service on a regular basis, One Identity recommends that you read the Google Pub/Sub overview documentation in the Google Pub/Sub online documentation to familiarize yourself with the messaging service's concepts and terminology before you continue reading these sections.

To configure the advanced message parameters for your Google Pub/Sub destination

  1. Navigate to Log > Destination > <your-pubsub-destination> > Advanced message parameters.

    Figure 185: Log > Destination > <your-pubsub-destination> — Configuring the advanced message parameters for your Google Pub/Sub destination

  2. In the Body field, customize the message format sent to the Google Pub/Sub messaging service by entering the message format you want.

CAUTION: Hazard of data loss!

Make sure that the customized message format is accepted by the Google Pub/Sub messaging service. For invalid messages, SSB will receive an HTTP 400 response code and messages with such a response code will be dropped.

For more information on dropping messages with HTTP 400 response code, see Limitations.

For more information on the $(format-json) and the $(format-flat-json) template functions, see Manipulating messages > Customizing message format using macros and templates > Template functions of syslog-ng PE in the syslog-ng PE Administration Guide.

For more information on Google Pub/Sub message format, see Publishing messages to topics > Message format or Pubsub Message in the Google Pub/Sub Reference Documentation.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating