The google-pubsub() source utilizes the At-Least-Once delivery behavior of Google Pub/Sub. This behavior is intentional, and its purpose is to avoid potential log loss. However, in certain cases, the At-Least-Once delivery behavior results in message duplication on the syslog-ng PE side.
On the Google Cloud Platform side, you can set the value of the Acknowledgment deadline (the default value is 10 seconds) when creating your Google Pub/Sub Subscription.
For more information, see Set up your Google Cloud project and Pub/Sub topic and subscriptions.
The syslog-ng PE application has to acknowledge log-fetch-limit() number of messages within the Acknowledgement deadline time limit. If syslog-ng PE does not acknowledge Google Pub/Sub messages no later than the time limit specified in the Acknowledgment deadline, the Google Pub/Sub service will attempt to redeliver the message to syslog-ng PE.
As a result, any acknowledgement sent later than the Acknowledgment deadline will result in message duplication on the syslog-ng PE side. This issue occurs most often if you have flow-control turned on, and your syslog-ng PE destinations are slow.
To avoid message duplication, you can use one of these methods:
Using the disk-buffer option if flow control is on
When using the disk-buffer option, syslog-ng PE acknowledges Pub/Sub messages as soon as they are sent to the output queue or overflow queue, instead of acknowledging them when the destination sends or rewrites them.
Adjusting the value of your ack-tracker-timeout() to the Acknowledgment deadline, and the value of your ack-tracker-batch-size() to your log-fetch-limit()
The syslog-ng PE application acknowledges messages in batches. You can set the size (ack-tracker-batch-size(), the default value is 100), and timeout (ack-tracker-timeout(), the default value is 3000 milliseconds, or 3 seconds) of these batches.
To avoid message duplication, set your ack-tracker-timeout() to a value not larger than the value of your Acknowledgment deadline, and your ack-tracker-batch-size() to a value not larger than your log-fetch-limit().