Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.34 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng PE The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files google-pubsub: collecting messages from the Google Pub/Sub messaging service wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mssql, oracle, sql: collecting messages from an SQL database network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6,udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2>: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files google_bigquery(): Sending logs to a Google BigQuery table google_bigquery_managedaccount(): Sending logs to a Google BigQuery table authenticated by Google Cloud managed service account google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service google_pubsub-managedaccount(): Sending logs to the Google Cloud Pub/Sub messaging service authenticated by Google Cloud managed service account hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka(): Publishing messages to Apache Kafka (Java implementation) (DEPRECATED) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel(): Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql(): Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transport Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

google-pubsub: collecting messages from the Google Pub/Sub messaging service

From version 7.0.22, syslog-ng Premium Edition (syslog-ng PE) can collect messages from the Google Pub/Sub messaging service using the google-pubsub() source.

NOTE: The rest of this section and its subsections assume that you are familiar with the Google Pub/Sub messaging service, and its concepts and terminology.

For more information about Google Pub/Sub's messaging service, see What Is Pub/Sub?.

For more information about setting up your Google Pub/Sub messaging service system, see Quickstart: building a functioning Pub/Sub system.

For more information about how the syslog-ng PE application sends logs to the Google Pub/Sub messaging service through the google_pubsub() destination, see google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service.

For more information about how the syslog-ng PE application sends logs to the Google Pub/Sub messaging service through the google_pubsub_managedaccounts() destination authenticating by Google Cloud managed service account, see google_pubsub-managedaccount(): Sending logs to the Google Cloud Pub/Sub messaging service authenticated by Google Cloud managed service account.

 Caution:

This is a Preview Feature, which provides an insight to planned enhancements to functionality in the product. Consider this Preview Feature a work in progress, as it may not represent the final design and functionality.

This feature has completed QA release testing, but its full impact on production systems has not been determined yet, and potential future changes in functionality and the user interface may result in compatibility issues in your current settings.

One Identity recommends the following:

  • Consider the potential risks when using this functionality in a production environment.
  • Consider the Support Policy on Product Preview Features before using this functionality in a production environment.
  • Closely and regularly keep track of official One Identity announcements about potential changes in functionality and the user interface. If these potential changes affect your configuration, check the changes you have to make in your configuration, otherwise your syslog-ng PE application may not start after upgrade.
  • Always perform tests prior to upgrades in order to avoid the risks mentioned.

However, you are welcome to try this feature and if you have any feedback, Contact One Identity.

Support Policy on Product Preview Features

The One Identity Support Team will:

  • Accept and review each service request opened regarding a Preview Feature.

  • Consider all service requests relating to a Preview Features as severity level 3.

  • Provide best effort support to resolve any issues relating to a Preview Feature.

  • Work with customers to log any product defects or enhancements relating to Preview Features.

  • Not accept requests for escalations regarding Preview Features.

  • Not provide after-hours support for Preview Features.

Topics:

Prerequisites

To configure the google-pubsub() source on syslog-ng Premium Edition (syslog-ng PE), you must have each of the following:

Google Cloud Platform side prerequisites
Google Pub/Sub Subscription prerequisites
Example: Google Pub/Sub

The following example shows a Google Pub/Sub with all necessary parameters configured.

Figure 26: A Google Pub/Sub subscription with all necessary parameters configured

Limitations

The current implementation of the google-pubsub() source has the following limitations:

  • The At-Least-Once delivery behavior takes effect after creating your subscription.

    NOTE: The At-Least-Once delivery behavior (which means that if an error occurs, it is more acceptable to duplicate messages than to lose any of them) only takes effect after you create your Google Pub/Sub subscription.

    The At-Least-Once delivery behavior is intentional, and its purpose is to avoid potential log loss.

    For more information, see Preventing message duplication resulting from the At-Least-Once delivery behavior.

  • The Google Pub/Sub service cannot guarantee message ordering.

    As a result of the At-Least-Once delivery behavior, messages may be delivered out of order, especially if an outstanding message is not acknowledged by the subscriber before the Acknowledgement deadline passes. In this case, the Google Pub/Sub service will attempt to redeliver the message. As a result, the Google Pub/Sub service cannot guarantee message ordering.

  • The syslog-ng PE application retains acknowledgements on the source side and either acknowledges an ack-tracker-batch-size() number of messages in a batch, or sends acknowledgements after the ack-tracker-timeout() expires. If the value of your ack-tracker-timeout() is larger than the value of your Acknowledgement deadline, it may result in message duplication.

    NOTE: In case you encounter message duplication, One Identity recommends that you check the following parameters in your configuration:

    • log-fetch-limit(): if it set to a high value, it is possible that syslog-ng PE can not process the fetched messages within the configured Acknowledgement deadline.

    • ack-tracker-timeout(): if it set to a higher value than the Acknowledgement deadline configured on the Google Cloud Platform, set it to a lower value.

    For more information about message duplication, see Preventing message duplication resulting from the At-Least-Once delivery behavior.

Supported platforms

The current implementation of the google-pubsub() source works on all supported syslog-ng PE 7 LTS platforms.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating