Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.29 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng PE The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files google-pubsub: collecting messages from the Google Pub/Sub messaging service wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mssql, oracle, sql: collecting messages from an SQL database network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP without Java kafka(): Publishing messages to Apache Kafka (Java implementation) (DEPRECATED) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel(): Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql(): Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

Supported platforms

The current implementation of the google-pubsub() source works on all supported syslog-ng PE 7 LTS platforms.

Declaration

You can use the following example as a configuration block template for declaring the google-pubsub() source in your configuration:

google-pubsub(project("project") 
  subscription("sub") 
  credentials("creds.json")
);

The Google Pub/Sub message format in syslog-ng PE

From version 7.0.22, syslog-ng Premium Edition (syslog-ng PE) can collect messages from Google Pub/Sub.

NOTE: The rest of this section assumes that you are familiar with the Google Pub/Sub messaging service, and its concepts and terminology.

Messages on the Google Cloud Platform side

The syslog-ng PE application's google-pubsub() source collects Google Pub/Sub messages in a format that has two message parts (Message body and Message attributes) on the Google Cloud Platform side.

NOTE: Google Pub/Sub messages must contain at least one of the Message body and Message attributes message parts.

For more information about Google Pub/Sub message format on the Google Cloud Platform side, see Message format.

For more information about publishing Google Pub/Sub messages on the Google Cloud Platform side, see Publishing messages.

Processing the Message body contents on the syslog-ng PE side

After collecting the contents of the Google Pub/Sub message's Message body field as raw, unformatted data, the google-pubsub() source stores the message contents in the $MESSAGE syslog-ng PE macro. Next, syslog-ng PE prepends a header to the message contents, and the resulting data will form the syslog-ng PE output.

Example: incoming Google Pub/Sub Cloud message contents, resulting $MESSAGE macro contents, and processed output message contents with a prepended Message header

Incoming message contents on the Google Pub/Sub Cloud side: 

<38>Feb 25 14:09:07 testhost testapp: test message - 1

The contents of the relevant $MESSAGE macro:

* name='MESSAGE', value='<38>Feb 25 14:09:07 testhost testapp: test message - 1'

By default, the syslog-ng PE application prepends a Message header to the $MESSAGE macro contents to form an output with a similar structure:

<13>Sep 29 14:59:28 ubuntu-xenail-amd64 <38>Feb 25 14:09:07 testhost testapp: test message - 1
Topics:

The contents of the Google Pub/Sub Message body on the syslog-ng Premium Edition (syslog-ng PE) side

The syslog-ng PE application's google-pubsub() source collects Google Pub/Sub messages in a format that has two message parts (Message body and Message attributes) on the Google Cloud Platform side.

NOTE: Google Pub/Sub messages must contain at least one of the Message body and Message attributes message parts.

After collecting the contents of the Google Pub/Sub message's Message body field as raw, unformatted data, the google-pubsub() source stores the message contents in the $MESSAGE syslog-ng PE macro. Next, syslog-ng PE prepends a header to the message contents, and the resulting data will form the syslog-ng PE output.

If the Google Pub/Sub message also contains data from Message attributes, these attributes are displayed in the output as attribute names following the prepended prefix.

Example: Message body contents in the $MESSAGE syslog-ng PE macro

The following example shows illustrates what Message body contents on the Google Pub/Sub Cloud side will look like in the $MESSAGE syslog-ng PE macro:

With the incoming Message body contents on the Google Pub/Sub Cloud side:

<38>Feb 25 14:09:07 testhost testapp: test message - 1

The contents of the relevant $MESSAGE macro will look similar to this:

* name='MESSAGE', value='<38>Feb 25 14:09:07 testhost testapp: test message - 1'
Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating