Chat now with support
Chat with Support

syslog-ng Open Source Edition 3.36 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mqtt: receiving messages from an MQTT broker network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local email messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP collectd: sending metrics to collectd discord: Sending alerts and notifications to Discord elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka (Java implementation) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) loggly: Using Loggly logmatic: Using Logmatic.io mongodb(): Storing messages in a MongoDB database mqtt() destination: sending messages from a local network to an MQTT broker network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (email) from logs snmp: Sending SNMP traps Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP Sumo Logic destinations: sumologic-http() and sumologic-syslog() syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages
Parsing syslog messages Parsing messages with comma-separated and similar values Parsing key=value pairs JSON parser XML parser Parsing dates and timestamps Python parser Parsing tags Apache access log parser Linux audit parser Cisco parser Parsing enterprise-wide message model (EWMM) messages iptables parser Netskope parser panos-parser(): parsing PAN-OS log messages Sudo parser Websense parser Fortigate parser Check Point Log Exporter parser Regular expression (regexp) parser db-parser: Process message content with a pattern database (patterndb)
Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License Glossary

Prerequisites to using the mqtt() destination

Using the current implementation of the mqtt() destination has the following prerequisites:

  • Installing the eclipse-paho-mqtt-c library.

    NOTE: The default package manager for some Linux operating systems contains the eclipse-paho-mqtt-c library, but depending on your OS, you may have to install the library manually. For more information about how you can download and install the eclipse-paho-mqtt-c library, see Eclipse Paho on the Eclipse Foundation website.

  • Having a broker entity in a functional MQTT system.

    NOTE: In your configuration, you will specify the broker entity of your MQTT system in the address() option of your mqtt() source.

Limitations to using the mqtt() destination

Using the mqtt() destination of syslog-ng OSE has the following limitations:

  • You can only use the mqtt() destination with syslog-ng OSE version 3.33 or higher.

  • You cannot use the mqtt() destination without installing the the eclipse-paho-mqtt-c library.

    For more information about how you can download and install the eclipse-paho-mqtt-c library, see Eclipse Paho on the Eclipse Foundation website.

  • The current implementation of the mqtt() destination supports versions 3.1 and 3.1.1 of the MQTT protocol.

Options of the mqtt() destination

The mqtt() destination has the following options.

Required options: address(), fallback-topic(), and topic().

address()
Type: string
Default: tcp://localhost:1883
Required: yes

Description: Specifies the hostname or IP address, and the port number of the MQTT broker to which syslog-ng OSE will send the log messages.

Syntax: <protocol type>://<host>:<port>

Supported protocol types: TCP, WS, SSL andWSS.

client-id()
Type: string
Default: syslog-ng-source-{topic option}
Required: no

Description: The client-id() is used to identify the client to the MQTT server, which stores session data for each client. The session data can contains information regarding which message has been sent, received. It is not possible to set the client-id() to an empty string. To always start a new session see the cleansession() option.

cleansession()
Type: yes | no
Default: no

Description: This option instruments the MQTT broker to clean the session data when connecting. The session data contains information about which message was processed.

fallback-topic()
Type: string
Default: N/A

Description: Required option when using templates in the topic() option.

If the resolved topic() template is not a valid topic, syslog-ng OSE will use the fallback-topic() option to send messages.

NOTE: If instead of strings, you use actual templates (that is, a macro like ${MESSAGE}, or a template function like $(format-json)) in the topic() option, configuring the fallback-topic() option is required.

TIP: Occasionally, the reason why syslog-ng OSE cannot post messages to the configured topic() is that the topic contains invalid characters that originate from templates.

http-proxy()
Type: URL
Default: N/A

Description: Specifies HTTP/HTTPS proxy for WebSocket connections.

keep-alive()
Type: positive integer number (in seconds)
Default: 60

Description: Specifies the number of seconds that syslog-ng OSE keeps the connection between the broker and clients open in case there is no message traffic. When keep-alive() number of seconds pass, the connection is terminated, and you have to reconnect.

On the MQTT side, the keep alive function provides a workaround method to access connections that are still open.

password()
Type: string
Default: N/A

Description: The password used to authenticate on the MQTT broker.

qos()
Type: number
Default: 0

Possible values:

0 - at most once (the fastest option)

1 - at least once (a much slower option than 0)

2 - exactly once (the slowest option)

Description: The Quality of Service (QoS) level in MQTT messaging is an agreement between sender and receiver on the guarantee of delivering a message.

template()
Type: string
Default: $ISODATE $HOST $MSGHDR$MSG

Description: Specifies the message template that syslog-ng OSE sends to the MQTT broker.

If you want to use macros in templates, see Macros of syslog-ng OSE.

tls()
Type: tls options
Default: n/a

Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.

The following options are relevant for the mqtt() tls() block: ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), ssl-version(), use-system-cert-store().

topic()
Type: string or template
Default: N/A

Description: Required option. Specifies the MQTT topic.

NOTE: The current implementation of the mqtt() destination does not support using the following characters for topic names:

  • $

  • +

  • #

username()
Type: string
Default: N/A

Description: The username used to authenticate on the MQTT broker.

Possible error messages you may encounter while using the mqtt() destination

While using the mqtt() destination, you may encounter issues and corresponding error messages originating from the MQTT system. The following table contains the error messages you may encounter, the possible reasons behind them, and potential workaround methods.

Complete error message Possible reason(s) Possible solution(s)
 
"ERROR, while init threaded dest. ..."

The syslog-ng OSE application will not start.

You can try the following methods:

  • Restart syslog-ng OSE.

  • Stop some of the programs running on your computer.

  • Restart your computer, and then restart syslog-ng OSE.

"mqtt: the topic() argument is required for mqtt destinations. ..."

The topic() option is not set in your configuration. The syslog-ng OSE application will not start.

Set the missing topic() option in your configuration, then restart . 

"The mqtt destination does not support the batching of messages, ..."

Your configuration may contain the batch-timeout() and / or batch-lines() options, which are not supported by the mqtt() destination. The syslog-ng OSE application will not start.

If your configuration contains the batch-timeout() and / or batch-lines() options, remove them from your configuration, and restart .

"Disconnected during publish!"

The syslog-ng OSE application can not send the message, because syslog-ng OSE disconnected from the broker. By default, syslog-ng OSE attempts to reconnect to the broker and send the messages 3 times.

If syslog-ng OSE fails all 3 attempts to reconnect to the broker and send the messages, you can try checking your configuration or restarting your MQTT system with syslog-ng OSE as a client.

"Max message inflight! (publish)"

The syslog-ng OSE application can not send the message due to the max message inflight broker response code (which signals that the broker has received too many messages, and it needs more time to process them). The syslog-ng OSE application will attempt to resend the message.

Wait until the broker can process the in-flight messages and syslog-ng OSE can attempt to resend the message.

"Failure during publishing!"

The syslog-ng OSE application can not send the message due to the failure broker response code. The syslog-ng OSE application will attempt to resend the message.

N/A

"Error during publish!"

The syslog-ng OSE application can not send the message, and drops it.

Possible reason: bad_utf8_string (topic), NULL parameter.

That is, the most probable reasons behind this issue are either that the topic name in your configuration is not correct, or that the message field is empty.

You can try the following methods:

  • Modify the name of the topic() option in your configuration.

  • Make sure that the message field is not empty.

"Disconnected while waiting the response!"

The syslog-ng OSE application has sent the message, but the client disconnected from the broker before syslog-ng OSE received the response. The syslog-ng OSE application will attempt to reconnect, or to resend the message.

The syslog-ng OSE application will attempt to reconnect to the broker and send the in-flight message. If the reconnect attempt fails, syslog-ng OSE will resend the message.

"Error while waiting the response!"

The syslog-ng OSE application can not get any response from the broker, due to the failure broker response code. The syslog-ng OSE will attempt to resend the message.

In this case, you will receive a further error message, depending on what the problem is. Wait for the second error message for more information about how you can proceed. 

"Error constructing topic ..."

Due to an issue with the configured topic template, the mqtt() destination will use the fallback-topic() option instead.

 N/A
"mqtt dest: topic name is illegal, it can't be empty"

This error message is related to the "Error constructing topic ..." error message.

In this case, the topic template returns a 0 length string. As a result, the mqtt() destination will use the fallback-topic() option instead.

 N/A 
"Error connecting mqtt client ..."

The syslog-ng OSE application can not connect to broker, and it will attempt to reconnect later.

If the issue persists, you can try the following:

  • Update your eclipse-paho-mqtt-c library.

  • Restart syslog-ng OSE.

"Error creat mqtt client ..."

The syslog-ng OSE application encountered an error while creating the MQTT client, and it will attempt to create it later.

Possible reasons:

  • There is a wrong address() set in your configuration.

  • The broker is not running.

You can try the following methods:

  • Check the address() option in your configuration, and modify if necessary.

  • Check if the specified broker is running by connecting to it manually, and then sending the broker a message.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating