A destination is where a log message is sent if the filtering rules match. Similarly to sources, destinations consist of one or more drivers, each defining where and how messages are sent.
|
TIP:
If no drivers are defined for a destination, all messages sent to the destination are discarded. This is equivalent to omitting the destination from the log statement. |
To define a destination, add a destination statement to the syslog-ng configuration file using the following syntax.
destination <identifier> { destination-driver(params); destination-driver(params); ... };
The following destination statement sends messages to the TCP port 1999 of the 10.1.2.3 host.
destination d_demo_tcp { network("10.1.2.3" port(1999)); };
If name resolution is configured, you can use the hostname of the target server as well.
destination d_tcp { network("target_host" port(1999)); };
|
Caution:
|
The following table lists the destination drivers available in syslog-ng PE.
Name | Description |
---|---|
elasticsearch and elasticsearch2 | Sends messages to an Elasticsearch server. The elasticsearch2 driver supports Elasticsearch version 2 and newer. |
file() | Writes messages to the specified file. |
graphite() | Sends metrics to a Graphite server to store numeric time-series data. |
hdfs() | Sends messages into a file on a Hadoop Distributed File System (HDFS) or MapR-FS node. |
http() | Sends messages over the HTTP protocol. |
kafka() | Publishes log messages to the Apache Kafka message bus, where subscribers can access them. |
logstore() | Writes messages securely into encrypted, compressed, and timestamped binary files. |
mongodb() | Sends messages to a MongoDB database. |
network() | Sends messages to a remote host using the BSD-syslog protocol over IPv4 and IPv6. Supports the TCP, UDP, ALTP, and TLS network protocols. |
pipe() | Writes messages to the specified named pipe. |
program() | Forks and launches the specified program, and sends messages to its standard input. |
python() | Send messages to a custom destination written in Python. |
smtp() | Sends e-mail messages to the specified recipients. |
splunk-hec() | Forward your log messages to Splunk. |
sql() | Sends messages into an SQL database. In addition to the standard syslog-ng packages, the sql() destination requires database-specific packages to be installed. Refer to the section appropriate for your platform in Installing syslog-ng. |
syslog() | Sends messages to the specified remote host using the IETF-syslog protocol. The IETF standard supports message transport using the UDP, TCP, and TLS networking protocols. |
syslog-ng() | The syslog-ng() destination driver forwards log messages to another syslog-ng node in EWMM format. |
unix-dgram() | Sends messages to the specified unix socket in SOCK_DGRAM style (BSD). |
unix-stream() | Sends messages to the specified unix socket in SOCK_STREAM style (Linux). |
usertty() | Sends messages to the terminal of the specified user, if the user is logged in. |
Starting with version
|
NOTE:
In order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ng Premium Edition server uses this destination. For details on the server mode, see Server mode. |
Note the following limitations when using the syslog-ng PE elasticsearch destination:
This destination is only supported on the Linux platforms that use the linux glibc2.11 installer, including: Debian 7 (wheezy), Red Hat ES 7, Ubuntu 12.04 (Precise Pangolin), Ubuntu 14.04 (Trusty Tahr).
Since syslog-ng PE uses the official Java Elasticsearch libraries, the elasticsearch destination has significant memory usage.
The log messages of the underlying client libraries are available in the internal() source of syslog-ng PE.
@module mod-java @include "scl.conf" elasticsearch( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("syslog-ng") );
The following example defines an elasticsearch destination that sends messages in transport mode to an Elasticsearch server version 1.x running on the localhost, using only the required parameters.
@module mod-java @include "scl.conf" destination d_elastic { elasticsearch( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") ); };
The following example sends 10000 messages in a batch, in transport mode, and includes a custom unique ID for each message.
@module mod-java @include "scl.conf" options { threaded(yes); use-uniqid(yes); }; source s_syslog { syslog(); }; destination d_elastic { elasticsearch( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("syslog-ng") client-mode("transport") custom-id("${UNIQID}") flush-limit("10000") ); }; log { source(s_syslog); destination(d_elastic); flags(flow-control); };
To install the software required for the elasticsearch destination, see Prerequisites.
For details on how the elasticsearch destination works, see How syslog-ng PE interacts with Elasticsearch.
For the list of options, see Elasticsearch destination options.
|
NOTE:
If you delete all Java destinations from your configuration and reload syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again. |
The following describes how to send messages from syslog-ng PE to Elasticsearch.
To send messages from syslog-ng PE to Elasticsearch
If you want to use the Java-based modules of syslog-ng PE (for example, the Elasticsearch, HDFS, or Kafka destinations), you must compile syslog-ng PE with Java support.
Download and install the Java Runtime Environment (JRE), 1.7 (or newer).
Install gradle version 2.2.1 or newer.
Set LD_LIBRARY_PATH to include the libjvm.so file, for example, LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH
Note that many platforms have a simplified links for Java libraries. Use the simplified path if available. If you use a startup script to start syslog-ng PE set LD_LIBRARY_PATH in the script as well.
If you are behind an HTTP proxy, create a gradle.properties under the modules/java-modules/ directory. Set the proxy parameters in the file. For details, see The Gradle User Guide.
Download the Elasticsearch libraries version 1.5 or newer from the 1.x line from https://www.elastic.co/downloads/elasticsearch.
Extract the Elasticsearch libraries into a temporary directory, then collect the various .jar files into a single directory (for example, /opt/elasticsearch/lib/) where syslog-ng PE can access them. You must specify this directory in the syslog-ng PE configuration file. The files are located in the lib directory and its subdirectories of the Elasticsearch release package.
The syslog-ng PE application sends the log messages to the official Elasticsearch client library, which forwards the data to the Elasticsearch nodes. The way how syslog-ng PE interacts with Elasticsearch is described in the following steps.
After syslog-ng PE is started and the first message arrives to the elasticsearch destination, the elasticsearch destination tries to connect to the Elasticsearch server or cluster. If the connection fails, syslog-ng PE will repeatedly attempt to connect again after the period set in time-reopen() expires.
If the connection is established, syslog-ng PE sends JSON-formatted messages to Elasticsearch.
If flush-limit is set to 1: syslog-ng PE sends the message reliably: it sends a message to Elasticsearch, then waits for a reply from Elasticsearch. In case of failure, syslog-ng PE repeats sending the message, as set in the retries() parameter. If sending the message fails for retries() times, syslog-ng PE drops the message.
This method ensures reliable message transfer, but is slow (about 1000 messages/second).
If flush-limit is higher than 1: syslog-ng PE sends messages in a batch, and receives the response asynchronously. In case of a problem, syslog-ng PE cannot resend the messages.
This method is relatively fast (depending on the size of flush-limit, about 8000 messages/second), but the transfer is not reliable. In transport mode, over 5000-30000 messages can be lost before syslog-ng PE recognizes the error. In node mode, about 1000 messages can be lost.
If concurrent-requests is higher than 1, syslog-ng PE can send multiple batches simultaneously, increasing performance (and also the number of messages that can be lost in case of an error). For details, see concurrent-requests().
© 2021 One Identity LLC. ALL RIGHTS RESERVED. Feedback Terms of Use Privacy