|
Caution:
This destination is deprecated and will be removed from a future version of syslog-ng PE. We recommend using the elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector destination instead. |
Starting with version
|
NOTE:
In order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ng Premium Edition server uses this destination. For details on the server mode, see Server mode. |
Note the following limitations when using the syslog-ng PE elasticsearch2 destination:
This destination is only supported on the Linux platforms that use the linux glibc2.11 installer, including: Red Hat ES 7, Ubuntu 14.04 (Trusty Tahr).
Since syslog-ng PE uses Java libraries, the elasticsearch2 destination has significant memory usage.
The log messages of the underlying client libraries are available in the internal() source of syslog-ng PE.
@module mod-java @include "scl.conf" elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("syslog-ng") );
The following example defines an elasticsearch2 destination that sends messages in transport mode to an Elasticsearch server running on the localhost, using only the required parameters.
@module mod-java @include "scl.conf" destination d_elastic { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") ); };
The following example sends 10000 messages in a batch, in transport mode, and includes a custom unique ID for each message.
@module mod-java @include "scl.conf" options { threaded(yes); use-uniqid(yes); }; source s_syslog { syslog(); }; destination d_elastic { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("syslog-ng") client-mode("transport") custom-id("${UNIQID}") flush-limit("10000") ); }; log { source(s_syslog); destination(d_elastic); flags(flow-control); };
The following example send messages to Elasticsearch over HTTP using its REST API:
@include "scl.conf" source s_network { network(port(5555)); }; destination d_elastic { elasticsearch2( client-mode("http") cluster("es-syslog-ng") index("x201") cluster-url("http://192.168.33.10:9200") type("slng_test_type") flush-limit("0") ); }; log { source(s_network); destination(d_elastic); flags(flow-control); };
To install the software required for the elasticsearch2 destination, see Prerequisites.
For details on how the elasticsearch2 destination works, see How syslog-ng PE interacts with Elasticsearch.
For the list of options, see Elasticsearch2 destination options (DEPRECATED).
|
NOTE:
If you delete all Java destinations from your configuration and reload syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again. |
The following describes how to send messages from syslog-ng PE to Elasticsearch.
To send messages from syslog-ng PE to Elasticsearch
Download and install the Java Runtime Environment (JRE), 2.x (or newer). The syslog-ng PE elasticsearch2 destination is tested and supported when using the Oracle implementation of Java. Other implementations are untested and unsupported, they may or may not work as expected.
|
NOTE:
This step is only required if you use the elasticsearch2 destination in node mode or transport mode. |
Download the Elasticsearch libraries (version 2.x or newer from the 2.x line) from https://www.elastic.co/downloads/elasticsearch.
|
NOTE:
This step is only required if you use the elasticsearch2 destination in node mode or transport mode. |
Extract the Elasticsearch libraries into a temporary directory, then collect the various .jar files into a single directory (for example, /opt/elasticsearch/lib/) where syslog-ng PE can access them. You must specify this directory in the syslog-ng PE configuration file. The files are located in the lib directory and its subdirectories of the Elasticsearch release package.
The syslog-ng PE application sends the log messages to the official Elasticsearch client library, which forwards the data to the Elasticsearch nodes. The way how syslog-ng PE interacts with Elasticsearch is described in the following steps.
After syslog-ng PE is started and the first message arrives to the elasticsearch2 destination, the elasticsearch2 destination tries to connect to the Elasticsearch server or cluster. If the connection fails, syslog-ng PE will repeatedly attempt to connect again after the period set in time-reopen() expires.
If the connection is established, syslog-ng PE sends JSON-formatted messages to Elasticsearch.
If flush-limit is set to 1: syslog-ng PE sends the message reliably: it sends a message to Elasticsearch, then waits for a reply from Elasticsearch. In case of failure, syslog-ng PE repeats sending the message, as set in the retries() parameter. If sending the message fails for retries() times, syslog-ng PE drops the message.
This method ensures reliable message transfer, but is slow (about 1000 messages/second).
If flush-limit is higher than 1: syslog-ng PE sends messages in a batch, and receives the response asynchronously. In case of a problem, syslog-ng PE cannot resend the messages.
This method is relatively fast (depending on the size of flush-limit, about 8000 messages/second), but the transfer is not reliable. In transport mode, over 5000-30000 messages can be lost before syslog-ng PE recognizes the error. In node mode, about 1000 messages can be lost.
If concurrent-requests is higher than 1, syslog-ng PE can send multiple batches simultaneously, increasing performance (and also the number of messages that can be lost in case of an error). For details, see concurrent-requests().
Version
The syslog-ng PE application can interact with Elasticsearch in the following modes of operation: node, and transport.
The syslog-ng PE application uses the transport client API of Elasticsearch, and uses the server(), port(), and cluster() options from the syslog-ng PE configuration file.
The syslog-ng PE application acts as an Elasticsearch node (client no-data), using the node client API of Elasticsearch. Further options for the node can be describe in an Elasticsearch configuration file specified in the resource() option.
|
NOTE:
In Node mode, it is required to define the home of the elasticsearch installation with the path.home parameter in the .yml file. For example: path.home: /usr/share/elasticsearch. |
© 2021 One Identity LLC. ALL RIGHTS RESERVED. Feedback Terms of Use Privacy