Chat now with support
Chat with Support

syslog-ng Open Source Edition 3.22 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local e-mail messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP collectd: sending metrics to collectd elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka (Java implementation) kafka: Publishing messages to Apache Kafka (C implementation, using the librdkafka client) loggly: Using Loggly logmatic: Using Logmatic.io mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (e-mail) from logs snmp: Sending SNMP traps Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng: Forwarding messages and tags to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages db-parser: Process message content with a pattern database (patterndb) Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License

Prerequisites

To send messages from syslog-ng OSE to Elasticsearch, complete the following steps.

Steps:

  1. Download and install the Java Runtime Environment (JRE), 2.x (or newer). The syslog-ng OSE elasticsearch2 destination is tested and supported when using the Oracle implementation of Java. Other implementations are untested and unsupported, they may or may not work as expected.

  		2.  NOTE:

    This step is only required if you use the elasticsearch2 destination in node mode or transport mode.

    Download the Elasticsearch libraries (version 2.x or newer from the 2.x line) from https://www.elastic.co/downloads/elasticsearch.

  		3.  NOTE:

    This step is only required if you use the elasticsearch2 destination in node mode or transport mode.

    Extract the Elasticsearch libraries into a temporary directory, then collect the various .jar files into a single directory (for example, /opt/elasticsearch/lib/) where syslog-ng OSE can access them. You must specify this directory in the syslog-ng OSE configuration file. The files are located in the lib directory and its subdirectories of the Elasticsearch release package.

How syslog-ng OSE interacts with Elasticsearch

The syslog-ng OSE application sends the log messages to the official Elasticsearch client library, which forwards the data to the Elasticsearch nodes. The way syslog-ng OSE interacts with Elasticsearch is described in the following steps.

  • After syslog-ng OSE is started and the first message arrives to the elasticsearch2 destination, the elasticsearch2 destination tries to connect to the Elasticsearch server or cluster. If the connection fails, syslog-ng OSE will repeatedly attempt to connect again after the period set in time-reopen() expires.

  • If the connection is established, syslog-ng OSE sends JSON-formatted messages to Elasticsearch.

    • If flush-limit is set to 1: syslog-ng OSE sends the message reliably: it sends a message to Elasticsearch, then waits for a reply from Elasticsearch. In case of failure, syslog-ng OSE repeats sending the message, as set in the retries() parameter. If sending the message fails for retries() times, syslog-ng OSE drops the message.

      This method ensures reliable message transfer, but is slow (about 1000 messages/second).

    • If flush-limit is higher than 1: syslog-ng OSE sends messages in a batch, and receives the response asynchronously. In case of a problem, syslog-ng OSE cannot resend the messages.

      This method is relatively fast (depending on the size of flush-limit, about 8000 messages/second), but the transfer is not reliable. In transport mode, over 5000-30000 messages can be lost before syslog-ng OSE recognizes the error. In node mode, about 1000 messages can be lost.

    • If concurrent-requests is higher than 1, syslog-ng OSE can send multiple batches simultaneously, increasing performance (and also the number of messages that can be lost in case of an error).

  • Version 3.10 and newer of syslog-ng OSE automatically converts the timestamp (date) of the message to UTC, as needed by Elasticsearch and Kibana.

Client modes

The syslog-ng OSE application can interact with Elasticsearch in the following modes of operation: http, https, node, searchguard, and transport.

  • HTTP mode

    The syslog-ng OSE application sends messages over HTTP using the REST API of Elasticsearch, and uses the cluster-url() and cluster() options from the syslog-ng OSE configuration file. In HTTP mode, syslog-ng OSE elasticsearch2 driver can send log messages to every Elasticsearch version, including 1.x-6.x. Note that HTTP mode is available in syslog-ng OSE version 3.8 and newer.

    In version 3.10 and newer, you can list multiple servers in HTTP and HTTPS mode in the cluster-url() and server() options. The syslog-ng OSE application will use these destination servers in load-balancing fashion. Note that load-balancing is handled by an external library (Jest), syslog-ng OSE does not have any direct influence on it.

  • HTTPS mode

    The syslog-ng OSE application sends messages over an encrypted and optionally authenticated HTTPS channel using the REST API of Elasticsearch, and uses the cluster-url() and cluster() options from the syslog-ng OSE configuration file. In HTTPS mode, syslog-ng OSE elasticsearch2 driver can send log messages to every Elasticsearch version, including 1.x-6.x. Note that HTTPS mode is available in syslog-ng OSE version 3.10 and newer.

    This mode supports password-based and certificate-based authentication of the client, and can verify the certificate of the server as well.

    In version 3.10 and newer, you can list multiple servers in HTTP and HTTPS mode in the cluster-url() and server() options. The syslog-ng OSE application will use these destination servers in load-balancing fashion. Note that load-balancing is handled by an external library (Jest), syslog-ng OSE does not have any direct influence on it.

  • Transport mode

    The syslog-ng OSE application uses the transport client API of Elasticsearch, and uses the server(), port(), and cluster() options from the syslog-ng OSE configuration file.

  • Node mode

    The syslog-ng OSE application acts as an Elasticsearch node (client no-data), using the node client API of Elasticsearch. Further options for the node can be describe in an Elasticsearch configuration file specified in the resource() option.

    		 NOTE:

    In Node mode, it is required to define the home of the elasticsearch installation with the path.home parameter in the .yml file. For example: path.home: /usr/share/elasticsearch.

  • Search Guard mode

    Use the Search Guard Elasticsearch plugin to encrypt and authenticate your connections from syslog-ng OSE to Elasticsearch 2.x. For Elasticsearch versions 5.x and newer, use HTTPS mode. For details on configuring Search Guard mode, see Search Guard and syslog-ng OSE.

Search Guard and syslog-ng OSE

Purpose:

Version 3.9 and later supports the Search Guard Elasticsearch plugin (version 2.4.1.16 and newer) to encrypt and authenticate your connections to from syslog-ng OSE to Elasticsearch 2 and newer. To configure syslog-ng OSE to send messages to an Elasticsearch 2.x cluster that uses Search Guard, complete the following steps.

To connect to an Elasticsearch 5.x or newer cluster, use HTTPS mode.

Steps:

  1. Install the Search Guard plugin on your syslog-ng OSE host. Use the plugin version that matches the version of your Elasticsearch installation.

    sudo /usr/share/elasticsearch/bin/plugin install -b com.floragunn/search-guard-ssl/<version-number-of-the-plugin>

  2. Create a certificate for your syslog-ng OSE host, and add the certificate to the SYSLOG_NG-NODE_NAME-keystore.jks file. You can configure the location of this file in the Elasticsearch resources file under the path.conf parameter. For details, see the Search Guard documentation.

  3. Configure an Elasticsearch destination in syslog-ng OSE that uses the searchguard client mode. For example:

    destination d_elasticsearch {
  elasticsearch2(
    client-lib-dir("/usr/share/elasticsearch/plugins/search-guard-ssl/*.jar:/usr/share/elasticsearch/lib")
    index("syslog-${YEAR}.${MONTH}.${DAY}")
    type("syslog")
    time-zone("UTC")
    client-mode("searchguard")
    resource("/etc/syslog-ng/elasticsearch.yml")
  );
};

  4. Configure the Elasticsearch resource file (for example, /etc/syslog-ng/elasticsearch.yml) as needed for your environment. Note the searchguard: section.

    cluster:
  name: elasticsearch
discovery:
  zen:
    ping:
      unicast:
        hosts:
          - <ip-address-of-the-elasticsearch-server>
node:
  name: syslog_ng_secure
  data; false
  master: false
path:
  home: /etc/syslog-ng
  conf: /etc/syslog-ng
searchguard:
  ssl:
    transport:
      keystore_filepath: syslog_ng-keystore.jks
      keystore_password: changeit
      truststore_filepath: truststore.jks
      truststore_password: changeit
      enforce_hostname_verification: true
Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating