syslog-ng Premium Edition 7.0.29 - Administration Guide

Starting with version 5.43.7, syslog-ng PE can directly publish log messages to the Apache Kafka message bus, where subscribers can access them.

• This destination is only supported on the Linux platform.

  This destination is only supported on the Linux platforms that use the linux glibc2.11 installer, including: Red Hat ES 7, Ubuntu 14.04 (Trusty Tahr).

• Since syslog-ng PE uses the official Java Kafka producer, the kafka() destination has significant memory usage.

• The log messages of the underlying client libraries are available in the internal() source of syslog-ng PE.

@module mod-java
@include "scl.conf"

kafka(
    client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-kafka-libraries>")
    kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
    topic("${HOST}")
);

@module mod-java
@include "scl.conf"

destination d_kafka {
    kafka(
        client-lib-dir(/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/)
        kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
        topic("${HOST}")
    );
};

• To install the software required for the kafka() destination, see Prerequisites.

• For details on how the kafka() destination works, see How syslog-ng PE interacts with Apache Kafka.

• For the list of options, see Kafka destination options.

The kafka() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of syslog-ng PE. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the kafka configuration snippet on GitHub. For details on extending syslog-ng PE in Java, see the Getting started with syslog-ng PE development guide.