kafka: Publishing messages to Apache Kafka

kafka: Publishing messages to Apache Kafka

Starting with version 3.7, syslog-ng OSE can directly publish log messages to the Apache Kafka message bus, where subscribers can access them.

• This destination is only supported on the Linux platform.

• Since syslog-ng OSE uses the official Java Kafka producer, the kafka destination has significant memory usage.

• The log messages of the underlying client libraries are available in the internal() source of syslog-ng OSE.

@module mod-java
@include "scl.conf"

kafka(
    client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-kafka-libraries>")
    kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
    topic("${HOST}")

);

@module mod-java
@include "scl.conf"

destination d_kafka {
  kafka(
    client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/")
    kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
    topic("${HOST}")
  );
};

• To install the software required for the kafka destination, see Prerequisites.

• For details on how the kafka destination works, see How syslog-ng OSE interacts with Apache Kafka.

• For the list of options, see Kafka destination options.

The kafka() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of syslog-ng OSE. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the kafka configuration snippet on GitHub. For details on extending syslog-ng OSE in Java, see the Getting started with syslog-ng development guide.