As of syslog-ng PE version 7.0.26, the kafka-c() destination can directly publish log messages to the Apache Kafka message bus using the librdkafka client. The new, C-based implementation has several advantages in comparison with the Java-based implementation, such as scalability, more efficient memory usage, and simpler setup.
The following figure illustrates how the kafka-c() destination works with syslog-ng PE.
Figure 29: How the kafka-c() destination works with syslog-ng PE
This section describes the prerequisites and restrictions for using the kafka-c() destination, and important information about the declaring the destination.
Prerequisites and restrictions
Declaration
kafka-c(
bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
topic("topic-name")
);
Example: Sending log data to Apache Kafka
The following example defines a kafka-c() destination in the new C implementation, using only the required parameters.
@include "scl.conf"
destination d_kafka {
kafka-c(
bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
topic("topic-name")
);
};
If you were using the Java-based kafka() destination and want to use the C-based kafka-c() destination, the following changes to the configuration file and considerations are necessary.
-
The client_lib_dir() option has been deprecated. Remove it from the configuration file.
-
The kafka-bootstrap-servers() option has been renamed as bootstrap-servers().
-
The option() option has been removed and replaced by the config() option, which has a different syntax.
-
Instead of the properties-file() option, you can use the config() option (using a config(key => value) format) to fine-tune your configuration.
NOTE: If you used the properties-file() option before, you can import the configuration parameters you were using earlier, with minor modifications in syntax into the config() option.
Syntactical differences between the properties-file() option and the config() option
The following examples illustrate the syntactical differences of using configuration parameters in the properties-file() option and the config() option:
-
The template() option has been renamed as message().
-
If you use templates with the topic() option, configuring the fallback-topic() option is also required.
For more information about these options, see Options of the kafka-c() destination.
A syslog-ng PE (syslog-ng PE) destination recognizes a message as sent when the message has been sent to the Kafka client, not when the Kafka server confirms its delivery.
If the Kafka client collects too many unsent messages, it will not accept any more messages from syslog-ng PE. The syslog-ng PE application detects this and stops sending messages to the Kafka client. Also, syslog-ng PE's flow control starts functioning in the direction of the sources (for example, syslog-ng PE will not read from the sources in that specific logpath).
|
Caution:
Hazard of data loss!
If sync-send() is set to "no", the messages passed to the Kafka client can be lost.
To avoid data loss, One Identity recommends that you set sync-send() to "yes", as this setting delivers messages to the Kafka client more reliably. |
For more information, see the description of the sync-send() option.