syslog-ng Open Source Edition 3.22 - Administration Guide

This section describes the prerequisites and restrictions for using the kafka destination in the new C implementation, and important information about the declaration of the destination.

Prerequisites and restrictions

• This destination is only supported on the Linux platform.

• Since the new C implementation uses the librdkafka client library, the kafka destination has less memory usage than the previous Java implementation (which uses the official Java Kafka producer).

• The log messages of the underlying client libraries are available in the internal() source of syslog-ng OSE.

• If you used the Java implementation before, see Shifting from Java implementation to C implementation .

• The syslog-ng OSE kafka destination supports all properties of the official Kafka producer. For details, see the librdkafka documentation.

• For the list of options, see Options of the kafka destination's C implementation .

Declaration:

@define kafka-implementation kafka-c

kafka(
    bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
    topic("{MYTOPIC}")

);

Required options:

The following options are required: bootstrap-servers(), topic(). Note that to use the C implementation of the kafka destination, you must add the following lines to the beginning of your syslog-ng OSE configuration:

@define kafka-implementation kafka-c

config( 
 “key1” => “value1” 
 “key2” => “value2” 
)

disk-buffer()

Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:

Options reliable() and disk-buf-size() are required options.

destination d_demo {
    network(
        "127.0.0.1"
        port(3333)
        disk-buffer(
            mem-buf-size(10000)
            disk-buf-size(2000000)
            reliable(yes)
            dir("/tmp/disk-buffer")
        )
    );
};

destination d_demo {
    network(
        "127.0.0.1"
        port(3333)
           disk-buffer(
            mem-buf-length(10000)
            disk-buf-size(2000000)
            reliable(no)
            dir("/tmp/disk-buffer")
        )
    );
};

flush-timeout-on-reload()

Description: When syslog-ng reloads, the Kafka client will also reload. The flush-timeout-on-reload() option specifies the number of milliseconds syslog-ng waits for the Kafka client to send the unsent messages. The unsent messages will be retained in syslog-ng's own queue and syslog-ng will continue sending them after reload. This works without disk-buffering, too.

flush-timeout-on-shutdown()

Description: When syslog-ng shuts down, the Kafka client will also shut down. The flush-timeout-on-shutdown() option specifies the number of milliseconds syslog-ng waits for the Kafka client to send the unsent messages. Any messages not sent after the specified time will be lost. To avoid losing messages, we recommend you use the disk-buffer option.

key()

Description: The key of the partition under which the message is published. You can use templates to change the topic dynamically based on the source or the content of the message, for example, key("${PROGRAM}").

local-time-zone()

Description: Sets the timezone used when expanding filename and tablename templates.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

on-error()

Description: Controls what happens when type-casting fails and syslog-ng OSE cannot convert some data to the specified type. By default, syslog-ng OSE drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().

• drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of syslog-ng OSE.

• drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.

• fallback-to-string: Convert the property to string and log an error message to the internal() source.

• silently-drop-message: Drop the entire message silently, without logging the error.

• silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.

• silently-fallback-to-string: Convert the property to string silently, without logging the error.

persist-name()

Description:If you receive the following error message during syslog-ng OSE startup, set the persist-name() option of the duplicate drivers:

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

acks=all
compression.type=snappy

retries()

Description: The number of times syslog-ng OSE attempts to send a message to this destination. If syslog-ng OSE could not send a message, it will try again until the number of attempts reaches retries, then drops the message.

send-time-zone()

Description: Specifies the time zone associated with the messages sent by syslog-ng, if not specified otherwise in the message or in the destination driver. For details, see Timezones and daylight saving.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

template()

Description: The message as published to Apache Kafka. You can use templates and template functions (for example, format-json()) to format the message, for example, template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE)").

For details on formatting messages in JSON format, see format-json.

topic()

Description: The Kafka topic under which the message is published.

workers()

Description: The workers are only responsible for formatting the messages that need to be delivered to the Kafka clients. Configure this option only if your Kafka clients have many threads and they do not receive enough messages.