syslog-ng Open Source Edition 3.25 - Administration Guide

Flow control in syslog-ng OSE and the Kafka client

A syslog-ng destination recognizes a message as sent when the message has been sent to the Kafka client, not when the Kafka server confirms its delivery.

If the Kafka client collects too many unsent messages, it will not accept any more messages from syslog-ng. The syslog-ng application detects this and stops sending messages to the Kafka client. Also, syslog-ng's flow control starts functioning in the direction of the sources (for example, syslog-ng will not read from the sources in that specific logpath).

You can specify a "high water mark" limit for the Kafka client in the properties-file().

For more information about how the C implementation of the kafka destination works with syslog-ng OSE, click here.

Options of the kafka destination's C implementation

The C implementation of the kafka destination of syslog-ng OSE can directly publish log messages to the Apache Kafka message bus, where subscribers can access them. The C implementation of the kafka destination has the following options.

@define kafka-implementation kafka-c

config( 
 “key1” => “value1” 
 “key2” => “value2” 
)

destination d_demo {
    network(
        "127.0.0.1"
        port(3333)
        disk-buffer(
            mem-buf-size(10000)
            disk-buf-size(2000000)
            reliable(yes)
            dir("/tmp/disk-buffer")
        )
    );
};

destination d_demo {
    network(
        "127.0.0.1"
        port(3333)
           disk-buffer(
            mem-buf-length(10000)
            disk-buf-size(2000000)
            reliable(no)
            dir("/tmp/disk-buffer")
        )
    );
};

frac-digits()

Description: The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using the hook-commands() when syslog-ng OSE starts or stops

To execute an external program when syslog-ng OSE starts or stops, use the following options:

Using the hook-commands() when syslog-ng OSE reloads

To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:

source {
   network(transport(udp)
	hook-commands(
          startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
          shutdown("iptables -D LOGCHAIN 1")
        )
     );
};

log-fifo-size()

Description: The number of messages that the output queue can store.

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

acks=all
compression.type=snappy.

throttle()

Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

time-zone()

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

ts-format()

Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().

loggly: Using Loggly

The loggly() destination sends log messages to the Loggly Logging-as-a-Service provider. You can send log messages over TCP, or encrypted with TLS.

Declaration:

loggly(token());

log {
    source { system(); };
    destination { loggly(token("<USER-TOKEN-AS-PROVIDED-BY-LOGGLY>")); };
};

log {
    destination {
        loggly(token("<USER-TOKEN-AS-PROVIDED-BY-LOGGLY>") port(6514)
            tls(peer-verify(required-trusted) ca-dir('/etc/ssl/certs'))
        );
    };
};

log {
    source { file("/var/log/apache2/access.log" flags(no-parse)); };
    parser { apache-accesslog-parser(); };
    destination {
        loggly(token("<USER-TOKEN-AS-PROVIDED-BY-LOGGLY>")
           tag(apache)
           template("$(format-json .apache.* timestamp=${ISODATE})"));
    };
}

To use the loggly() driver, the scl.conf file must be included in your syslog-ng OSE configuration:

@include "scl.conf"

The loggly() driver is actually a reusable configuration snippet configured to send log messages using the tcp() driver using a template. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

loggly() destination options

The loggly() destination has the following options. You can also set other options of the underlying tcp() driver (for example, port number or TLS-encryption).

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using the hook-commands() when syslog-ng OSE starts or stops

To execute an external program when syslog-ng OSE starts or stops, use the following options:

Using the hook-commands() when syslog-ng OSE reloads

To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:

source {
   network(transport(udp)
	hook-commands(
          startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
          shutdown("iptables -D LOGCHAIN 1")
        )
     );
};

token()

Description: Your Customer Token that you received from Loggly.