syslog-ng Open Source Edition 3.25 - Administration Guide

Example: Using log path flags

Let's suppose that you have two hosts (myhost_A and myhost_B) that run two applications each (application_A and application_B), and you collect the log messages to a central syslog-ng server. On the server, you create two log paths:

• one that processes only the messages sent by myhost_A, and

• one that processes only the messages sent by application_A.

This means that messages sent by application_A running on myhost_A will be processed by both log paths, and the messages of application_B running on myhost_B will not be processed at all.

• If you add the final flag to the first log path, then only this log path will process the messages of myhost_A, so the second log path will receive only the messages of application_A running on myhost_B.

• If you create a third log path that includes the fallback flag, it will process the messages not processed by the first two log paths, in this case, the messages of application_B running on myhost_B.

• Adding a fourth log path with the catchall flag would process every message received by the syslog-ng server.

  log { source(s_localhost); destination(d_file); flags(catchall); };

The following example shows a scenario that can result in message loss. Do NOT use such a configuration, unless you know exactly what you are doing. The problem is if a message matches the filters in the first part of the first log path, syslog-ng OSE treats the message as 'processed'. Since the first log path includes the final flag, syslog-ng OSE will not pass the message to the second log path (the one with the fallback flag). As a result, syslog-ng OSE drops messages that do not match the filter of the embedded log path.

# Do not use such a configuration, unless you know exactly what you are doing.
log {
    source(s_network);
    # Filters in the external log path.
    # If a message matches this filter, it is treated as 'processed'
    filter(f_program);
    filter(f_message);
    log {
        # Filter in the embedded log path.
        # If a message does not match this filter, it is lost, it will not be processed by the 'fallback' log path
        filter(f_host);
        destination(d_file1);
    };
    flags(final);
};

log {
    source(s_network);
    destination(d_file2);
    flags(fallback);
};

Example: Using the drop-unmatched flag

In the following example, if a log message arrives whose $MSG part does not contain the string foo, then syslog-ng OSE will discard the message and will not check compliance with the second if condition.

...
if {
    filter { message('foo') };
    flags(drop-unmatched)
};
if {
    filter { message('bar') };
};
...

(Without the drop-unmatched flag, syslog-ng OSE would check if the message complies with the second if condition, that is, whether or not the message contains the string bar .)

Example: Sizing parameters for flow-control

Suppose that syslog-ng has a source that must accept up to 300 parallel connections. Such situation can arise when a network source receives connections from many clients, or if many applications log to the same socket.

Set the max-connections() parameter of the source to 300. However, the log-fetch-limit() (default value: 10) parameter applies to every connection of the source individually, while the log-iw-size() (default value: 1000) parameter applies to the source. In a worst-case scenario, the destination does not accept any messages, while all 300 connections send at least log-fetch-limit() number of messages to the source during every poll loop. Therefore, the control window must accommodate at least max-connections()*log-fetch-limit() messages to be able to read every incoming message of a poll loop. In the current example this means that log-iw-size() should be greater than 300*10=3000. If the control window is smaller than this value, the control window might fill up with messages from the first connections — causing syslog-ng to read only one message of the last connections in every poll loop.

The output buffer of the destination must accommodate at least log-iw-size() messages, but use a greater value: in the current example 3000*10=30000 messages. That way all incoming messages of ten poll loops fit in the output buffer. If the output buffer is full, syslog-ng does not read any messages from the source until some messages are successfully sent to the destination.

source s_localhost {
    network(
        ip(127.0.0.1)
        port(1999)
        max-connections(300)
    );
};
destination d_tcp {
    network("10.1.2.3"
        port(1999)
        localport(999)
        log-fifo-size(30000)
    );
};
log {
    source(s_localhost);
    destination(d_tcp);
    flags(flow-control);
};

If other sources send messages to this destination, then the output buffer must be further increased. For example, if a network host with maximum 100 connections also logs into the destination, then increase the log-fifo-size() by 10000.

source s_localhost {
    network(
        ip(127.0.0.1)
        port(1999)
        max-connections(300)
    );
};
source s_tcp {
    network(
        ip(192.168.1.5)
        port(1999)
        max-connections(100)
    );
};
destination d_tcp {
    network("10.1.2.3"
        port(1999)
        localport(999)
        log-fifo-size(40000)
    );
};
log {
    source(s_localhost);
    destination(d_tcp);
    flags(flow-control);
};