syslog-ng Open Source Edition 3.30 - Administration Guide

LogMessage() method: Create log message objects

You can use the LogMessage() method to create a structured log message instance. For example:

from syslogng import LogMessage

msg = LogMessage() # Initialize an empty message with default values (recvd timestamp, rcptid, hostid, ...)
msg = LogMessage("string or bytes-like object") # Initialize a message and set its ${MESSAGE} field to the specified argument

You can also explicitly set the different values of the log message. For example:

msg["MESSAGE"] = "message"
msg["HOST"] = "hostname"

You can set certain special field (timestamp, priority) by using specific methods.

Note the following points when creating a log message:

• When setting the hostname, syslog-ng OSE takes the following hostname-related options of the configuration into account: chain-hostnames(), keep-hostname(), use-dns(), and use-fqdn().

• Python sources ignore the log-msg-size() option.

• The syslog-ng OSE application accepts only one message from every LogSource::post_message() or fetch() call, batching is currently not supported. If your Python code accepts batches of messages, you must pass them to syslog-ng OSE one-by-one. Similarly, if you need to split messages in the source, you must do so in your Python code, and pass the messages separately.

• Do not reuse or store LogMessage objects after posting (calling post_message()) or returning the message from fetch().

parse() method: Parse syslog messages

The parse() method allows you to parse incoming messages as syslog messages. By default, the parse() method attempts to parse the message as an IETF-syslog (RFC5424) log message. If that fails, it parses the log message as a BSD-syslog (RFC3164) log message. Note that syslog-ng OSE takes the parsing-related options of the configuration into account: flags(), keep-hostname(), recv-time-zone().

If keep-hostname() is set to no, syslog-ng OSE ignores the hostname set in the message, and uses the IP address of the syslog-ng OSE host as the hostname (to use the hostname instead of the IP address, set the use-dns() or use-fqdn() options in the Python source).

msg_ietf = LogMessage.parse('<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] An application event log entry', self.parse_options)
msg_bsd = LogMessage.parse('<34>Oct 11 22:14:15 mymachine su: \'su root\' failed for lonvick on /dev/pts/8', self.parse_options)

set_pri() method

You can set the priority of the message with the set_pri() method.

msg.set_pri(165)

set_timestamp() method

You can use the set_timestamp() method to set the date and time of the log message.

timestamp = datetime.fromisoformat("2018-09-11T14:49:02.100+02:00")
msg.set_timestamp(timestamp) # datetime object, includes timezone information

In Python 2, timezone information cannot be attached to the datetime instance without using an external library. The syslog-ng OSE represents naive datetime objects in UTC.

class()

Description: The name of the Python class that implements the source, for example:

python(
    class("MyPythonSource")
);

If you want to store the Python code in an external Python file, the class() option must include the name of the Python file containing the class, without the path and the .py extension, for example:

python(
    class("MyPythonfilename.MyPythonSource")
);

For details, see Python code in external files

fetch-no-data-delay()

Description: If the fetch method of a python-fetcher() source returns with the LogFetcher.FETCH_NO_DATA constant, the source waits time-reopen() seconds before calling the fetch method again. If you want to call the fetch method sooner, set the fetch-no-data-delay() option to the number of seconds to wait before calling the fetch method.

flags()

Description: Specifies the log parsing options of the source.

• assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

• empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.

• expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

• guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

• kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

• no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

  source s_dell {
      network(
          port(2000)
          flags(no-hostname)
      );
  };

• no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

• no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

  If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, syslog-ng OSE generates a new syslog header (timestamp, host, and so on) automatically. Note that since flags(no-parse) disables message parsing, it interferes with other flags, for example, disables flags(no-multi-line).

• dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

• sanitize-utf8: When using the sanitize-utf8 flag, syslog-ng OSE converts non-UTF-8 input to an escaped form, which is valid UTF-8.

• store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another syslog-ng node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

• syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

• validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see IETF-syslog messages). If the BOM1 character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.

The flags and the hostname-related options (for example, use-dns) set in the configuration file influence the behavior of the LogMessage.parse() method of the Python source. They have no effect if you set the message or the hostname directly, without using LogMessage.parse().

loaders()

Description: The syslog-ng OSE application imports Python modules specified in this option, before importing the code of the Python class. This option has effect only when the Python class is provided in an external Python file. This option has no effect when the Python class is provided within the syslog-ng OSE configuration file (in a python{} block). You can use the loaders() option to modify the import mechanism that imports Python class. For example, that way you can use hy in your Python class.

python(class(usermodule.HyParser) loaders(hy))

options()

Description: This option allows you to pass custom values from the configuration file to the Python code. Enclose both the option names and their values in double-quotes. The Python code will receive these values during initialization as the options dictionary. For example, you can use this to set the IP address of the server from the configuration file, so it is not hard-coded in the Python object.

python(
    class("MyPythonClass")
    options(
        "host" "127.0.0.1"
        "port" "1883"
        "otheroption" "value")
);

For example, you can refer to the value of the host field in the Python code as options["host"]. Note that the Python code receives the values as strings, so you might have to cast them to the type required, for example: int(options["port"])

options(
  "host" => "localhost"
  "port" => "1883"
  "otheroption" => "value"
)			

persist-name()

Description:If you receive the following error message during syslog-ng OSE startup, set the persist-name() option of the duplicate drivers:

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

log {
    source { python(class(PyNetworkSource) options("port" "8080") persist-name("<unique-string>); };
    source { python(class(PyNetworkSource) options("port" "8081")); };
};

from syslogng import LogSource
  class PyNetworSource(LogSource):
    @staticmethod
    def generate_persist_name(options):
        return options["port"]
    def run(self):
        pass
    def request_exit(self):
        pass

time-zone()

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

Declaration:

Python sources consist of two parts. The first is a syslog-ng OSE source object that you define in your syslog-ng OSE configuration and use in the log path. This object references a Python class, which is the second part of the Python source. The Python class receives or fetches the log messages, and can do virtually anything that you can code in Python. You can either embed the Python class into your syslog-ng OSE configuration file, or store it in an external Python file.

source <name_of_the_python_source>{
    python-fetcher(
        class("<name_of_the_python_class_executed_by_the_source>")
    );
};

python {
from syslogng import LogFetcher
from syslogng import LogMessage

class <name_of_the_python_class_executed_by_the_source>(LogFetcher):
    def init(self, options): # optional
        print("init")
        print(options)
        return True

    def deinit(self): # optional
        print("deinit")

    def open(self): # optional
        print("open")
        return True

    def fetch(self): # mandatory
        print("fetch")
        # return LogFetcher.FETCH_ERROR,
        # return LogFetcher.FETCH_NOT_CONNECTED,
        # return LogFetcher.FETCH_TRY_AGAIN,
        # return LogFetcher.FETCH_NO_DATA,
        return LogFetcher.FETCH_SUCCESS, msg

    def request_exit(self):
        print("request_exit")
        # If your fetching method is blocking, do something to break it
        # For example, if it reads a socket: socket.shutdown()

    def close(self): # optional
        print("close")
};

Methods of the python-fetcher() source

Fetcher-style Python sources must be inherited from the syslogng.LogFetcher class, and must implement at least the fetch method. Multiple inheritance is allowed, but only for pure Python super classes.

For fetcher-style Python sources, syslog-ng OSE handles the event loop and the scheduling automatically. You can use simple blocking server/client libraries to receive or fetch logs.

You can retrieve messages using the fetch() method.

Limitations:

• The snmptrap() source has only the options listed in snmptrap() source options. Other options commonly available in other source drivers are not supported.

• In addition to traps, the log of snmptrapd may contain other messages (for example, daemon start/stop information, debug logs) as well. Currently syslog-ng OSE discards these messages.

• The syslog-ng OSE application cannot resolve OIDs, you have to configure snmptrapd to do so. Note that because of a bug, if snmptrapd does not escape String values in the VarBindList if it can resolve an OID to a symbolic name. As a result, syslog-ng OSE cannot process traps that contain the = in the value of the string. To overcome this problem, disable resolving OIDs in snmptrapd. For details, see the documentation of snmptrapd.

• The colon (:) character is commonly used in SNMP traps. However, this character cannot be used in the name of syslog-ng OSE macros (name-value pairs). Therefore, the syslog-ng OSE application automatically replaces all consecutive : characters with a single underscore (_) character. For example, you can reference the value of the NET-SNMP-EXAMPLES-MIB::netSnmpExampleString key using the ${NET-SNMP-EXAMPLES-MIB_netSnmpExampleString} macro.

  Note that this affects only name-value pairs (macros). The generated message always contains the original name of the key.

Prerequisites:

• Configure snmptrapd to log into a file.

• If you use SMIv1 traps, include the following format string in the configuration file of snmptrapd:

  format1 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]: %N\n\t%W Trap (%q) Uptime: %#T\n%v\n

• If you use SMIv2 traps, use the default format. The snmptrap() source of syslog-ng OSE expects this default format:

  format2 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]:\n%v\n

• Beacause of an snmptrapd bug, if you specify the filename in the configuration file with logOption, you must also specify another output as a command line argument (-Lf, -Ls). Otherwise, snmptrapd will not apply the the trap format.

To use the snmptrap() driver, the scl.conf file must be included in your syslog-ng OSE configuration:

@include "scl.conf"