syslog-ng Premium Edition 7.0.14 - Administration Guide

python: writing custom Python destinations

The Python destination allows you to write your own destination in Python. You can import external Python modules to process the messages, and send them to other services or servers. Since many services have a Python library, the Python destination makes integrating syslog-ng PE very easy and quick.

The following points apply to using Python blocks in syslog-ng PE in general.

• Python parsers and template functions are available in syslog-ng PE version 7.0.2 and later.

  Python destinations and sources are available in syslog-ng PE version 7.0.11 and later.

• Supported Python versions: 2.7

• The Python block must be a top-level block in the syslog-ng PE configuration file.

• If you store the Python code in a separate Python file and only include it in the syslog-ng PE configuration file, make sure that the PYTHON_PATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start syslog-ng PE manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc

  In production, when syslog-ng PE starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting syslog-ng PE. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc"

• The Python object is initiated every time when syslog-ng PE is started or reloaded.

  Caution: If you reload syslog-ng PE, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of syslog-ng PE typically involves a reload.

• The Python block can contain multiple Python functions.

• Using Python code in syslog-ng PE can significantly decrease the performance of syslog-ng PE, especially if the Python code is slow. In general, the features of syslog-ng PE are implemented in C, and are faster than implementations of the same or similar features in Python.

• Validate and lint the Python code before using it. The syslog-ng PE application does not do any of this.

• Python error messages are available in the internal() source of syslog-ng PE.

• You can access the name-value pairs of syslog-ng PE directly through a message object or a dict.

• Support disclaimer

  Using Python in syslog-ng PE is recommended only if you are familiar with both Python and syslog-ng PE. Product support applies only to syslog-ng PE: that is, until the entry point of the Python code and passing the specified arguments to the Python code. One Identity is not responsible for the quality, resource requirements, or any bugs in the Python code, nor any syslog-ng PE crashes, message losses, or any other damage caused by the improper use of this feature, unless explicitly stated in a contract with One Identity.

destination <name_of_the_python_destination>{
    python(
        class("<name_of_the_python_class_executed_by_the_destination>")
    );
};

python {
class <name_of_the_python_class_executed_by_the_destination>(object):

    def open(self):
        """Open a connection to the target service

        Should return False if opening fails"""
        return True

    def close(self):
        """Close the connection to the target service"""
        pass

    def is_opened(self):
        """Check if the connection to the target is able to receive messages"""
        return True

    def init(self, options):
        """This method is called at initialization time

        Should return false if initialization fails"""
        return True

    def deinit(self):
        """This method is called at deinitialization time"""
        pass

    def send(self, msg):
        """Send a message to the target service

        It should return True to indicate success. False will suspend the
        destination for a period specified by the time-reopen() option.
        This will be repeated for the same message retries() times.

        Alternatively, it can return the following integer values:
        self.SUCCESS: message sending was successful (same as boolean True)
        self.ERROR: message sending was unsuccessful (same as boolean False)
        self.DROP: the message cannot be sent, it should be dropped immediately
        self.QUEUED: the message is not sent immediately, it will be sent in a batch with the flush method
        self.NOT_CONNECTED: the message is put back into the queue, the open method will be called until it succeeds
        self.RETRY: the message is put back into the queue, to retry send it retries() times, then fallback to self.NOT_CONNECTED
        """
        return True
};

destination d_python_to_file {
    python(
        class("TextDestination")
    );
};
log {
    source(src);
    destination(d_python_to_file);
};
python {
class TextDestination(object):
    def send(self, msg):
        self.outfile = open("/tmp/example.txt", "a")
        self.outfile.write("MESSAGE = %s\n" % msg["MESSAGE"])
        self.outfile.flush()
        self.outfile.close();
        return True
};

destination d_python_to_file {
    python(
        class("TextDestination")
    );
};
log {
    source(src);
    destination(d_python_to_file);
};
python {
class TextDestination(object):
    def open(self):
        try:
            self.outfile = open("/tmp/example.txt", "a")
            return True
        except:
            return False

    def send(self, msg):
        self.outfile.write("MESSAGE = %s\n" % msg["MESSAGE"])
        self.outfile.flush()
        return True

    def close(self):
        try:
            self.outfile.flush()
            self.outfile.close();
            return True
        except:
            return False
};

For the list of available optional parameters, see python() destination options.

python() destination options

The Python destination allows you to write your own destination in Python. The python() destination has the following options. The class() option is mandatory. For details on writing destinations in Python, see python: writing custom Python destinations.

python(
    class("MyPythonDestination")
);

python(
    class("MyPythonfilename.MyPythonDestination")
);

disk-buffer()

Description: This option enables putting outgoing messages into the disk-buffer file of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:

Options reliable() and disk-buf-size() are required options.

destination d_demo {
    network("127.0.0.1"
        port(3333)
        disk-buffer(
            mem-buf-size(10000)
            disk-buf-size(2000000)
            reliable(yes)
            dir("/tmp/disk-buffer")
        )
    );
};

destination d_demo {
    network("127.0.0.1"
        port(3333)
        disk-buffer(
            mem-buf-length(10000)
            disk-buf-size(2000000)
            reliable(no)
            dir("/tmp/disk-buffer")
        )
    );
};

frac-digits()

Description: The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

log-fifo-size()

Description: The number of messages that the output queue can store.

on-error()

Description: Controls what happens when type-casting fails and syslog-ng PE cannot convert some data to the specified type. By default, syslog-ng PE drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().

• drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of syslog-ng PE.

• drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.

• fallback-to-string: Convert the property to string and log an error message to the internal() source.

• silently-drop-message: Drop the entire message silently, without logging the error.

• silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.

• silently-fallback-to-string: Convert the property to string silently, without logging the error.

python(
    class("MyPythonClass")
    options(
        "host" "127.0.0.1"
        "port" "1883"
        "otheroption" "value")
);

persist-name()

Description:If you receive the following error message during syslog-ng PE startup, set the persist-name() option of the duplicate drivers:

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

throttle()

Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using the disk-buffer option as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

scope("selected-macros" "nv-pairs")

destination d_python_to_file {
    python(
        class("pythonexample.TextDestination")
        value-pairs(key(MESSAGE))
    );
};

smtp: Generating SMTP messages (e-mail) from logs

The destination is aimed at a fully controlled local, or near-local, trusted SMTP server. The goal is to send mail to trusted recipients, through a controlled channel. It hands mails over to an SMTP server, and that is all it does, therefore the resulting solution is as reliable as sending an e-mail in general. For example, syslog-ng PE does not verify whether the recipient exists.

The smtp() driver sends e-mail messages triggered by log messages. The smtp() driver uses SMTP, without needing external applications. You can customize the main fields of the e-mail, add extra headers, send the e-mail to multiple recipients, and so on.

The subject(), body(), and header() fields may include macros which get expanded in the e-mail. For more information on available macros see Macros of syslog-ng PE.

The smtp() driver has the following required parameters: host(), port(), from(), to(), subject(), and body(). For the list of available optional parameters, see smtp() destination options.

Declaration

smtp(host() port() from() to() subject() body() options());

destination d_smtp {
    smtp(
        host("localhost")
        port(25)
        from("syslog-ng alert service" "noreply@example.com")
        to("Admin #1" "admin1@example.com")
        subject("[ALERT] Important log message of $LEVEL condition received from $HOST/$PROGRAM!")
        body("Hi!\nThe syslog-ng alerting service detected the following important log message:\n $MSG\n-- \nsyslog-ng\n")
    );
};

destination d_smtp {
    smtp(
        host("localhost")
        port(25)
        from("syslog-ng alert service" "noreply@example.com")
        to("Admin #1" "admin1@example.com")
        to("Admin #2" "admin2@example.com")
        cc("Admin BOSS" "admin.boss@example.com")
        bcc("Blind CC" "blindcc@example.com")
        subject("[ALERT] Important log message of $LEVEL condition received from $HOST/$PROGRAM!")
        body("Hi!\nThe syslog-ng alerting service detected the following important log message:\n $MSG\n-- \nsyslog-ng\n")
        header("X-Program", "$PROGRAM")
    );
};

filter f_linkdown {
    match("eth0: link down" value("MESSAGE"));
};
destination d_alert {
    smtp(
        host("localhost") port(25)
        from("syslog-ng alert service" "syslog@localhost")
        reply-to("Admins" "root@localhost")
        to("Ennekem" "me@localhost")
        subject("[SYSLOG ALERT]: eth0 link down")
        body("Syslog received an alert:\n$MSG")
    );
};

log {
    source(s_local);
    filter(f_linkdown);
    destination(d_alert);
};

smtp() destination options

The smtp() sends e-mail messages using SMTP, without needing external applications. The smtp() destination has the following options:

body()

Description: The BODY field of the e-mail. You can also use macros in the string. Use \n to start a new line. For example:

body("syslog-ng PE received the following alert from $HOST:\n$MSG")

bcc()

Description: The BCC recipient of the e-mail (contents of the BCC field). You can specify the e-mail address, or the name and the e-mail address. Set the bcc() option multiple times to send the e-mail to multiple recipients. For example: bcc("admin@example.com") or bcc("Admin" "admin@example.com") or bcc("Admin" "admin@example.com") bcc("Admin2" "admin2@example.com")

You can also use macros to set the value of this parameter.

cc()

Description: The CC recipient of the e-mail (contents of the CC field). You can specify the e-mail address, or the name and the e-mail address. Set the cc() option multiple times to send the e-mail to multiple recipients. For example: cc("admin@example.com") or cc("Admin" "admin@example.com") or cc("Admin" "admin@example.com") cc("Admin2" "admin2@example.com")

You can also use macros to set the value of this parameter.

disk-buffer()

Description: This option enables putting outgoing messages into the disk-buffer file of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:

Options reliable() and disk-buf-size() are required options.

destination d_demo {
    network("127.0.0.1"
        port(3333)
        disk-buffer(
            mem-buf-size(10000)
            disk-buf-size(2000000)
            reliable(yes)
            dir("/tmp/disk-buffer")
        )
    );
};

destination d_demo {
    network("127.0.0.1"
        port(3333)
        disk-buffer(
            mem-buf-length(10000)
            disk-buf-size(2000000)
            reliable(no)
            dir("/tmp/disk-buffer")
        )
    );
};

from()

Description: The sender of the e-mail (contents of the FROM field). You can specify the e-mail address, or the name and the e-mail address. For example:

from("admin@example.com")

from("Admin" "admin@example.com")

If you specify the from() option multiple times, the last value will be used. Instead of the from() option, you can also use sender(), which is just an alias of the from() option.

You can also use macros to set the value of this parameter.

header()

Description: Adds an extra header to the e-mail with the specified name and content. The first parameter sets the name of the header, the second one its value. The value of the header can contain macros. Set the header() option multiple times to add multiple headers. For example:

header("X-Program", "$PROGRAM")

When using the header option, note the following points:

• Do not use the header() option to set the values of headers that have dedicated options. Use it only to add extra headers.

• If you set the same custom header multiple times, only the first will be added to the e-mail, other occurrences will be ignored.

• It is not possible to set the DATE, Return-Path, Original-Recipient, Content-*, MIME-*, Resent-*, Received headers.

host()

Description: Hostname or IP address of the SMTP server.

log-fifo-size()

Description: The number of messages that the output queue can store.

port()

Description: The port number of the SMTP server.

reply-to()

Description: Replies of the recipient will be sent to this address (contents of the REPLY-TO field). You can specify the e-mail address, or the name and the e-mail address. Set the reply-to() option multiple times to send the e-mail to multiple recipients. For example: reply-to("admin@example.com") or reply-to("Admin" "admin@example.com") or reply-to("Admin" "admin@example.com") reply-to("Admin2" "admin2@example.com")

You can also use macros to set the value of this parameter.

retries()

Description: The number of times syslog-ng PE attempts to send a message to this destination. If syslog-ng PE could not send a message, it will try again until the number of attempts reaches retries, then drops the message.

subject()

Description: The SUBJECT field of the e-mail. You can also use macros. For example:

subject("[SYSLOG ALERT]: Critical error message received from $HOST")

If you specify the subject() option multiple times, the last value will be used.

throttle()

Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using the disk-buffer option as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

to()

Description: The recipient of the e-mail (contents of the TO field). You can specify the e-mail address, or the name and the e-mail address. Set the to() option multiple times to send the e-mail to multiple recipients. For example: to("admin@example.com") or to("Admin" "admin@example.com") or to("Admin" "admin@example.com") to("Admin2" "admin2@example.com")

You can also use macros to set the value of this parameter.