syslog-ng Open Source Edition 3.25 - Administration Guide

python-fetcher: writing fetcher-style Python sources

The Python source allows you to write your own source in Python. You can import external Python modules to receive or fetch the messages. Since many services have a Python library, the Python source makes integrating syslog-ng OSE very easy and quick.

You can write two different type of sources in Python:

• Server-style sources that receives messages. Write server-style sources if you want to use an event-loop based, nonblocking server framework in Python, or if you want to implement a custom loop.

• Fetcher-style sources that actively fetch messages. In general, write fetcher-style sources (for example, when using simple blocking APIs), unless you explicitly need a server-style source.

This section describes fetcher-style sources. For details on server-style sources, see python: writing server-style Python sources.

The following points apply to using Python blocks in syslog-ng OSE in general:

• Python parsers and template functions are available in syslog-ng OSE version 3.10 and later.

  Python destinations and sources are available in syslog-ng OSE version 3.18 and later.

• Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).

• The Python block must be a top-level block in the syslog-ng OSE configuration file.

• If you store the Python code in a separate Python file and only include it in the syslog-ng OSE configuration file, make sure that the PYTHON_PATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start syslog-ng OSE manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc

  In production, when syslog-ng OSE starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting syslog-ng OSE. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc"

• The Python object is initiated every time when syslog-ng OSE is started or reloaded.

  Caution: If you reload syslog-ng OSE, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of syslog-ng OSE typically involves a reload.

• The Python block can contain multiple Python functions.

• Using Python code in syslog-ng OSE can significantly decrease the performance of syslog-ng OSE, especially if the Python code is slow. In general, the features of syslog-ng OSE are implemented in C, and are faster than implementations of the same or similar features in Python.

• Validate and lint the Python code before using it. The syslog-ng OSE application does not do any of this.

• Python error messages are available in the internal() source of syslog-ng OSE.

• You can access the name-value pairs of syslog-ng OSE directly through a message object or a dictionary.

• To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of syslog-ng OSE. For details, see Logging from your Python code.

source <name_of_the_python_source>{
    python-fetcher(
        class("<name_of_the_python_class_executed_by_the_source>")
    );
};

python {
from syslogng import LogFetcher
from syslogng import LogMessage

class <name_of_the_python_class_executed_by_the_source>(LogFetcher):
    def init(self, options): # optional
        print("init")
        print(options)
        return True

    def deinit(self): # optional
        print("deinit")

    def open(self): # optional
        print("open")
        return True

    def fetch(self): # mandatory
        print("fetch")
        # return LogFetcher.FETCH_ERROR,
        # return LogFetcher.FETCH_NOT_CONNECTED,
        # return LogFetcher.FETCH_TRY_AGAIN,
        # return LogFetcher.FETCH_NO_DATA,
        return LogFetcher.FETCH_SUCCESS, msg

    def request_exit(self):
        print("request_exit")
        # If your fetching method is blocking, do something to break it
        # For example, if it reads a socket: socket.shutdown()

    def close(self): # optional
        print("close")
};

For the list of available optional parameters, see python() and python-fetcher() source options.

snmptrap: Read Net-SNMP traps

Using the snmptrap() source, you can read and parse the SNMP traps of the Net-SNMP's snmptrapd application. syslog-ng OSE can read these traps from a log file, and extract their content into name-value pairs, making it easy to forward them as a structured log message (for example, in JSON format). The syslog-ng OSE application automatically adds the .snmp. prefix to the name of the fields the extracted from the message.

The snmptrap() source is available in syslog-ng OSE version 3.10 and later.

@include "scl.conf"

authCommunity log,execute,net public
format1 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]: %N\n\t%W Trap (%q) Uptime: %#T\n%v\n
outputOption s

2017-05-23 15:29:40 localhost [UDP: [127.0.0.1]:59993->[127.0.0.1]:162]:
SNMPv2-SMI::mib-2.1.3.0 = Timeticks: (666) 0:00:06.66   SNMPv2-SMI::snmpModules.1.1.4.1.0 = OID: NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification     NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatRate = INTEGER: 60        NET-SNMP-EXAMPLES-MIB::netSnmpExampleString = STRING: string

@include "scl.conf"
log {
  source {
    snmptrap(filename("/var/log/snmptrapd.log"));
  };
  destination {
    file("/var/log/example.log");
  };
};

May 23 15:29:40 myhostname snmptrapd: hostname='localhost', transport_info='UDP: [127.0.0.1]:59993->[127.0.0.1]:162', SNMPv2-SMI::mib-2.1.3.0='(666) 0:00:06.66', SNMPv2-SMI::snmpModules.1.1.4.1.0='NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification', NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatRate='60', NET-SNMP-EXAMPLES-MIB::netSnmpExampleString='string'

@include "scl.conf"
log {
  source {
    snmptrap(
      filename("/var/log/snmptrapd.log")
      set-message-macro(no)
    );
  };

  destination {
    file("/var/log/example.log" template("$(format-json --scope dot-nv-pairs)\n"));
  };
};

{
   "_snmp":{
      "transport_info":"UDP: [127.0.0.1]:59993->[127.0.0.1]:162",
      "hostname":"localhost",
      "SNMPv2-SMI_snmpModules":{
         "1":{
            "1":{
               "4":{
                  "1":{
                     "0":"NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification"
                  }
               }
            }
         }
      },
      "SNMPv2-SMI_mib-2":{
         "1":{
            "3":{
               "0":"(666) 0:00:06.66"
            }
         }
      },
      "NET-SNMP-EXAMPLES-MIB_netSnmpExampleString":"string",
      "NET-SNMP-EXAMPLES-MIB_netSnmpExampleHeartbeatRate":"60"
   }
}

snmptrap() source options

The snmptrap() driver has the following options. Only the filename() option is required, the others are optional.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using the hook-commands() when syslog-ng OSE starts or stops

To execute an external program when syslog-ng OSE starts or stops, use the following options:

Using the hook-commands() when syslog-ng OSE reloads

To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:

source {
   network(transport(udp)
	hook-commands(
          startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
          shutdown("iptables -D LOGCHAIN 1")
        )
     );
};

persist-name()

Description:If you receive the following error message during syslog-ng OSE startup, set the persist-name() option of the duplicate drivers:

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

sun-streams: Collecting messages on Sun Solaris

Solaris uses its STREAMS framework to send messages to the syslogd process. Solaris 2.5.1 and above uses an IPC called door in addition to STREAMS, to confirm the delivery of a message. The syslog-ng application supports the IPC mechanism via the door() option (see below).

The sun-streams() driver has a single required argument specifying the STREAMS device to open, and the door() option. For the list of available optional parameters, see sun-streams() source options.

sun-streams(<name_of_the_streams_device> door(<filename_of_the_door>));

source s_stream {
    sun-streams("/dev/log" door("/etc/.syslog_door"));
};