syslog-ng Open Source Edition 3.30 - Administration Guide

The syslog-ng OSE application supports adding custom headers to HTTP requests using the Python programming language.

Prerequisites

Configuration

destination d_http {
  http(
    python_http_header(
      class("<class-name>")
      options("options-key1", "option-value1")
      options("options-key2", "option-value2")
      mark-errors-as-critical(no))
    url("http://127.0.0.1:8888")
  );
};

Options used in the configuration

• class: Mandatory option. It refers to the user's Python class that implements the python-http-header interface. It can be mymodule.MyClass if the class MyClass is put into a mymodule.py module, or simply MyClass if the user's code is provided inline in the configuration, using the python { ... }; keyword.

  NOTE: If you put the class implementation into its own module, it should be put into a standard location, or made available with the PYTHONPATH environment variable.

• options("key" "value"): Optional option. Multiple options can be specified at the same time. The syslog-ng OSE application will build a Python dictionary, which will be available in the __init__ method.
• mark-errors-as-critical(yes|no): Optional option. Its default value is yes. In case there is a Python error, this parameter decides if the HTTP destination will still try to send the request with the failed headers, or disconnect instead.

Defining the python-http-header() interface

You can define the Python interface with the following:

class TestCounter():
  def __init__(self, options):
    self.key = options["value"]

  def get_headers(self, body, headers):
    return ["header1: value1", "header2: value2"]

  def on_http_response_received(self, http_code):
    print("HTTP response code received: {}".format(http_code))

By default, when the signal_http_header_request is emitted by the HTTP module, the connected slot automatically executes the Python code.

Declaration:

@include "scl.conf"

kafka(
    client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-kafka-libraries>")
    kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
    topic("${HOST}")

);

Steps:

1. If you want to use the Java-based modules of syslog-ng OSE (for example, the Elasticsearch, HDFS, or Kafka destinations), you must compile syslog-ng OSE with Java support.

   • Download and install the Java Runtime Environment (JRE), 1.7 (or newer). You can use OpenJDK or Oracle JDK, other implementations are not tested.

   • Install gradle version 2.2.1 or newer.

   • Set LD_LIBRARY_PATH to include the libjvm.so file, for example:LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH

     Note that many platforms have a simplified links for Java libraries. Use the simplified path if available. If you use a startup script to start syslog-ng OSE set LD_LIBRARY_PATH in the script as well.

   • If you are behind an HTTP proxy, create a gradle.properties under the modules/java-modules/ directory. Set the proxy parameters in the file. For details, see The Gradle User Guide.

2. Download the latest stable binary release of the Apache Kafka libraries (version 0.9 or newer) from http://kafka.apache.org/downloads.html.

3. Extract the Apache Kafka libraries into a single directory. If needed, collect the various .jar files into a single directory (for example, /opt/kafka/lib/) where syslog-ng OSE can access them. You must specify this directory in the syslog-ng OSE configuration file.

4. Check if the following files in the Kafka libraries have the same version number: slf4j-api-<version-number>.jar, slf4j-log4j12-<version-number>.jar. If the version number of these files is different, complete the following steps:

   1. Delete one of the files (for example, slf4j-log4j12-<version-number>.jar).

   2. Download a version that matches the version number of the other file (for example, 1.7.6) from the official SLF4J distribution.

   3. Copy the downloaded file into the directory of your Kafka library files (for example, /opt/kafka/lib/).