syslog-ng Open Source Edition 3.25 - Administration Guide

Posting messages over HTTP

Version 3.7 of syslog-ng OSE can directly post log messages to web services using the HTTP protocol. Error and status messages received from the HTTP server are forwarded to the internal logs of syslog-ng OSE. The current implementation has the following limitations:

• This destination is only supported on the Linux platform.

• Only HTTP connections are supported, HTTPS is not.

• This destination requires Java. For an http destination that does not use Java, see http: Posting messages over HTTP without Java.

@module mod-java

java(
    class-path("/syslog-ng/install_dir/lib/syslog-ng/java-modules/*.jar")
    class-name("org.syslog_ng.http.HTTPDestination")

    option("url", "http://<server-address>:<port-number>")

);

@module mod-java

destination d_http {
    java(
        class-path("/syslog-ng/install_dir/lib/syslog-ng/java-modules/*.jar")
        class-name("org.syslog_ng.http.HTTPDestination")

        option("url", "http://192.168.1.1:80")
    );
};

log
    { source(s_file); destination(d_http); flags(flow-control); };

HTTP destination options

The http destination of syslog-ng OSE can directly post log messages to web services using the HTTP protocol. The http destination has the following options. Some of these options are directly used by the Java code underlying the http destination, therefore these options must be specified in the following format:

option("<option-name>", "<option-value>")

For example, option("url", "http://<server-address>:<port-number>"). The exact format to use is indicated in the description of the option.

@module mod-java

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using the hook-commands() when syslog-ng OSE starts or stops

To execute an external program when syslog-ng OSE starts or stops, use the following options:

Using the hook-commands() when syslog-ng OSE reloads

To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:

source {
   network(transport(udp)
	hook-commands(
          startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
          shutdown("iptables -D LOGCHAIN 1")
        )
     );
};

jvm-options()

Description: Specify the Java Virtual Machine (JVM) settings of your Java destination from the syslog-ng OSE configuration file.

For example:

jvm-options("-Xss1M -XX:+TraceClassLoading")

You can set this option only as a global option, by adding it to the options statement of the syslog-ng configuration file.

log-fifo-size()

Description: The number of messages that the output queue can store.

retries()

Description: The number of times syslog-ng OSE attempts to send a message to this destination. If syslog-ng OSE could not send a message, it will try again until the number of attempts reaches retries, then drops the message.

template()

Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of syslog-ng OSE. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.

throttle()

Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

http: Posting messages over HTTP without Java

Version 3.8 of syslog-ng OSE can directly post log messages to web services using the HTTP protocol, without having to use Java. The current implementation has the following limitations:

• Only the PUT and the POST methods are supported.

HTTPS connection, as well as password- and certificate-based authentication is supported.

If the server returns a status code beginning with 2 (for example, 200), syslog-ng OSE assumes the message was successfully sent. For other response codes, see HTTP destination options. You can override the behavior of syslog-ng OSE using the response-action() option.

destination d_https {
    http(
        [...]
        tls(
        ca-file("/<path-to-certificate-directory>/ca-crt.pem")
        ca-dir("/<path-to-certificate-directory>/")
        cert-file("/<path-to-certificate-directory>/server-crt.pem")
        key-file("/<path-to-certificate-directory>/server-key.pem")
            )
        [...]
    );
};

destination d_http {
    http(
        url("<web-service-IP-or-hostname>")
        method("<HTTP-method>")
        user-agent("<USER-AGENT-message-value>")
        user("<username>")
        password("<password>")
    );
};

The driver automatically uses the proxy settings of the host, there is no option to set it in the syslog-ng OSE configuration. Specifically, the driver uses the settings of the http_proxy and https_proxy environment variables.

destination d_http {
    http(
        url("http://127.0.0.1:8000")
        method("PUT")
        user-agent("syslog-ng User Agent")
        user("user")
        password("password")
        headers("HEADER1: header1", "HEADER2: header2")
        body("${ISODATE} ${MESSAGE}")
    );
};

log {
    source(s_file);
    destination(d_http);
    flags(flow-control);
};

You can also use the http() destination to forward log messages to Splunk using syslog-ng OSE.

Batch mode and load balancing

value of body-prefix() option
log-line-1 (as formatted in the body() option)
log-line-2 (as formatted in the body() option)
....
log-line-n (the number of log lines is batch-lines(), or less if batch-timeout() has elapsed or the batch would be longer than batch-bytes())
value of body-suffix() option

destination d_http {
    http(url("http://your-elasticsearch-server/_bulk")
        method("POST")
        batch-lines(100)
        batch-bytes(512Kb)
        batch-timeout(10000)
        headers("Content-Type: application/x-ndjson")
        body-suffix("\n")
        body('{ "index":{} }
             $(format-json --scope rfc5424 --key ISODATE)')
    );
};

url("server1", "server2", "server3"); # comma-separated strings
url("server1" "server2" "server3"); # space-separated strings
url("server1 server2 server3"); # space-separated within a single string
url("server1,server2,server3"); # comma-separated within a single string

destination d_http {
    http(url("http://your-elasticsearch-server/_bulk" "http://your-second-ingest-node/_bulk" "http://your-third-ingest-node/_bulk")
        method("POST")
        batch-lines(100)
        batch-bytes(512Kb)
        batch-timeout(10000)
        workers(3)
        headers("Content-Type: application/x-ndjson")
        body-suffix("\n")
        body('{ "index":{} }
             $(format-json --scope rfc5424 --key ISODATE)')
        persist-name("d_http-load-balance")
    );
};