graylog2(): Sending logs to Graylog
You can use the graylog2() destination and a Graylog Extended Log Format (GELF) template to send syslog messages to Graylog.
You can forward simple name-value pairs where the name starts with a dot or underscore. If names of your name-value pairs include dots other than the first character, you should use JSON formatting directly instead of the GELF template and send logs to a raw TCP port in Graylog, which can then extract fields from nested JSON. Version 3.21 and later also supports TLS-encrypted connection to the Graylog server.
Example: Using the graylog2() driver
You can send syslog messages to Graylog using the graylog2() destination. The graylog2() destination uses the GELF template, the native data format of Graylog.
-
On the Graylog side, configure a GELF TCP input. For more information, see the relevant Graylog documentation.
-
On the syslog-ng side, configure the name or IP address of the host running Graylog.
destination d_graylog {
graylog2(
host("172.16.146.142")
transport(tcp)
);
};
If you parsed your messages using syslog-ng, the template also forwards any name-value pairs where the name starts with a dot or underscore.
NOTE: If there is a dot in a field name other than the first character, syslog-ng creates nested JSON while formatting the message. Nested JSON is not automatically parsed in GELF messages.
Sending nested JSON to Graylog
While sending nested JSON inside GELF is possible, it is not convenient. If you use parsing and normalization in syslog-ng and dot notation in field names, use pure JSON instead of GELF to forward your messages.
-
On the Graylog side, create a new raw TCP input.
-
Still in Graylog, once the raw TCP input is ready, add a JSON extractor to it.
-
On the syslog-ng side, use a network destination combined with a template utilizing format-json as shown in the example below:
destination d_jsontcp {
network(
"172.16.146.142"
port("5555")
transport(tcp)
template("$(format-json --scope all-nv-pairs)\n")
);
};
The graylog2() destination has the following options:
ca-dir()
| Accepted values: |
Directory name |
| Default: |
none |
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject's name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the syslog-ng clients. The syslog-ng OSE application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
| Accepted values: |
File name |
| Default: |
empty |
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The syslog-ng OSE application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
NOTE: The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable syslog-ng OSE to execute external applications.
Using the hook-commands() when syslog-ng OSE starts or stops
To execute an external program when syslog-ng OSE starts or stops, use the following options:
| startup() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines the external program that is executed as syslog-ng OSE starts. |
| shutdown() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines the external program that is executed as syslog-ng OSE stops. |
Using the hook-commands() when syslog-ng OSE reloads
To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:
| setup() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload. |
| teardown() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload. |
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as syslog-ng OSE is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed.
source {
network(transport(udp)
hook-commands(
startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
shutdown("iptables -D LOGCHAIN 1")
)
);
};
tls()
| Type: |
tls options |
| Default: |
n/a |
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
transport()
| Type: |
udp, tcp, or tls |
| Default: |
tcp |
Description: Specifies the protocol used to send messages to the destination server.
If you use the udp transport, syslog-ng OSE automatically sends multicast packets if a multicast destination address is specified. The tcp transport does not support multicasting.
Starting with version 3.7, syslog-ng OSE can send plain-text log files to the Hadoop Distributed File System (HDFS), allowing you to store your log data on a distributed, scalable file system. This is especially useful if you have huge amounts of log messages that would be difficult to store otherwise, or if you want to process your messages using Hadoop tools (for example, Apache Pig).
For more information about the benefits of using syslog-ng as a data collection, processing, and filtering tool in a Hadoop environment, see the blog post Filling your data lake with log messages: the syslog-ng Hadoop (HDFS) destination.
Note the following limitations when using the syslog-ng OSE hdfs destination:
-
This destination is only supported on the Linux platform.
-
Since syslog-ng OSE uses the official Java HDFS client, the hdfs destination has significant memory usage (about 400MB).
-
You cannot set when log messages are flushed. Hadoop performs this action automatically, depending on its configured block size, and the amount of data received. There is no way for the syslog-ng OSE application to influence when the messages are actually written to disk. This means that syslog-ng OSE cannot guarantee that a message sent to HDFS is actually written to disk. When using flow-control, syslog-ng OSE acknowledges a message as written to disk when it passes the message to the HDFS client. This method is as reliable as your HDFS environment.
-
The log messages of the underlying client libraries are available in the internal() source of syslog-ng OSE.
Declaration:
@include "scl.conf"
hdfs(
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-hadoop-libraries>")
hdfs-uri("hdfs://NameNode:8020")
hdfs-file("<path-to-logfile>")
);
Example: Storing logfiles on HDFS
The following example defines an hdfs destination using only the required parameters.
@include "scl.conf"
destination d_hdfs {
hdfs(
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/hadoop/libs")
hdfs-uri("hdfs://10.140.32.80:8020")
hdfs-file("/user/log/logfile.txt")
);
};
The hdfs() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of syslog-ng OSE. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the hdfs configuration snippet on GitHub. For details on extending syslog-ng OSE in Java, see the Getting started with syslog-ng development guide.
NOTE: If you delete all Java destinations from your configuration and reload syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again.
To send messages from syslog-ng OSE to HDFS, complete the following steps.
Steps:
-
If you want to use the Java-based modules of syslog-ng OSE (for example, the Elasticsearch, HDFS, or Kafka destinations), you must compile syslog-ng OSE with Java support.
-
Download and install the Java Runtime Environment (JRE), 1.7 (or newer). You can use OpenJDK or Oracle JDK, other implementations are not tested.
-
Install gradle version 2.2.1 or newer.
-
Set LD_LIBRARY_PATH to include the libjvm.so file, for example:LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH
Note that many platforms have a simplified links for Java libraries. Use the simplified path if available. If you use a startup script to start syslog-ng OSE set LD_LIBRARY_PATH in the script as well.
-
If you are behind an HTTP proxy, create a gradle.properties under the modules/java-modules/ directory. Set the proxy parameters in the file. For details, see The Gradle User Guide.
-
Download the Hadoop Distributed File System (HDFS) libraries (version 2.x) from http://hadoop.apache.org/releases.html.
-
Extract the HDFS libraries into a temporary directory, then collect the various .jar files into a single directory (for example, /opt/hadoop/lib/) where syslog-ng OSE can access them. You must specify this directory in the syslog-ng OSE configuration file. The files are located in the various lib directories under the share/ directory of the Hadoop release package. (For example, in Hadoop 2.7, required files are common/hadoop-common-2.7.0.jar, common/libs/*.jar, hdfs/hadoop-hdfs-2.7.0.jar, hdfs/lib/*, but this may change between Hadoop releases, so it is easier to copy every .jar file into a single directory.
