You can use the graylog2() destination and a Graylog Extended Log Format (GELF) template to send syslog messages to Graylog.
You can forward simple name-value pairs where the name starts with a dot or underscore. If names of your name-value pairs include dots other than the first character, you should use JSON formatting directly instead of the GELF template and send logs to a raw TCP port in Graylog, which can then extract fields from nested JSON.
graylog2();
You can send syslog messages to Graylog using the graylog2() destination. The graylog2() destination uses the GELF template, the native data format of Graylog.
On the Graylog side, configure a GELF TCP input. For more information, see the relevant Graylog documentation.
On the syslog-ng side, configure the name or IP address of the host running Graylog.
destination d_graylog { graylog2( host("172.16.146.142") transport(tcp) ); };
If you parsed your messages using syslog-ng, the template also forwards any name-value pairs where the name starts with a dot or underscore.
If there is a dot in a field name other than the first character, syslog-ng creates nested JSON while formatting the message. Nested JSON is not automatically parsed in GELF messages.
While sending nested JSON inside GELF is possible, it is not convenient. If you use parsing and normalization in syslog-ng and dot notation in field names, use pure JSON instead of GELF to forward your messages.
On the Graylog side, create a new raw TCP input.
Still in Graylog, once the raw TCP input is ready, add a JSON extractor to it.
On the syslog-ng side, use a network destination combined with a template utilizing format-json as shown in the example below:
destination d_jsontcp { network( "172.16.146.142" port("5555") transport(tcp) template("$(format-json --scope all-nv-pairs)\n") ); };
The graylog2() destination has the following options:
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable syslog-ng OSE to execute external applications.
To execute an external program when syslog-ng OSE starts or stops, use the following options:
startup() | |
Type: | string |
Default: | N/A |
Description: Defines the external program that is executed as syslog-ng OSE starts. |
shutdown() | |
Type: | string |
Default: | N/A |
Description: Defines the external program that is executed as syslog-ng OSE stops. |
To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:
setup() | |
Type: | string |
Default: | N/A |
Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload. |
teardown() | |
Type: | string |
Default: | N/A |
Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload. |
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as syslog-ng OSE is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed.
source { network(transport(udp) hook-commands( startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT") shutdown("iptables -D LOGCHAIN 1") ) ); };
Type: | tls options |
Default: | n/a |
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
Type: | udp, tcp, or tls |
Default: | tcp |
Description: Specifies the protocol used to send messages to the destination server.
If you use the udp transport, syslog-ng OSE automatically sends multicast packets if a multicast destination address is specified. The tcp transport does not support multicasting.
Starting with version
For more information about the benefits of using syslog-ng as a data collection, processing, and filtering tool in a Hadoop environment, see the blog post Filling your data lake with log messages: the syslog-ng Hadoop (HDFS) destination.
Note the following limitations when using the syslog-ng OSE hdfs destination:
This destination is only supported on the Linux platform.
Since syslog-ng OSE uses the official Java HDFS client, the hdfs destination has significant memory usage (about 400MB).
You cannot set when log messages are flushed. Hadoop performs this action automatically, depending on its configured block size, and the amount of data received. There is no way for the syslog-ng OSE application to influence when the messages are actually written to disk. This means that syslog-ng OSE cannot guarantee that a message sent to HDFS is actually written to disk. When using flow-control, syslog-ng OSE acknowledges a message as written to disk when it passes the message to the HDFS client. This method is as reliable as your HDFS environment.
The log messages of the underlying client libraries are available in the internal() source of syslog-ng OSE.
@module mod-java @include "scl.conf" hdfs( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-hadoop-libraries>") hdfs-uri("hdfs://NameNode:8020") hdfs-file("<path-to-logfile>") );
The following example defines an hdfs destination using only the required parameters.
@module mod-java @include "scl.conf" destination d_hdfs { hdfs( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/hadoop/libs") hdfs-uri("hdfs://10.140.32.80:8020") hdfs-file("/user/log/logfile.txt") ); };
To install the software required for the hdfs destination, see Prerequisites.
For details on how the hdfs destination works, see How syslog-ng OSE interacts with HDFS.
For details on using MapR-FS, see Storing messages with MapR-FS.
For details on using Kerberos authentication, see Kerberos authentication with syslog-ng hdfs() destination.
For the list of options, see HDFS destination options.
The hdfs() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of syslog-ng OSE. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the hdfs configuration snippet on GitHub. For details on extending syslog-ng OSE in Java, see the Getting started with syslog-ng development guide.
If you delete all Java destinations from your configuration and reload syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again.
To send messages from syslog-ng OSE to HDFS, complete the following steps.
If you want to use the Java-based modules of syslog-ng OSE (for example, the Elasticsearch, HDFS, or Kafka destinations), you must compile syslog-ng OSE with Java support.
Download and install the Java Runtime Environment (JRE), 1.7 (or newer).
Install gradle version 2.2.1 or newer.
Set LD_LIBRARY_PATH to include the libjvm.so file, for example:LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH
Note that many platforms have a simplified links for Java libraries. Use the simplified path if available. If you use a startup script to start syslog-ng OSE set LD_LIBRARY_PATH in the script as well.
If you are behind an HTTP proxy, create a gradle.properties under the modules/java-modules/ directory. Set the proxy parameters in the file. For details, see The Gradle User Guide.
Download the Hadoop Distributed File System (HDFS) libraries (version 2.x) from http://hadoop.apache.org/releases.html.
Extract the HDFS libraries into a temporary directory, then collect the various .jar files into a single directory (for example, /opt/hadoop/lib/) where syslog-ng OSE can access them. You must specify this directory in the syslog-ng OSE configuration file. The files are located in the various lib directories under the share/ directory of the Hadoop release package. (For example, in Hadoop 2.7, required files are common/hadoop-common-2.7.0.jar, common/libs/*.jar, hdfs/hadoop-hdfs-2.7.0.jar, hdfs/lib/*, but this may change between Hadoop releases, so it is easier to copy every .jar file into a single directory.
© 2024 One Identity LLC. ALL RIGHTS RESERVED. Terms of Use Privacy Cookie Preference Center