The graphite() destination can send metrics to a Graphite server to store numeric time-series data. There are many ways to feed the Graphite template function with name value pairs. The syslog-ng OSE CSV and PatternDB parsers (for details, see Using pattern parsers) can parse log messages and generate name value pairs based on message content. The CSV parser (for details, see Parsing messages with comma-separated and similar values) can be used for logs that have a constant field based structure, like the Apache web server access logs. The patterndb parser can parse information and can extract important fields from free form log messages, as long as patterns describing the log messages are available. Another way is to send JSON-based log messages (for details, see JSON parser) to syslog-ng OSE, like running a simple shell script collecting metrics and running it from cron regularly.
To see an example of how the graphite() destination is used to collect statistics coming from syslog-ng, see the blog post Collecting syslog-ng statistics to Graphite.
Declaration:
graphite(payload());
Example: Using the graphite() driver
To use the graphite() destination, the only mandatory parameter is payload, which specifies the value pairs to send to graphite. In the following example any value pairs starting with "monitor." are forwarded to graphite.
destination d_graphite { graphite(payload("--key monitor.*")); };
NOTE: The graphite() destination is only a wrapper around the network() destination and the graphite-output template function. If you want to fine-tune the TCP parameters, use the network() destination instead, as described in graphite-output.
The graphite() destination has the following options:
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable syslog-ng OSE to execute external applications.
Using the hook-commands() when syslog-ng OSE starts or stops
To execute an external program when syslog-ng OSE starts or stops, use the following options:
| startup() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines the external program that is executed as syslog-ng OSE starts. |
| shutdown() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines the external program that is executed as syslog-ng OSE stops. |
Using the hook-commands() when syslog-ng OSE reloads
To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:
| setup() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload. |
| teardown() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload. |
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as syslog-ng OSE is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed.
source {
network(transport(udp)
hook-commands(
startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
shutdown("iptables -D LOGCHAIN 1")
)
);
};
host()
| Type: |
hostname or IP address |
| Default: |
localhost |
Description: The hostname or IP address of the Graphite server.
port()
| Type: |
number |
| Default: |
2003 |
Description: The port number of the Graphite server.
payload()
| Type: |
parameter list of the payload() option |
| Default: |
empty string |
Description: The payload() option allows you to select which value pairs to forward to graphite.
The syntax of payload is different from the syntax of value-pairs(): use the command-line syntax used in the format-json template function. For details on using the payload() option, see graphite-output.
NOTE: If left empty, there is no data to be forwarded to Graphite.
graylog2(): Sending logs to Graylog
You can use the graylog2() destination and a Graylog Extended Log Format (GELF) template to send syslog messages to Graylog.
You can forward simple name-value pairs where the name starts with a dot or underscore. If names of your name-value pairs include dots other than the first character, you should use JSON formatting directly instead of the GELF template and send logs to a raw TCP port in Graylog, which can then extract fields from nested JSON. Version 3.21 and later also supports TLS-encrypted connection to the Graylog server.
Example: Using the graylog2() driver
You can send syslog messages to Graylog using the graylog2() destination. The graylog2() destination uses the GELF template, the native data format of Graylog.
-
On the Graylog side, configure a GELF TCP input. For more information, see the relevant Graylog documentation.
-
On the syslog-ng side, configure the name or IP address of the host running Graylog.
destination d_graylog {
graylog2(
host("172.16.146.142")
transport(tcp)
);
};
If you parsed your messages using syslog-ng, the template also forwards any name-value pairs where the name starts with a dot or underscore.
NOTE: If there is a dot in a field name other than the first character, syslog-ng creates nested JSON while formatting the message. Nested JSON is not automatically parsed in GELF messages.
Sending nested JSON to Graylog
While sending nested JSON inside GELF is possible, it is not convenient. If you use parsing and normalization in syslog-ng and dot notation in field names, use pure JSON instead of GELF to forward your messages.
-
On the Graylog side, create a new raw TCP input.
-
Still in Graylog, once the raw TCP input is ready, add a JSON extractor to it.
-
On the syslog-ng side, use a network destination combined with a template utilizing format-json as shown in the example below:
destination d_jsontcp {
network(
"172.16.146.142"
port("5555")
transport(tcp)
template("$(format-json --scope all-nv-pairs)\n")
);
};
The graylog2() destination has the following options:
ca-dir()
| Accepted values: |
Directory name |
| Default: |
none |
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject's name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the syslog-ng clients. The syslog-ng OSE application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
| Accepted values: |
File name |
| Default: |
empty |
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The syslog-ng OSE application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
NOTE: The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable syslog-ng OSE to execute external applications.
Using the hook-commands() when syslog-ng OSE starts or stops
To execute an external program when syslog-ng OSE starts or stops, use the following options:
| startup() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines the external program that is executed as syslog-ng OSE starts. |
| shutdown() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines the external program that is executed as syslog-ng OSE stops. |
Using the hook-commands() when syslog-ng OSE reloads
To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:
| setup() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload. |
| teardown() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload. |
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as syslog-ng OSE is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed.
source {
network(transport(udp)
hook-commands(
startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
shutdown("iptables -D LOGCHAIN 1")
)
);
};
tls()
| Type: |
tls options |
| Default: |
n/a |
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
transport()
| Type: |
udp, tcp, or tls |
| Default: |
tcp |
Description: Specifies the protocol used to send messages to the destination server.
If you use the udp transport, syslog-ng OSE automatically sends multicast packets if a multicast destination address is specified. The tcp transport does not support multicasting.
