Modules in syslog-ng PE
The syslog-ng PE application is modular, to increase its flexibility and also to simplify the development of additional modules. Most of the functionality of syslog-ng PE is in separate modules. That way it becomes also possible to finetune the resource requirements of syslog-ng PE, for example, by loading only the modules that are actually used in the configuration, or simply omitting modules that are not used but require large amount of memory.
Each module contains one or more plugins, which add some functionality to syslog-ng PE, for example, a destination or a source driver.
-
To display the list of available modules, execute the syslog-ng --version command.
-
To the description of the available modules, execute the syslog-ng --module-registry command.
-
To customize which modules are loaded automatically when syslog-ng PE is started, use the --default-modules command-line option of syslog-ng PE.
-
To request loading a module from the syslog-ng PE configuration file, see the section called “Loading modules”.
For details on the command-line parameters of syslog-ng PE mentioned in the previous list, see the syslog-ng PE man page at syslog-ng(8).
The syslog-ng Premium Edition application loads every available module during startup, except the snmp() module, and Java-related modules like hdfs() (Java-related modules require the mod-java module). For details on using the snmp() destination driver, see the section called “Sending SNMP traps”.
To load a module that is not loaded automatically, include the following statement in the syslog-ng PE configuration file:
@module <module-name>
Note the following points about the @module statement:
-
The
@modulestatement is a top-level statement, that is, it cannot be nested into any other statement. Usually it is used immediately after the@versionstatement. -
Every
@modulestatement loads a single module: loading multiple modules requires a separate@modulestatement for every module. -
In the configuration file, the
@modulestatement of a module must be earlier than the module is used.
Managing complex syslog-ng configurations
The following sections describe some methods that can be useful to simplify the management of large-scale syslog-ng PE installations. If you are using Puppet to manage your IT infrastructure, you can use it to manage your syslog-ng PE configurations as well. For details, see Procedure 3.10, “Managing syslog-ng PE from Puppet”.
The syslog-ng application supports including external files in its configuration file, so parts of its configuration can be managed separately. To include the contents of a file in the syslog-ng configuration, use the following syntax:
@include "<path to filename>"
|
|
NOTE:
If you enter only the filename, syslog-ng PE will search for the file in the default directory: |
This imports the entire file into the configuration of syslog-ng PE, at the location of the include statement. The <filename> can be one of the following:
-
A filename, optionally with full path. The filename (not the path) can include UNIX-style wildcard characters (
*,?). When using wildcard characters, syslog-ng PE will include every matching file. For details on using wildcard characters, see the section called “glob”. -
A directory. When including a directory, syslog-ng PE will try to include every file from the directory, except files beginning with a ~ (tilde) or a . (dot) character. Including a directory is not recursive. The files are included in alphabetic order, first files beginning with uppercase characters, then files beginning with lowercase characters. For example, if the directory contains the
a.conf, B. conf, c.conf, D.conffiles, they will be included in the following order:B.conf, D. conf, a.conf, c.conf.
When including configuration files, consider the following points:
-
If an object is defined twice (for example the original syslog-ng configuration file and the file imported into this configuration file both define the same option, source, or other object), then the object that is defined later in the configuration file will be effective. For example, if you set a global option at the beginning of the configuration file, and later include a file that defines the same option with a different value, then the option defined in the imported file will be used.
-
Files can be embedded into each other: the included files can contain include statements as well, up to a maximum depth of 15 levels.
-
You cannot include complete configuration files into each other, only configuration snippets can be included. This means that the included file cannot have a
@versionstatement. -
Include statements can only be used at top level of the configuration file. For example, the following is correct:
@version: 6.0 @include "example.conf"
But the following is not:
source s_example { @include "example.conf" };
|
|
Caution:
The syslog-ng application will not start if it cannot find a file that is to be included in its configuration. Always double-check the filenames, paths, and access rights when including configuration files, and use the --syntax-only command-line option to check your configuration. |
To create a reusable configuration snippet and reuse parts of a configuration file, you have to define the block (for example, a source) once, and reference it later. (Such reusable blocks are sometimes called a Source Configuration Library, or SCL.) Any syslog-ng object can be a block. Use the following syntax to define a block:
block type name() {<contents of the block>};
Type must be one of the following: destination, filter, log, parser, rewrite, root, source. The root blocks can be used in the "root" context of the configuration file, that is, outside any other statements.
Blocks may be nested into each other, so for example a block can be built from other blocks. Blocks are somewhat similar to C++ templates.
The type and name combination of each block must be unique, that is, two blocks can have the same name if their type is different.
To use a block in your configuration file, you have to do two things:
-
Include the file defining the block in the
syslog-ng.conffile — or a file already included intosyslog-ng.conf. -
Reference the name of the block in your configuration file. This will insert the block into your configuration. For example, to use a block called
myblock, include the following line in your configuration:myblock()
Blocks may have parameters, but even if they do not, the reference must include opening and closing parentheses like in the previous example.
The contents of the block will be inserted into the configuration when syslog-ng PE is started or reloaded.
Example 5.4. Reusing configuration blocks
Suppose you are running an application on your hosts that logs into the /opt/var/myapplication.log file. Create a file (for example, myblocks.conf) that stores a source describing this file and how it should be read:
block source myappsource() {
file("/opt/var/myapplication.log" follow-freq(1) default-facility(syslog)); };
Include this file in your main syslog-ng configuration file, reference the block, and use it in a logpath:
@version: 6.0
@include "<correct/path>/myblocks.conf"
source s_myappsource { myappsource(); };
...
log { source(s_myappsource); destination(...); };To define a block that defines more than one object, use root as the type of the block, and reference the block from the main part of the syslog-ng PE configuration file.
|
|
TIP:
Since the block is inserted into the syslog-ng PE configuration when syslog-ng PE is started, the block can be generated dynamically using an external script if needed. This is useful when you are running syslog-ng PE on different hosts and you want to keep the main configuration identical. If you want to reuse more than a single configuration object, for example, a logpath and the definitions of its sources and destinations, use the include feature to reuse the entire snippet. For details, see the section called “Including configuration files”. |
Configuration blocks can receive arguments as well. The parameters the block can receive must be specified when the block is defined, using the following syntax:
block type block_name(argument1(<default-value-of-the-argument>) argument2(<default-value-of-the-argument>) argument3())
If an argument does not have a default value, add parentheses with empty quotes after the name of the argument, like this: example-option(""). To refer the value of the argument in the block, use the name of the argument between backticks (for example, `argument1`).
Example 5.6. Passing arguments to blocks
The following sample defines a file source block, which can receive the name of the file as a parameter. If no parameter is set, it reads messages from the /var/log/messages file.
block source s_logfile (filename("messages")) {
file("/var/log/`filename`" );
};
source s_example {
s_logfile(filename("logfile.log"));
};If you reference the block with more arguments then specified in its definition, you can use these additional arguments as a single argument-list within the block. That way, you can use a variable number of optional arguments in your block. This can be useful when passing arguments to a template, or optional arguments to an underlying driver. To reference this argument-list, insert `__VARARGS__` to the place in the block where you want to insert the argument-list. Note that you can use this only once in a block. The following definition extends the logfile block from the previous example, and passes the optional arguments (follow-freq(1) flags(no-parse)) to the file() source.
block source s_logfile (filename("messages")) {
file("/var/log/`filename`" `__VARARGS__`);
};
source s_example {
s_logfile(filename("logfile.log") follow-freq(1) flags(no-parse));
};Chapter 6. Collecting log messages — sources and source drivers
A source is where syslog-ng receives log messages. Sources consist of one or more drivers, each defining where and how messages are received.
To define a source, add a source statement to the syslog-ng configuration file using the following syntax:
source <identifier> { source-driver(params); source-driver(params); ... };
Example 6.1. A simple source statement
The following source statement receives messages on the TCP port 1999 of the interface having the 10.1.2.3 IP address.
source s_demo_tcp { network(ip(10.1.2.3) port(1999)); };Example 6.2. A source statement using two source drivers
The following source statement receives messages on the 1999 TCP port and the 1999 UDP port of the interface having the 10.1.2.3 IP address.
source s_demo_two_drivers {
network(ip(10.1.2.3) port(1999));
network(ip(10.1.2.3) port(1999) transport("udp")); };Example 6.3. Setting default priority and facility
If the message received by the source does not have a proper syslog header, you can use the default-facility() and default-priority() options to set the facility and priority of the messages. Note that these values are applied only to messages that do not set these parameters in their header.
source headerless_messages { network(default-facility(syslog) default-priority(emerg)); };Define a source only once. The same source can be used in several log paths. Duplicating sources causes syslog-ng to open the source (TCP/IP port, file, and so on) more than once, which might cause problems. For example, include the /dev/log file source only in one source statement, and use this statement in more than one log path if needed.
|
|
Caution:
Sources and destinations are initialized only when they are used in a log statement. For example, syslog-ng PE starts listening on a port or starts polling a file only if the source is used in a log statement. For details on creating log statements, see Chapter 8, Routing messages: log paths, reliability, and filters. |
To collect log messages on a specific platform, it is important to know how the native syslogd communicates on that platform. The following table summarizes the operation methods of syslogd on some of the tested platforms:
Table 6.1. Communication methods used between the applications and syslogd
Each possible communication mechanism has a corresponding source driver in syslog-ng. For example, to open a unix socket with SOCK_DGRAM style communication use the driver unix-dgram. The same socket using the SOCK_STREAM style — as used under Linux — is called unix-stream.
Example 6.4. Source statement on a Linux based operating system
The following source statement collects the following log messages:
-
internal(): Messages generated by syslog-ng.
-
network(transport("udp")): Messages arriving to the
514/UDPport of any interface of the host. -
unix-dgram("/dev/log");: Messages arriving to the
/dev/logsocket.
source s_demo {
internal();
network(transport("udp"));
unix-dgram("/dev/log"); };The following table lists the source drivers available in syslog-ng.
Table 6.2. Source drivers available in syslog-ng
| Name | Description |
|---|---|
| file() | Opens the specified file and reads messages. |
| internal() | Messages generated internally in syslog-ng. |
| network() | Receives messages from remote hosts using the BSD-syslog protocol over IPv4 and IPv6. Supports the TCP, UDP,RLTP™, and TLS network protocols. |
| pipe() | Opens the specified named pipe and reads messages. |
| program() | Opens the specified application and reads messages from its standard output. |
| sql() | Collects logs from tables of relational database |
| sun-stream(), sun-streams() | Opens the specified STREAMS device on Solaris systems and reads incoming messages. |
| syslog() | Listens for incoming messages using the new IETF-standard syslog protocol. |
| system() | Automatically detects which platform syslog-ng PE is running on, and collects the native log messages of that platform. |
| systemd-journal() | Collects messages directly from the journal of platforms that use systemd. |
| systemd-syslog() | Collects messages from the journal using a socket on platforms that use systemd. |
| unix-dgram() | Opens the specified unix socket in SOCK_DGRAM mode and listens for incoming messages. |
| unix-stream() | Opens the specified unix socket in SOCK_STREAM mode and listens for incoming messages. |
Collecting internal messages
All messages generated internally by syslog-ng use this special source. To collect warnings, errors and notices from syslog-ng itself, include this source in one of your source statements.
internal()
The syslog-ng application will issue a warning upon startup if none of the defined log paths reference this driver.
The syslog-ng PE application sends the following message types from the internal() source:
-
fatal: Priority value: critical (2), Facility value: syslog (5)
-
error: Priority value: error (3), Facility value: syslog (5)
-
warning: Priority value: warning (4), Facility value: syslog (5)
-
notice: Priority value: notice (5), Facility value: syslog (5)
-
info: Priority value: info (6), Facility value: syslog (5)
The internal() driver has the following options:
| Type: | string |
| Default: |
Description: Replaces the ${HOST} part of the message with the parameter string.
| Type: | number (messages) |
| Default: | 1000 |
Description: The size of the initial window, this value is used during flow control. If the max-connections() option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection. For optimal performance when receiving messages from syslog-ng PE clients, make sure that the window size is larger than the flush-lines() option set in the destination of your clients.
Example 6.7. Initial window size of a connection
If log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.
| Accepted values: | yes | no |
| Default: | no |
Description: If enabled (normalize-hostnames(yes)), syslog-ng PE converts the hostnames to lowercase.
| Type: | string |
| Default: |
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
| Type: | string |
| Default: |
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example tags("dmz", "router"). This option is available only in syslog-ng 3.1 and later.