syslog-ng Premium Edition 6.0.23 - Administration Guide

Using pattern parsers

Pattern parsers attempt to parse a part of the message using rules specific to the type of the parser. Parsers are enclosed between @ characters. The syntax of parsers is the following:

a beginning @ character,
the type of the parser written in capitals,
optionally a name,
parameters of the parser, if any, and
a closing @ character.

Example 16.16. Pattern parser syntax

A simple parser:

@STRING@

A named parser:

@STRING:myparser_name@

A named parser with a parameter:

@STRING:myparser_name:*@

A parser with a parameter, but without a name:

@STRING::*@

Patterns and literals can be mixed together. For example, to parse a message that begins with the Host: string followed by an IP address (for example, Host: 192.168.1.1), the following pattern can be used: Host:@IPv4@.

NOTE:

Note that using parsers is a CPU-intensive operation. Use the ESTRING and QSTRING parsers whenever possible, as these can be processed much faster than the other parsers.

Example 16.17. Using the STRING and ESTRING parsers

For example, if the message is user=joe96 group=somegroup, @STRING:mytext:@ parses only to the first non-alphanumeric character (=), parsing only user. @STRING:mytext:=@ parses the equation mark as well, and proceeds to the next non-alphanumeric character (the whitespace), resulting in user=joe96 being parsed. @STRING:mytext:= @ will parse the whitespace as well, and proceed to the next non-alphanumeric non-equation mark non-whitespace character, resulting in user=joe96 group=somegroup.

Of course, usually it is better to parse the different values separately, like this: "user=@STRING:user@ group=@STRING:group@".

If the username or the group may contain non-alphanumeric characters, you can either include these in the second parameter of the parser (as shown at the beginning of this example), or use an ESTRING parser to parse the message till the next whitespace: "user=@ESTRING:user: @group=@ESTRING:group: @".

Pattern parsers of syslog-ng PE

The following parsers are available in syslog-ng PE.

@ANYSTRING@

Parses everything to the end of the message, you can use it to collect everything that is not parsed specifically to a single macro. In that sense its behavior is similar to the greedy() option of the CSV parser.

@DOUBLE@

An obsolete alias of the @FLOAT@ parser.

@ESTRING@

This parser has a required parameter that acts as the stopcharacter: the parser parses everything until it finds the stopcharacter. For example, to stop by the next " (double quote) character, use @ESTRING::"@. You can use the colon (:) as stopcharacter as well, for example: @ESTRING:::@. You can also specify a stopstring instead of a single character, for example, @ESTRING::stop_here.@. The @ character cannot be a stopcharacter, nor can line-breaks or tabs.

@FLOAT@

A floating-point number that may contain a dot (.) character. (Up to syslog-ng 3.1, the name of this parser was @DOUBLE@.)

@IPv4@

Parses an IPv4 IP address (numbers separated with a maximum of 3 dots).

@IPv6@

Parses any valid IPv6 IP address.

@IPvANY@

Parses any IP address.

@NUMBER@

A sequence of decimal (0-9) numbers (for example, 1, 0687, and so on). Note that if the number starts with the 0x characters, it is parsed as a hexadecimal number, but only if at least one valid character follows 0x. A leading hyphen (–) is accepted for non-hexadecimal numbers, but other separator characters (for example, dot or comma) are not. To parse floating-point numbers, use the @FLOAT@ parser.

@QSTRING@

Parse a string between the quote characters specified as parameter. Note that the quote character can be different at the beginning and the end of the quote, for example: @QSTRING::"@ parses everything between two quotation marks ("), while @QSTRING:<>@ parses from an opening bracket to the closing bracket. The @ character cannot be a quote character, nor can line-breaks or tabs.

@STRING@

A sequence of alphanumeric characters (0-9, A-z), not including any whitespace. Optionally, other accepted characters can be listed as parameters (for example, to parse a complete sentence, add the whitespace as parameter, like: @STRING:: @). Note that the @ character cannot be a parameter, nor can line-breaks or tabs.

The syslog-ng pattern database format

Pattern databases are XML files that contain rules describing the message patterns. For sample pattern databases, see the section called “Downloading sample pattern databases”.

The following scheme describes the V4 format of the pattern database. This format is used by syslog-ng PE 4 F1 and later, and is backwards-compatible with the earlier V3 format.

For a sample database containing only a single pattern, see Example 16.18, “A V4 pattern database containing a single rule”.

TIP:

Use the pdbtool utility that is bundled with syslog-ng to test message patterns and convert existing databases to the latest format. For details, see pdbtool(1).

To automatically create an initial pattern database from an existing log file, use the pdbtool patternize command. For details, see the section called “The patternize command”.

Example 16.18. A V4 pattern database containing a single rule

The following pattern database contains a single rule that matches a log message of the ssh application. A sample log message looks like:

Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2

The following is a simple pattern database containing a matching rule.

<patterndb version='4' pub_date='2010-10-17'>    <ruleset name='ssh' id='123456678'>        <pattern>ssh</pattern>            <rules>                <rule provider='me' id='182437592347598' class='system'>                    <patterns>                        <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>                    </patterns>                </rule>            </rules>    </ruleset></patterndb>

Note that the rule uses macros that refer to parts of the message, for example, you can use the ${SSH_USERNAME} macro refer to the username used in the connection.

The following is the same example, but with a test message and test values for the parsers.

<patterndb version='4' pub_date='2010-10-17'>    <ruleset name='ssh' id='123456678'>        <pattern>ssh</pattern>            <rules>                <rule provider='me' id='182437592347598' class='system'>                    <patterns>                        <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>                    </patterns>                    <examples>                        <example>                            <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>                            <test_values>                                <test_value name="SSH.AUTH_METHOD">password</test_value>                                <test_value name="SSH_USERNAME">sampleuser</test_value>                                <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>                                <test_value name="SSH_PORT_NUMBER">42156</test_value>                            </test_values>                       </example>                    </examples>                </rule>            </rules>    </ruleset></patterndb>

Element: patterndb

Location

/patterndb

Description

The container element of the pattern database.

Attributes

version: The schema version of the pattern database. The current version is 4.
pubdate: The publication date of the XML file.

Children

ruleset

Example

<patterndb version='4' pub_date='2010-10-25'>

Element: ruleset

Location

/patterndb/ruleset

Description

A container element to group log patterns for an application or program. A <patterndb> element may contain any number of <ruleset> elements.

Attributes

name: The name of the application. Note that the function of this attribute is to make the database more readable, syslog-ng uses the <pattern> element to identify the applications sending log messages.
id: A unique ID of the application, for example, the md5 sum of the name attribute.
description: OPTIONAL — A description of the ruleset or the application.
url: OPTIONAL — An URL referring to further information about the ruleset or the application.

Example

<ruleset name='su' id='480de478-d4a6-4a7f-bea4-0c0245d361e1'>

Element: patterns

Location

/patterndb/ruleset/patterns

Description

A container element. A <patterns> element may contain any number of <pattern> elements.

Attributes

N/A

Children

pattern: The name of the application — syslog-ng matches this value to the ${PROGRAM} header of the syslog message to find the rulesets applicable to the syslog message.

Specifying multiple patterns is useful if two or more applications have different names (that is, different ${PROGRAM} fields), but otherwise send identical log messages.

It is not necessary to use multiple patterns if only the end of the ${PROGRAM} fields is different, use only the beginning of the ${PROGRAM} field as the pattern. For example, the Postfix e-mail server sends messages using different process names, but all of them begin with the postfix string.

NOTE:

If the <pattern> element of a ruleset is not specified, syslog-ng PE will use this ruleset as a fallback ruleset: it will apply the ruleset to messages that have an empty PROGRAM header, or if none of the program patterns matched the PROGRAM header of the incoming message.

Example

<patterns>    <pattern>firstapplication</pattern>    <pattern>otherapplication</pattern></patterns>

Element: rules

Location

/patterndb/ruleset/rules

Description

A container element for the rules of the ruleset.

Attributes

N/A

Children

rule

Example

<rules>    <rule provider='me' id='182437592347598' class='system'>        <patterns>            <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>        </patterns>    </rule></rules>

Element: rule

Location

/patterndb/ruleset/rules/rule

Description

An element containing message patterns and how a message that matches these patterns is classified.

NOTE:

If the following characters appear in the message, they must be escaped in the rule as follows:

@: Use @@, for example user@@example.com
<: Use <
>: Use >
&: Use &

The <rules> element may contain any number of <rule> elements.

Attributes

provider: The provider of the rule. This is used to distinguish between who supplied the rule, that is, if it has been created by One Identity, or added to the XML by a local user.
id: The globally unique ID of the rule.
class: The class of the rule — syslog-ng assigns this class to the messages matching a pattern of this rule.

context-id: OPTIONAL — An identifier to group related log messages when using the pattern database to correlate events. The ID can be a descriptive string describing the events related to the log message (for example, ssh-sessions for log messages related to SSH traffic), but can also contain macros to generate IDs dynamically. When using macros in IDs, see also the context-scope attribute. For details on correlating messages, see the section called “Correlating log messages”.

NOTE:

The syslog-ng PE application determines the context of the message after the pattern matching is completed. This means that macros and name-value pairs created by the matching pattern database rule can be used as context-id macros.

context-timeout: OPTIONAL — The number of seconds the context is stored. Note that for high-traffic log servers, storing open contexts for long time can require significant amount of memory. For details on correlating messages, see the section called “Correlating log messages”.
context-scope: OPTIONAL — Specifies which messages belong to the same context. This attribute is used to determine the context of the message if the context-id does not specify any macros. Usually, context-scope acts a filter for the context, with context-id refining the filtering if needed. The context-scope attribute has the following possible values:
- process: Only messages that are generated by the same process of a client belong to the same context, that is, messages that have identical ${HOST}, ${PROGRAM} and ${PID} values. This is the default behavior of syslog-ng PE if context-scope is not specified.
- program: Messages that are generated by the same application of a client belong to the same context, that is, messages that have identical ${HOST} and ${PROGRAM} values.
- host: Every message generated by a client belongs to the same context, only the ${HOST} value of the messages must be identical.
- global: Every message belongs to the same context.
NOTE:
Using the context-scope attribute is significantly faster than using macros in the context-id attribute.

For details on correlating messages, see the section called “Correlating log messages”.

Children

patterns

Example

<rule provider='balabit' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation'>

The following example specifies attributes for correlating messages as well. For details on correlating messages, see the section called “Correlating log messages”.

<rule provider='balabit' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation' context-id='same-session' context-scope='process' context-timeout='360'>

Element: patterns

Location

/patterndb/ruleset/rules/rule/patterns

Description

An element containing the patterns of the rule. If a <patterns> element contains multiple <pattern> elements, the class of the <rule> is assigned to every syslog message matching any of the patterns.

Attributes

N/A

Children

pattern: A pattern describing a log message. This element is also called message pattern. For example:
```
<pattern>+ ??? root-</pattern>
```
NOTE:
Support for XML entities is limited, you can use only the following entities: & < > " '. User-defined entities are not supported.
description: OPTIONAL — A description of the pattern or the log message matching the pattern.
urls
values
examples

Example

<patterns>    <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern></patterns>

Element: urls

Location

/patterndb/ruleset/rules/rule/patterns/urls

Description

OPTIONAL — An element containing one or more URLs referring to further information about the patterns or the matching log messages.

Attributes

N/A

Children

url: OPTIONAL — An URL referring to further information about the patterns or the matching log messages.

Example

N/A

Element: values

Location

/patterndb/ruleset/rules/rule/patterns/values

Description

OPTIONAL — Name-value pairs that are assigned to messages matching the patterns, for example, the representation of the event in the message according to the Common Event Format (CEF) or Common Event Exchange (CEE). The names can be used as macros to reference the assigned values.

Attributes

N/A

Children

value: OPTIONAL — Contains the value of the name-value pair that is assigned to the message.

The <value> element of name-value pairs can include template functions. For details, see the section called “Using template functions”, for examples, see the section called “if”.

When used together with message correlation, the <value> element of name-value pairs can include references to the values of earlier messages from the same context. For details, see the section called “Correlating log messages”.
name: The name of the name-value pair. It can also be used as a macro to reference the assigned value.

Example

<values>    <value name=".classifier.outcome">/Success</value></values>

Element: examples

Location

/patterndb/ruleset/rules/rule/patterns/examples

Description

OPTIONAL — A container element for sample log messages that should be recognized by the pattern. These messages can be used also to test the patterns and the parsers.

Attributes

N/A

Children

example

Example

<examples>    <example>        <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>        <test_values>            <test_value name="SSH.AUTH_METHOD">password</test_value>            <test_value name="SSH_USERNAME">sampleuser</test_value>            <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>            <test_value name="SSH_PORT_NUMBER">42156</test_value>        </test_values>    </example></examples>

Element: example

Location

/patterndb/ruleset/rules/rule/patterns/examples/example

Description

OPTIONAL — A container element for a sample log message.

Attributes

N/A

Children

test_message: OPTIONAL — A sample log message that should match this pattern. For example:

<test_message program="myapplication">Content filter has been enabled</test_message>

program: The program pattern of the test message. For example:

<test_message program="proftpd">ubuntu (::ffff:192.168.2.179[::ffff:192.168.2.179]) - FTP session closed.</test_message>

test_values: OPTIONAL — A container element to test the results of the parsers used in the pattern.
- test_value: OPTIONAL — The expected value of the parser when matching the pattern to the test message. For example:
```
<test_value name=".dict.ContentFilter">enabled</test_value>
```
  - name: The name of the parser to test.

Example

<examples>    <example>        <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>        <test_values>            <test_value name="SSH.AUTH_METHOD">password</test_value>            <test_value name="SSH_USERNAME">sampleuser</test_value>            <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>            <test_value name="SSH_PORT_NUMBER">42156</test_value>        </test_values>    </example></examples>

Element: actions

Location

/patterndb/ruleset/actions

Description

OPTIONAL — A container element for actions that are performed if a message is recognized by the pattern. For details on actions, see the section called “Triggering actions for identified messages”.

Attributes

N/A

Children

action

Example

Example 16.19. Generating messages for pattern database matches

When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.

<actions>    <action>        <message>            <values>                <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>            </values>        </message>    </action></actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.

Example 16.20. Generating messages with inherited values

The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name

<actions>    <action>        <message inherit-properties='TRUE'>            <values>                <value name="PROGRAM">overriding-original-program-name</value>            </values>        </message>    </action></actions>

Element: action

Location

/patterndb/ruleset/actions/action

Description

OPTIONAL — A container element describing an action that is performed when a message matching the rule is received.

Attributes

condition: A syslog-ng filter expression. The action is performed only if the message matches the filter. The filter can include macros and name-value pairs extracted from the message.
rate: Specifies maximum how many messages should be generated in the specified time period in the following format: <number-of-messages>/<period-in-seconds>. For example: 1/60 allows 1 message per minute. Rates apply within the scope of the context, that is, if context-scope="host" and rate="1/60", then maximum one message is generated per minute for every host that sends a log message matching the rule. Excess messages are dropped. Note that when applying the rate to the generated messages, syslog-ng PE uses the timestamps of the log messages, similarly to calculating the context-timeout. That way rate is applied correctly even if the log messages are processed offline.
trigger: Specifies when the action is executed. The trigger attribute has the following possible values:
- match: Execute the action immediately when a message matching the rule is received.
- timeout: Execute the action when the correlation timer (context-timeout) expires. This is available only if actions are used together with correlating messages.

Children

message: A container element storing the message to be sent when the action is executed. Currently syslog-ng PE sends these messages to the internal() destination.
- inherit-properties: If set to TRUE, the original message that triggered the action is cloned, including its name-value pairs and tags. For details, see the section called “Triggering actions for identified messages”.
  
  If set to context, syslog-ng PE collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context. For details on the message context, see the section called “Correlating log messages” and the section called “Actions and message correlation”.
  
  This option is available in syslog-ng PE 5.3.2 and later.
values: A container element for values and fields that are used to create the message generated by the action.
- value: Sets the value of the message field specified in the name attribute of the element. For example, to specify the body of the generated message, use the following syntax:
```
<value name="MESSAGE">A log message matched rule number $.classifier.rule_id</value>
```
  Note that currently it is not possible to add DATE, FACILITY, or SEVERITY fields to the message.
  
  When the action is used together with message correlation, the syslog-ng PE application automatically adds fields to the message based on the context-scope parameter. For example, using context-scope="process" automatically fills the HOST, PROGRAM, and PID fields of the generated message.
- name: Name of the message field set by the value element.

Example

Example 16.21. Generating messages for pattern database matches

When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.

<actions>    <action>        <message>            <values>                <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>            </values>        </message>    </action></actions>

Example 16.22. Generating messages with inherited values

The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name

<actions>    <action>        <message inherit-properties='TRUE'>            <values>                <value name="PROGRAM">overriding-original-program-name</value>            </values>        </message>    </action></actions>

Element: tags

Location

/patterndb/ruleset/tags

Description

OPTIONAL — An element containing custom keywords (tags) about the messages matching the patterns. The tags can be used to label specific events (for example user logons). It is also possible to filter on these tags later (for details, see the section called “Tagging messages”). Starting with syslog-ng Premium Edition 3.2, the list of tags assigned to a message can be referenced with the ${TAGS} macro.

Attributes

N/A

Children

tag: OPTIONAL — A keyword or tags applied to messages matching the rule.

Example

<tags><tag>UserLogin</tag></tags>

Chapter 17. Statistics and metrics of syslog-ng

Periodically, syslog-ng sends a message containing statistics about the received messages, and about any lost messages since the last such message. It includes a processed entry for every source and destination, listing the number of messages received or sent, and a dropped entry including the IP address of the server for every destination where syslog-ng has lost messages. The center(received) entry shows the total number of messages received from every configured sources.

The following is a sample log statistics message for a configuration that has a single source (s_local) and a network and a local file destination (d_network and d_local, respectively). All incoming messages are sent to both destinations.

Log statistics;
        dropped='tcp(AF_INET(192.168.10.1:514))=6439',
        processed='center(received)=234413',
        processed='destination(d_tcp)=234413',
        processed='destination(d_local)=234413',
        processed='source(s_local)=234413'

Log statistics can be also retrieved on-demand using one of the following options:

Using the syslog-ng-ctl stats command.
Using the syslog-ng-query sum "*" command. Note that the output of the syslog-ng-query command is better structured, and also allows you to select which metrics you want to display. For details, see syslog-ng-query(1).
Use the socat application: echo STATS | socat -vv UNIX-CONNECT:/opt/syslog-ng/var/run/syslog-ng.ctl -
If you have an OpenBSD-style netcat application installed, use the echo STATS | nc -U /opt/syslog-ng/var/run/syslog-ng.ctl command. Note that the netcat included in most Linux distributions is a GNU-style version that is not suitable to query the statistics of syslog-ng.

The statistics include a list of source groups and destinations, as well as the number of processed messages for each. The verbosity of the statistics can be set using the stats-level() option. For details, see the section called “Global options”. An example output is shown below.

src.internal;s_all#0;;a;processed;6445
src.internal;s_all#0;;a;stamp;1268989330
destination;df_auth;;a;processed;404
destination;df_news_dot_notice;;a;processed;0
destination;df_news_dot_err;;a;processed;0
destination;d_ssb;;a;processed;7128
destination;df_uucp;;a;processed;0
source;s_all;;a;processed;7128
destination;df_mail;;a;processed;0
destination;df_user;;a;processed;1
destination;df_daemon;;a;processed;1
destination;df_debug;;a;processed;15
destination;df_messages;;a;processed;54
destination;dp_xconsole;;a;processed;671
dst.tcp;d_network#0;10.50.0.111:514;a;dropped;5080
dst.tcp;d_network#0;10.50.0.111:514;a;processed;7128
dst.tcp;d_network#0;10.50.0.111:514;a;stored;2048
destination;df_syslog;;a;processed;6724
destination;df_facility_dot_warn;;a;processed;0
destination;df_news_dot_crit;;a;processed;0
destination;df_lpr;;a;processed;0
destination;du_all;;a;processed;0
destination;df_facility_dot_info;;a;processed;0
center;;received;a;processed;0
destination;df_kern;;a;processed;70
center;;queued;a;processed;0
destination;df_facility_dot_err;;a;processed;0

The statistics are semicolon separated: every line contains statistics for a particular object (for example source, destination, tag, and so on). The statistics have the following fields:

The type of the object (for example dst.file, tag, src.facility)
The ID of the object used in the syslog-ng configuration file, for example d_internal or source.src_tcp. The #0 part means that this is the first destination in the destination group.
The instance ID (destination) of the object, for example the filename of a file destination, or the name of the application for a program source or destination.

The status of the object. One of the following:

a - active. At the time of quering the statistics, the source or the destination was still alive (it continuously received statistical data).
d - dynamic. Such objects may not be continuously available, for example, like statistics based on the sender's hostname.

o - This object was once active, but stopped receiving messages. (For example a dynamic object may disappear and become orphan.)

NOTE:

The syslog-ng PE application stores the statistics of the objects when syslog-ng PE is reloaded. However, if the configuration of syslog-ng PE was changed since the last reload, the statistics of orphaned objects are deleted.

The type of the statistics:
- processed: The number of messages that successfully reached their destination driver. Note that this does not necessarily mean that the destination driver successfully delivered the messages (for example, written to disk or sent to a remote server).
- dropped: The number of dropped messages — syslog-ng PE could not send the messages to the destination and the output buffer got full, so messages were dropped by the destination driver.
- stored: The number of messages stored in the message queue of the destination driver, waiting to be sent to the destination.
- suppressed: The number of suppressed messages (if the suppress() feature is enabled).
- stamp: The UNIX timestamp of the last message sent to the destination.
The number of such messages.

NOTE:

Certain statistics are available only if the stats-level() option is set to a higher value.

When receiving messages with non-standard facility values (that is, higher than 23), these messages will be listed as other facility instead of their facility number.

To reset the statistics to zero, use the following command: syslog-ng-ctl stats --reset

Chapter 18. Multithreading and scaling in syslog-ng PE

Starting with version 4 F1, syslog-ng PE can process sources and destinations in multithreaded mode to scale to multiple CPUs or cores for increased performance. Starting with version 5 F4, this multithreaded mode is the default.

Multithreading concepts of syslog-ng PE

This section is a brief overview on how syslog-ng PE works in multithreaded mode. It is mainly for illustration purposes: the concept has been somewhat simplified and may not completely match reality.

NOTE:

The way syslog-ng PE uses multithreading may change in future releases. The current documentation applies to version 6 LTS.

syslog-ng PE always uses multiple threads:

A main thread that is always running
A number of worker threads that process the messages. You can influence the behavior of worker threads using the threaded() option and the --worker-threads command-line option.
Some other, special threads for internal functionalities. For example, certain destinations run in a separate thread, independently of the multithreading (threaded()) and --worker-threads settings of syslog-ng PE.

The maximum number of worker threads syslog-ng PE uses is the number of CPUs or cores in the host running syslog-ng PE (up to 64). You can limit this value using the --worker-threads command-line option that sets the maximum total number of threads syslog-ng PE can use, including the main syslog-ng PE thread. However, the --worker-threads option does not affect the supervisor of syslog-ng PE. The supervisor is a separate process (see syslog-ng(8)), but certain operating systems might display it as a thread. In addition, certain destinations always run in a separate thread, independently of the multithreading (threaded()) and --worker-threads settings of syslog-ng PE.

When an event requiring a new thread occurs (for example, syslog-ng PE receives new messages, or a destination becomes available), syslog-ng PE tries to start a new thread. If there are no free threads, the task waits until a thread finishes its task and becomes available. There are two types of worker threads:

Reader threads read messages from a source (as many as possible, but limited by the log-fetch-limit() and log-iw-size() options. The thread then processes these messages, that is, performs filtering, rewriting and other tasks as necessary, and puts the log message into the queue of the destination. If the destination does not have a queue (for example, usertty), the reader thread sends the message to the destination, without the interaction of a separate writer thread.
Writer threads take the messages from the queue of the destination and send them to the destination, that is, write the messages into a file, or send them to the syslog server over the network. The writer thread starts to process messages from the queue only if the destination is writable, and there are enough messages in the queue, as set in the flush-lines() and the flush-timeout() options. Writer threads stop processing messages when the destination becomes unavailable, or there are no more messages in the queue.

Sources and destinations affected by multithreading. The following list describes which sources and destinations can use multiple threads. Changing the --worker-threads command-line option changes the number of threads available to these sources and destinations.

The tcp and syslog(tcp) sources can process independent connections in separate threads. The number of independent connections is limited by the max-connections() option of the source. Separate sources are processed by separate thread, for example, if you have two separate tcp sources defined that receive messages on different IP addresses or port, syslog-ng PE will use separate threads for these sources even if they both have only a single active connection.
The udp, file, and pipe sources use a single thread for every source statement.
The tcp, syslog, and pipe destinations use a single thread for every destination.
The file destination uses a single thread for writing the destination file, but may use a separate thread for each destination file if the filename includes macros.

Sources and destinations not affected by multithreading. The following list describes sources and destinations that use a separate thread even if you disable multithreading in syslog-ng PE, in addition to the limit set in the --worker-threads command-line option.

The logstore destination uses separate threads for writing the messages from the journal to the logstore files, and also for timestamping. These threads are independent from the setting of the --worker-threads command-line option.
Every sql destination uses its own thread. These threads are independent from the setting of the --worker-threads command-line option.
The java destinations use one thread, even if there are multiple Java-based destinations configured. This thread is independent from the setting of the --worker-threads command-line option.

Please select your product:

To serve you better, please complete the Purpose of your Chat:

Recommended Solutions for Your Problem

syslog-ng Premium Edition 6.0.23 - Administration Guide

Creating pattern databases

Using pattern parsers

Pattern parsers of syslog-ng PE

@ANYSTRING@

@DOUBLE@

@ESTRING@

@FLOAT@

@IPv4@

@IPv6@

@IPvANY@

@NUMBER@

@QSTRING@

@STRING@

The syslog-ng pattern database format

Element: patterndb

Location

Description

Attributes

Children

Example

Element: ruleset

Location

Description

Attributes

Children

Example

Element: patterns

Location

Description

Attributes

Children

Example

Element: rules

Location

Description

Attributes

Children

Example

Element: rule

Location

Description

Attributes

Children

Example

Element: patterns

Location

Description

Attributes

Children

Example

Element: urls

Location

Description

Attributes

Children

Example

Element: values

Location

Description

Attributes

Children

Example

Element: examples

Location

Description

Attributes

Children

Example

Element: example

Location

Description

Attributes

Children

Example

Element: actions

Location