syslog-ng Premium Edition 6.0.21 - Administration Guide

To classify messages using a pattern database, include a db-parser() statement in your syslog-ng configuration file using the following syntax:

Declaration: 

parser <identifier> {db-parser(file("<database_filename>"));};

Note that using the parser in a log statement only performs the classification, but does not automatically do anything with the results of the classification.

parser pattern_db {
            db-parser(
                file("/opt/syslog-ng/var/db/patterndb.xml")
            );
            };

log {
        source(s_all);
        parser(pattern_db);
        destination( di_messages_class);
        };

destination di_messages_class {
        file("/var/log/messages-${.classifier.class}"
        template("${.classifier.rule_id};${S_UNIXTIME};${SOURCEIP};${HOST};${PROGRAM};${PID};${MSG}\n")
        template-escape(no)
    );
};

For details on how to create your own pattern databases see the section called “The syslog-ng pattern database format”.

filter fi_class_violation {
                    match("violation"
                    value(".classifier.class")
                    type("string")
                    );
                    };

log {
                    source(s_all);
                    parser(pattern_db);
                    filter(fi_class_violation);
                    destination(di_class_violation);
                    };

filter fi_class_rule {
                    match("e1e9c0d8-13bb-11de-8293-000c2922ed0a"
                    value(".classifier.rule_id")
                    type("string")
                    );
                    };

filter f_tag_filter {tags(".classifier.system");};

'Transaction: @ESTRING::.@'

'Transaction: @ESTRING:TRANSACTIONTYPE:.@'

match("accepted" value("TRANSACTIONTYPE"));

The syslog-ng PE application is able to correlate log messages identified using pattern databases.

Log messages are supposed to describe events, but applications often separate information about a single event into different log messages. For example, the Postfix e-mail server logs the sender and recipient addresses into separate log messages, or in case of an unsuccessful login attempt, the OpenSSH server sends a log message about the authentication failure, and the reason of the failure in the next message.

Of course, messages that are not so directly related can be correlated as well, for example, login-logout messages, and so on.

To correlate log messages, syslog-ng PE uses the pattern database to add messages into message-groups called contexts. A context consists of a series of log messages that are related to each other in some way, for example, the log messages of an SSH session can belong to the same context. As new messages come in, they may be added to a context. Also, when an incoming message is identified it can trigger actions to be performed, for example, generate a new message that contains all the important information that was stored previously in the context. (For details on triggering actions and generating messages, see the section called “Triggering actions for identified messages”.)

There are two attributes for pattern database rules that determine if a message matching the rule is added to a context: context-scope and context-id. The context-scope attribute acts as an early filter, selecting messages sent by the same process (${HOST}${PROGRAM}${PID} is identical), application (${HOST}${PROGRAM} is identical), or host, while the context-id actually adds the message to the context specified in the id. The context-id can be a simple string, or can contain macros or values extracted from the log messages for further filtering.

Another parameter of a rule is the context-timeout attribute, which determines how long a context is stored, that is, how long syslog-ng PE waits for related messages to arrive. Note the following points about timeout values:

<rule xml:id="..." context-id="ssh-session" context-timeout="86400" context-scope="process">    <patterns>        <pattern>Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern>    </patterns>...
</rule>

For details on configuring message correlation, see the description of the context-id, context-timeout, and context-scope attributes of pattern database rules.

<actions>    <action>        <message>            <values>                <value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 ${DATE} </value>            </values>        </message>    </action></actions>

The syslog-ng PE application is able to generate (trigger) messages automatically if certain events occur, for example, a specific log message is received, or the correlation timeout of a message expires. Basically, you can define messages for every pattern database rule that are emitted when a message matching the rule is received. Triggering messages is often used together with message correlation, but can also be used separately.

The generated message is injected into the same place where the db-parser() statement is referenced in the log path. To post the generated message into the internal() source instead, use the inject-mode() option in the definition of the parser.

parser p_db {db-parser(
    file("mypatterndbfile.xml")
    inject-mode(internal)
);}; 

parser p_db {db-parser(
    file("mypatterndbfile.xml")
    inject-mode(pass-through)
);}; 

The generated message must be configured in the pattern database rule. It is possible to create an entire message, use macros and values extracted from the original message with pattern database, and so on.

<actions>    <action>        <message>            <values>                <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>            </values>        </message>    </action></actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.

<actions>    <action>        <message inherit-properties='TRUE'>            <values>                <value name="PROGRAM">overriding-original-program-name</value>            </values>        </message>    </action></actions>

For details on configuring actions, see the description of the pattern database format.

<action condition="'${HOST}' == 'example'">

<actions>    <action>        <message>            <values>                <value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 ${DATE} </value>            </values>        </message>    </action></actions>

<actions>    <action condition='"$(context-length)" >= "4"'>        <message>            <values>                <value name="PROGRAM">event</value>                <value name="MESSAGE">Event complete</value>            </values>        </message>    </action>    <action condition='"$(context-length)" < "4"'>        <message>            <values>                <value name="PROGRAM">error</value>            <value name="MESSAGE">Error detected</value>            </values>        </message>    </action></actions>