Chat now with support
Chat with Support

syslog-ng Store Box 7.0 LTS - Administration Guide

Preface Introduction The concepts of SSB The Welcome Wizard and the first login Basic settings User management and access control Managing SSB Configuring message sources Storing messages on SSB Forwarding messages from SSB Log paths: routing and processing messages Configuring syslog-ng options Searching log messages Searching the internal messages of SSB Classifying messages with pattern databases The SSB RPC API Monitoring SSB Troubleshooting SSB Security checklist for configuring SSB Glossary

Using pattern parsers

Pattern parsers attempt to parse a part of the message using rules specific to the type of the parser. Parsers are enclosed between @ characters. The syntax of parsers is the following:

  • an opening @ character,

  • the type of the parser written in capitals,

  • optionally a name,

  • parameters of the parser, if any, and

  • a closing @ character.

Example: Pattern parser syntax

A simple parser:

@STRING@

A named parser:

@STRING:myparser_name@

A named parser with a parameter:

@STRING:myparser_name:*@

A parser with a parameter, but without a name:

@STRING::*@

The following parsers are available:

  • @ANYSTRING@: Parses everything to the end of the message. You can use it to collect everything that is not parsed specifically to a single macro. In that sense its behavior is similar to the greedy() option of the CSV parser.

  • @DOUBLE@: An obsolete alias of the @FLOAT@ parser.

  • @ESTRING@: This parser has a required parameter that acts as the stopcharacter: the parser parses everything until it find the stopcharacter. For example to stop by the next " (double quote) character, use @ESTRING::"@. As of syslog-ng 3.1, it is possible to specify a stopstring instead of a single character, for example @ESTRING::stop_here.@.

  • @FLOAT@: A floating-point number that may contain a dot (.) character. (Up to syslog-ng 3.1, the name of this parser was @DOUBLE@.)

  • @IPv4@: Parses an IPv4 IP address (numbers separated with a maximum of 3 dots).

  • @IPv6@: Parses any valid IPv6 IP address.

  • @IPvANY@: Parses any IP address.

  • @NUMBER@: A sequence of decimal (0-9) numbers (for example 1, 0687, and so on). Note that if the number starts with the 0x characters, it is parsed as a hexadecimal number, but only if at least one valid character follows 0x.

  • @QSTRING@: Parse a string between the quote characters specified as parameter. Note that the quote character can be different at the beginning and the end of the quote, for example: @QSTRING::"@ parses everything between two quotation marks ("), while @QSTRING:<>@ parses from an opening bracket to the closing bracket.

  • @STRING@: A sequence of alphanumeric characters (0-9, A-z), not including any whitespace. Optionally, other accepted characters can be listed as parameters (for example to parse a complete sentence, add the whitespace as parameter, like: @STRING:: @). Note that the @ character cannot be a parameter, nor can line-breaks or tabs.

Patterns and literals can be mixed together. For example, to parse a message that begins with the Host: string followed by an IP address (for example Host: 192.168.1.1), the following pattern can be used: Host:@IPv4@.

NOTE: Note that using parsers is a CPU-intensive operation. Use the ESTRING and QSTRING parsers whenever possible, as these can be processed much faster than the other parsers.

Example: Using the STRING and ESTRING parsers

For example, if the message is user=joe96 group=somegroup, @STRING:mytext:@ parses only to the first non-alphanumeric character (=), parsing only user. @STRING:mytext:=@ parses the equation mark as well, and proceeds to the next non-alphanumeric character (the whitespace), resulting in user=joe96 being parsed. @STRING:mytext:= @ will parse the whitespace as well, and proceed to the next non-alphanumeric non-equation mark non-whitespace character, resulting in user=joe96 group=somegroup.

Of course, usually it is better to parse the different values separately, like this: "user=@STRING:user@ group=@STRING:group@".

If the username or the group may contain non-alphanumeric characters, you can either include these in the second parameter of the parser (as shown at the beginning of this example), or use an ESTRING parser to parse the message till the next whitespace: "user=@ESTRING:user: @group=@ESTRING:group: @".

Example: Patterns for multiline messages

Patterns can be created for multiline log messages. For example, the following pattern will find the multiline message where a line ends with first and the next line starts with second:

first
second

Using parser results in filters and templates

The results of message classification and parsing can be used in custom filters and file and database templates as well. There are two built-in macros in syslog-ng Store Box(SSB) that allow you to use the results of the classification: the .classifier.class macro contains the class assigned to the message (for example violation, security, or unknown), while the .classifier.rule_id macro contains the identifier of the message pattern that matched the message.

NOTE: ID of the message pattern is automatically inserted into the template if the messages are forwarded to an SQL database.

The following describes how to use these macros as filters in a log path.

To use macros as filters in a log path

  1. Navigate to Log > Paths and select the log path to use.

  2. To filter on a specific message class, select Add filter > classifier_class, select , then select the class to match (for example Violation) from the classifier_class field.

    Figure 239: Log > Paths — Filtering messages based on the classification

  3. To filter on messages matching a specific classification rule, Add filter > classifier_rule_id, select , then enter the unique identifier of the rule (for example e1e9c0d8-13bb-11de-8293-000c2922ed0a) into the classifier_rule_id field.

    NOTE: To filter messages based on other classification data like tags, you have to use Custom filters. For details, see Filtering messages.

  4. Click .

Using the values of pattern parsers in filters and templates

Similarly to Using parser results in filters and templates, the results of pattern parsers can be used as well. To accomplish this, you have to add a name to the parser, and then you can use this name as a macro that refers to the parsed value of the message.

For example, you want to parse messages of an application that look like "Transaction: <type>.", where <type> is a string that has different values (for example refused, accepted, incomplete, and so on). To parse these messages, you can use the following pattern:

'Transaction: @ESTRING::.@'

Here the @ESTRING@ parser parses the message until the next full stop character. To use the results in a filter or a filename template, include a name in the parser of the pattern, for example:

'Transaction:
            @ESTRING:TRANSACTIONTYPE:.@'

After that, add a custom template to the logpath that uses this template. For example, to select every accepted transaction, use the following custom filter in the log path:

match("accepted" value("TRANSACTIONTYPE"));

NOTE: The above macros can be used in database columns and filename templates as well, if you create custom templates for the destination or logspace.

The SSB RPC API

The syslog-ng Store Box(SSB) appliance can be accessed using a Remote-Procedure Call Application Programming Interface (RPC API).

The SSB RPC API allows you to access and query SSB logspaces from remote applications. You can access the API using a RESTful protocol over HTTPS, meaning that you can use any programming language that has access to a RESTful HTTPS client to integrate SSB to your environment. Sample shell code snippets are provided in the API documentation.

Accessing SSB with the RPC API offers several advantages:

  • Integration into custom applications and environments

  • Flexible, dynamic search queries

SSB prevents brute force attacks when logging in. If you repeatedly try logging in to SSB using incorrect login details within a short period of time (10 times within 60 seconds), the source IP gets blocked for 5 minutes.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating