Chat now with support
Chat with Support

syslog-ng Store Box 7.2.0 - Administration Guide

Preface Introduction The concepts of SSB The Welcome Wizard and the first login Basic settings User management and access control Managing SSB Configuring message sources Storing messages on SSB Forwarding messages from SSB Log paths: routing and processing messages Configuring syslog-ng options Searching log messages Searching the internal messages of SSB Classifying messages with pattern databases The SSB RPC API Monitoring SSB Troubleshooting SSB Security checklist for configuring SSB Glossary

Filtering messages

This section describes the filters that can be used in log paths. Every filter can be used to select (for example, priority is) or exclude (for example, priority is not) messages. The following filters are available:

  • facility: Select messages sent by a specific facility (for example, kernel).

  • host: Select messages sent by a specific host. Enter the a hostname, IP address, or a POSIX (extended) regular expression.

  • message: Select messages containing a specific keyword or POSIX (extended) regular expression in the text of the log message (excluding the headers).

  • priority: Select messages of a specific priority.

  • program: Select messages sent by a specific application. Enter the name of the application or a POSIX (extended) regular expression.

  • sender: Filter on the address of the host that sent the message to syslog-ng Store Box(SSB).

    NOTE: To be able to use this filter, as a prerequisite, you must have a hostlist defined. For more information, see Creating hostlist policies.

    NOTE: When using the host, message, and program filters, remember to escape special characters. The characters ()[]{}.*?+^$|\ are treated as special symbols and have to be escaped with a backslash (\) in order to be interpreted as literal characters.

    NOTE: The effect of the sender and the host filters is the same if every client sends the logs directly to SSB. But if SSB receives messages from relays, then the host filter applies to the address of the clients, while the sender applies to the address of the relays.

If multiple filters are set for a log path, only messages complying to every filter are sent to the destinations. (In other words, filters are added using the logical AND operation.)

Figure 195: Log > Paths — Using custom filters

If you need more complex filtering in your log path, select the of the log path and enter a custom filter into the appearing field. The contents of the Custom filter field are pasted into the filter() parameter of the syslog-ng log path definition.

When defining custom filters, you can use regular expressions. By default, custom filters use POSIX-style (extended) regular expressions.

NOTE: When using POSIX regular expressions, the characters ()[]{}.*?+^$|\ are used as special symbols. Depending on how you want to use these characters and which quotation mark you use, these characters must be used differently, as summarized below:

  • When enclosing strings between double-quotes ("string"), the string is interpreted and you have to escape special characters, that is, prefix them with a backslash (\) if they are meant literally.

  • Strings between single quotes ('string') are treated as literals and are not interpreted at all, so you do not have to escape special characters.

To use other expression types, add the type() option after the regular expression. For example:

message("([0-9]+)=\\1" type("pcre"))

In this example, a PCRE regular expression with backreference is used and a match is returned if the message contains identical numbers separated by the equal sign (=). For example:

123=123

Replace message parts or create new macros with rewrite rules

The syslog-ng Store Box(SSB) appliance can rewrite parts of the messages using rewrite rules. Almost all part (macro) of the message can be rewritten. The rules use a key-value pair format.

The Replace with value completely replaces the old value of the message part. If the message part does not already exist, SSB automatically creates it. If you want to perform search and replace in the text of the log message, see Find and replace the text of the log message instead.

NOTE: You cannot change the values of hard macros in rewrite rules. For the list of hard macros, see Hard versus soft macros in the syslog-ng PE Administration Guide.

To replace message parts or create new macros with rewrite rules

  1. Navigate to Log > Paths.

  2. Select the path(s) where you want to use rewrite rules.

  3. In the Rewrites section, click to add a new rewrite rule. Rewrite rules can be applied before filtering, or after filtering.

    The sequence of filtering and rewrite rules depends on how it was specified in the log path. The sequence of the process is the following:

    1. Parse the message as a syslog message (unless message parsing is explicitly disabled for the source).

    2. Classify the message using a pattern database.

    3. Modify the message using rewrite rules (before filtering).

    4. Filter the messages, for example, based on sender hostname or message content. If the message does not match the configured filter, syslog-ng Store Box(SSB) will not send it to the destination.

    5. Parse the text of the message (that is, the ${MESSAGE} part) using a key-value parser or the sudo parser.

    6. Modify the message using rewrite rules (after filtering and other parsing).

    7. SSB sends the message to the destinations set in the log path. The destinations are local, optionally encrypted files on SSB, or remote servers, such as a database server.

    Figure 196: Log > Paths — Modifying messages using rewrite

  4. Enter the part of the message to rewrite into the In Message part field. For example, MESSAGE, HOST, .SDATA.meta.custom. If the specified field does not exist, it is automatically created and set to the Replace with field.

  5. Enter the value of the message part after rewriting into the Replace with field. To use macros, begin with a $ sign and enclose the name of the macro between braces, for example ${MSG}, ${.SDATA.meta.custom}.

    NOTE:

    • The replacement value completely replaces the old value of the message part.

    • You cannot change the values of hard macros in rewrite rules. For the list of hard macros, see Hard versus soft macros in the syslog-ng PE Administration Guide.

  6. Click .

Find and replace the text of the log message

You can perform search and replace operations on the log messages to rewrite the messages. Almost all part (macro) of the message can be rewritten. You can use PCRE regular expressions.

If you want to completely replace a message part, or create a new one that does not already exist, see Replace message parts or create new macros with rewrite rules instead.

NOTE: You cannot change the values of hard macros in rewrite rules. For the list of hard macros, see Hard versus soft macros in the syslog-ng PE Administration Guide.

To find and replace the text of the log message

  1. Navigate to Log > Paths.

  2. Select the path(s) where you want to use rewrite rules.

  3. In the Rewrites section, click to add a new rewrite rule. Rewrite rules can be applied before filtering, or after filtering.

    The sequence of filtering and rewrite rules depends on how it was specified in the log path. The sequence of the process is the following:

    1. Parse the message as a syslog message (unless message parsing is explicitly disabled for the source).

    2. Classify the message using a pattern database.

    3. Modify the message using rewrite rules (before filtering).

    4. Filter the messages, for example, based on sender hostname or message content. If the message does not match the configured filter, syslog-ng Store Box(SSB) will not send it to the destination.

    5. Parse the text of the message (that is, the ${MESSAGE} part) using a key-value parser or the sudo parser.

    6. Modify the message using rewrite rules (after filtering and other parsing).

    7. SSB sends the message to the destinations set in the log path. The destinations are local, optionally encrypted files on SSB, or remote servers, such as a database server.

    The message part you want to modify must already exist — if you want to modify a macro that a parser creates, you must add the rewrite rule into the After filtering section.

    Figure 197: Log > Paths — Find and replace in the text of log messages

  4. Enter the part of the message to modify into the In Message part field. For example, MESSAGE, HOST, .SDATA.meta.custom.

    You cannot change the values of hard macros in rewrite rules. For the list of hard macros, see Hard versus soft macros in the syslog-ng PE Administration Guide.

  5. Enter the expression you want to find into the Find field. You can use PCRE regular expressions.

  6. Enter the expression that will replace the Find expression into the Replace with field. By default, SSB replaces the first occurrence of the expression. To use macros, begin with a $ sign and enclose the name of the macro between braces, for example ${MSG}, ${.SDATA.meta.custom}.

    You can use matches of the Find expression as well: ${0} stores the entire match, ${1} is the first group of the match (parentheses), and so on. If you use named patterns in the Find expression (?<name>pattern), you can use ${name} as well.

  7. To replace every occurrence of the Find expression, select the Global option.

  8. To make the Find expression case sensitive, select the Match case option.

  9. Click .

Parsing sudo log messages

The sudo parser separates sudo log messages into name-value pairs.

Use this parser to enrich your log message data with details of privilege escalation events, such as who initiated the event, what command was issued, and so on. The parsed values are automatically assigned metadata, which you can then display on the syslog-ng Store Box(SSB) search interface as dynamic columns.

The aim is to enrich log data with semantic value, and consistently apply the same metadata to the same type of log message data. For example, any information about the client where sudo was executed will always be displayed in the dest and src dynamic columns.

The sudo parser maps the contents of log messages to the dynamic columns listed in Mapping sudo log message contents to dynamic columns.

Example log message:

2016-08-12T06:57:12+02:00 mail sudo: johndoe : TTY=tty ; PWD=pwd ; USER=usr ; GROUP=grp ; TSID=000001 ; ENV=PATH=/usr/local/bin ; COMMAND=cmd -w 20 -c 40
Table 8: Mapping sudo log message contents to dynamic columns
Dynamic column Parsed value Description
action success

The action performed on the resource.

Possible values: success

app sudo The application where the command was issued. Currently, the value of this column is always sudo.
dest mail The IP address or hostname of the entity that validates the authentication request.
src mail The IP address or hostname of the entity that sends the authentication request.
src_user johndoe The user identifier showing which user executed sudo.
tty tty The terminal device name where sudo was executed.
pwd pwd The working directory where sudo was issued.
user usr The user identifier showing who the new user is after executing sudo.
group grp The sudo group target (if present).
tsid 000001 The sudo terminal session (log) identifier (if present).
env PATH=/usr/local/bin The sudo environment variable (if present).
command cmd -w 20 -c 40 The command that was issued by the src_user as a superuser.
tags

authentication

privileged

The metadata to flag the message as a sudo log message.

You can also use the enriched metadata generated from the parsed values in statistics and custom reports.

To use the sudo parser in a specific log path

  1. Navigate to Log > Paths.

  2. Select the path where you want to use the parser.

  3. In the Parser field, Predefined group, select sudo_parser from the drop-down list.

    Figure 198: Log > Paths — Using the sudo_parser in the log path

  4. Click .

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating