You can perform search and replace operations on the log messages to rewrite the messages. Almost all part (macro) of the message can be rewritten. You can use PCRE regular expressions.
If you want to completely replace a message part, or create a new one that does not already exist, see Replace message parts or create new macros with rewrite rules instead.
NOTE: You cannot change the values of hard macros in rewrite rules. For the list of hard macros, see Hard versus soft macros in the syslog-ng PE Administration Guide.
To find and replace the text of the log message
-
Navigate to Log > Paths.
-
Select the path(s) where you want to use rewrite rules.
-
In the Rewrites section, click 
to add a new rewrite rule. Rewrite rules can be applied before filtering, or after filtering.
The sequence of filtering and rewrite rules depends on how it was specified in the log path. The sequence of the process is the following:
-
Parse the message as a syslog message (unless message parsing is explicitly disabled for the source).
-
Classify the message using a pattern database.
-
Modify the message using rewrite rules (before filtering).
-
Filter the messages, for example, based on sender hostname or message content. If the message does not match the configured filter, syslog-ng Store Box(SSB) will not send it to the destination.
-
Parse the text of the message (that is, the ${MESSAGE} part) using a key-value parser or the sudo parser.
-
Modify the message using rewrite rules (after filtering and other parsing).
-
SSB sends the message to the destinations set in the log path. The destinations are local, optionally encrypted files on SSB, or remote servers, such as a database server.
The message part you want to modify must already exist — if you want to modify a macro that a parser creates, you must add the rewrite rule into the After filtering section.
Figure 195: Log > Paths — Find and replace in the text of log messages
-
Enter the part of the message to modify into the In Message part field. For example, MESSAGE, HOST, .SDATA.meta.custom.
You cannot change the values of hard macros in rewrite rules. For the list of hard macros, see Hard versus soft macros in the syslog-ng PE Administration Guide.
-
Enter the expression you want to find into the Find field. You can use PCRE regular expressions.
-
Enter the expression that will replace the Find expression into the Replace with field. By default, SSB replaces the first occurrence of the expression. To use macros, begin with a $ sign and enclose the name of the macro between braces, for example ${MSG}, ${.SDATA.meta.custom}.
You can use matches of the Find expression as well: ${0} stores the entire match, ${1} is the first group of the match (parentheses), and so on. If you use named patterns in the Find expression (?<name>pattern), you can use ${name} as well.
-
To replace every occurrence of the Find expression, select the Global option.
-
To make the Find expression case sensitive, select the Match case option.
-
Click 
.
The sudo parser separates sudo log messages into name-value pairs.
Use this parser to enrich your log message data with details of privilege escalation events, such as who initiated the event, what command was issued, and so on. The parsed values are automatically assigned metadata, which you can then display on the syslog-ng Store Box(SSB) search interface as dynamic columns.
The aim is to enrich log data with semantic value, and consistently apply the same metadata to the same type of log message data. For example, any information about the client where sudo was executed will always be displayed in the dest and src dynamic columns.
The sudo parser maps the contents of log messages to the dynamic columns listed in Mapping sudo log message contents to dynamic columns.
Example log message:
2016-08-12T06:57:12+02:00 mail sudo: johndoe : TTY=tty ; PWD=pwd ; USER=usr ; GROUP=grp ; TSID=000001 ; ENV=PATH=/usr/local/bin ; COMMAND=cmd -w 20 -c 40
Table 8: Mapping sudo log message contents to dynamic columns
| action |
success |
The action performed on the resource.
Possible values: success |
| app |
sudo |
The application where the command was issued. Currently, the value of this column is always sudo. |
| dest |
mail |
The IP address or hostname of the entity that validates the authentication request. |
| src |
mail |
The IP address or hostname of the entity that sends the authentication request. |
| src_user |
johndoe |
The user identifier showing which user executed sudo. |
| tty |
tty |
The terminal device name where sudo was executed. |
| pwd |
pwd |
The working directory where sudo was issued. |
| user |
usr |
The user identifier showing who the new user is after executing sudo. |
| group |
grp |
The sudo group target (if present). |
| tsid |
000001 |
The sudo terminal session (log) identifier (if present). |
| env |
PATH=/usr/local/bin |
The sudo environment variable (if present). |
| command |
cmd -w 20 -c 40 |
The command that was issued by the src_user as a superuser. |
| tags |
authentication
privileged |
The metadata to flag the message as a sudo log message. |
You can also use the enriched metadata generated from the parsed values in statistics and custom reports.
To use the sudo parser in a specific log path
-
Navigate to Log > Paths.
-
Select the path where you want to use the parser.
-
In the Parser field, Predefined group, select sudo_parser from the drop-down list.
Figure 196: Log > Paths — Using the sudo_parser in the log path
-
Click .
The syslog-ng Store Box(SSB) appliance can separate a message consisting of key-value pairs (for example, Postfix log messages) into name-value pairs. The parsed values are automatically added to the metadata about the message, and you can display them on the SSB search interface as dynamic columns. You can specify the separator character to parse different log messages, for example, colon (:) to parse MySQL log messages, or the equal sign (=) for firewall logs. For details on when the key-value parser is executed related to other message processing operations, see the following list.
|
|
Caution:
If the names of keys in the message is the same as the names of SSB soft macros, the value from the parsed message will overwrite the value of the macro. For example, the PROGRAM=value1, MESSAGE=value2 content will overwrite the ${PROGRAM} and ${MESSAGE} macros. To avoid overwriting such macros, use the prefix() option.
Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see .
The parser discards message sections that are not key=value pairs, even if they appear between key=value pairs that can be parsed. |
-
Parse the message as a syslog message (unless message parsing is explicitly disabled for the source).
-
Classify the message using a pattern database.
-
Modify the message using rewrite rules (before filtering).
-
Filter the messages, for example, based on sender hostname or message content. If the message does not match the configured filter, syslog-ng Store Box(SSB) will not send it to the destination.
-
Parse the text of the message (that is, the ${MESSAGE} part) using a key-value parser or the sudo parser.
-
Modify the message using rewrite rules (after filtering and other parsing).
-
SSB sends the message to the destinations set in the log path. The destinations are local, optionally encrypted files on SSB, or remote servers, such as a database server.
NOTE: If a log message contains the same key multiple times (for example, key1=value1, key2=value2, key1=value3, key3=value4, key1=value5), then SSB stores only the last (rightmost) value for the key. Using the previous example, SSB will store the following pairs: key1=value5, key2=value2, key3=value4..
NOTE: The names of the keys can contain only the following characters: numbers (0-9), letters (a-z,A-Z), underscore (_), dot (.), hyphen (-). Other special characters are not permitted.
To configure parsing key-value pairs
-
Navigate to Log > Parsers and select . A new parser is added to the list of parsers.
Figure 197: Log > Parsers — Creating a key=value parser
-
Enter a name for the parser.
-
Enter the character that separates the keys from the values in the incoming messages into the Value separator character field. For example, if your messages look like key1:value1, key2:value2, key3:value3, enter :.
- Enter the character or string that separates the key-value pairs from each other into the Pair separator string field. For example, if your messages look like key1:value1, key2:value2, key3:value3, enter ,. If you use a string, it can be a maximum of 20 characters long.
-
Enter a prefix before the key part of the parsed key-value pairs to help further processing into the Namespace field. For example, to insert the my-parsed-data prefix, enter my-parsed-data. Note the following points:
-
SSB automatically adds the .SDATA. prefix before the value you enter into the Namespace field. That way these values are automatically included in the structured data (SDATA) part of the log message if you forward the message using the IETF-syslog protocol.
-
SSB automatically adds a dot (.) character as a separator between the namespace and the key parsed from the message.
For example, if you entered my-parsed-data as the namespace, and the keys in the message are key1, key2, and so on, then the full name of the macro that contains the parsed values is ${.SDATA.my-parsed-data.key1}, ${.SDATA.my-parsed-data.key2}, and so on. The parsed values are also automatically available as dynamic columns in the SSB search interface (the name of the column is the name of the macro).
-
Click .
-
Navigate to Log > Paths.
-
Select the path where you want to use the parser.
-
In the Parser field, Custom group, select the parser you want to use in this log path.
Figure 198: Log > Paths — Using a key=value parser in the log path
-
Click .
Configuring syslog-ng options
There are several options of the syslog-ng server running on syslog-ng Store Box(SSB) that can be configured. These include:
