The Websense parser can parse the log messages of Websense Content Gateway (Raytheon|Websense, now Forcepoint). These messages do not completely comply with the syslog RFCs, making them difficult to parse. The websense-parser() of syslog-ng OSE solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs in syslog-ng OSE see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following format:
<PRI><DATE> <TIMEZONE> <IP-ADDRESS> <NAME=VALUE PAIRS>
For example:
<159>Dec 19 10:48:57 EST 192.168.1.1 vendor=Websense product=Security product_version=7.7.0 action=permitted severity=1 category=153 user=- src_host=192.168.2.1 src_port=62189 dst_host=example.com dst_ip=192.168.3.1 dst_port=443 bytes_out=197 bytes_in=76 http_response=200 http_method=CONNECT http_content_type=- http_user_agent=Mozilla/5.0_(Windows;_U;_Windows_NT_6.1;_enUS;_rv:1.9.2.23)_Gecko/20110920_Firefox/3.6.23 http_proxy_status_code=200 reason=- disposition=1034 policy=- role=8 duration=0 url=https://example.com
If you find a message that the websense-parser() cannot properly parse, open a GitHub issue so we can improve the parser.
The syslog-ng OSE application sets the ${PROGRAM} field to Websense.
By default, the websense-specific fields are extracted into name-value pairs prefixed with .websense. For example, the product_version in the previous message becomes ${.websense.product_version}. You can change the prefix using the prefix option of the parser.
@version: 3.25
@include "scl.conf"
log {
source { network(flags(no-parse)); };
parser { websense-parser(); };
destination { ... };
};Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.
The websense-parser() is actually a reusable configuration snippet configured to parse websense messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
| Synopsis: | prefix() |
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by syslog-ng OSE. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, websense-parser() uses the websense. prefix. To modify it, use the following format:
parser {
websense-parser(prefix("myprefix."));
};The Check Point Log Exporter parser can parse Check Point log messages. These messages do not completely comply with the syslog RFCs, making them difficult to parse. The checkpoint-parser() of syslog-ng OSE solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs in syslog-ng OSE see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following formats:
<PRI><VERSION> <YYYY-MM-DD> <HH-MM-SS> <PROGRAM> <PID> <MSGID> - [key1:value1; key2:value2; ... ]
For example:
<134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update"; flags:"150784"; ifdir:"inbound"; logid:"160571424"; loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}";]
Splunk format:
time=1557767758|hostname=r80test|product=Firewall|layer_name=Network|layer_uuid=c0264a80-1832-4fce-8a90-d0849dc4ba33|match_id=1|parent_rule=0|rule_action=Accept|rule_uid=4420bdc0-19f3-4a3e-8954-03b742cd3aee|action=Accept|ifdir=inbound|ifname=eth0|logid=0|loguid={0x5cd9a64e,0x0,0x5060a8c0,0xc0000001}|origin=192.168.96.80|originsicname=cn\=cp_mgmt,o\=r80test..ymydp2|sequencenum=1|time=1557767758|version=5|dst=192.168.96.80|inzone=Internal|outzone=Local|proto=6|s_port=63945|service=443|service_id=https|src=192.168.96.27|
If you find a message that the checkpoint-parser() cannot properly parse, open a GitHub issue so we can improve the parser.
By default, the Check Point-specific fields are extracted into name-value pairs prefixed with .checkpoint. For example, the action in the previous message becomes ${.checkpoint.action}. You can change the prefix using the prefix option of the parser.
@version: 3.25
@include "scl.conf"
log {
source { network(flags(no-parse)); };
parser { checkpoint-parser(); };
destination { ... };
};Note that the parser expects that the entire incorrectly formatted syslog message (starting with its <PRI> value) is in $MSG, which you can achieve by using flags(no-parse) on the input driver.
The checkpoint-parser() is actually a reusable configuration snippet configured to parse Check Point messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
| Synopsis: | prefix() |
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by syslog-ng OSE. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, checkpoint-parser() uses the checkpoint. prefix. To modify it, use the following format:
parser {
checkpoint-parser(prefix("myprefix."));
};The structure of the pattern database
Using parser results in filters and templates
Downloading sample pattern databases
Correlating log messages using pattern databases
Triggering actions for identified messages
Actions and message correlation
The syslog-ng application can compare the contents of the received log messages to predefined message patterns. By comparing the messages to the known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example, can label the messages as user login, application crash, file transfer, and so on events.
To find the pattern that matches a particular message, syslog-ng uses a method called longest prefix match radix tree. This means that syslog-ng creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree.
To classify a message, syslog-ng selects the first character of the message (the text of message, not the header), and selects the patterns starting with this character, other patterns are ignored for the rest of the process. After that, the second character of the message is compared to the second character of the selected patterns. Again, matching patterns are selected, and the others discarded. This process is repeated until a single pattern completely matches the message, or no match is found. In the latter case, the message is classified as unknown, otherwise the class of the matching pattern is assigned to the message.
To make the message classification more flexible and robust, the patterns can contain pattern parsers: elements that match on a set of characters. For example, the NUMBER parser matches on any integer or hexadecimal number (for example, 1, 123, 894054, 0xFFFF, and so on). Other pattern parsers match on various strings and IP addresses. For the details of available pattern parsers, see Using pattern parsers.
The functionality of the pattern database is similar to that of the logcheck project, but it is much easier to write and maintain the patterns used by syslog-ng, than the regular expressions used by logcheck. Also, it is much easier to understand syslog-ng pattens than regular expressions.
Pattern matching based on regular expressions is computationally very intensive, especially when the number of patterns increases. The solution used by syslog-ng can be performed real-time, and is independent from the number of patterns, so it scales much better. The following patterns describe the same message: Accepted password for bazsi from 10.50.0.247 port 42156 ssh2
A regular expression matching this message from the logcheck project: Accepted (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?
A syslog-ng database pattern for this message: Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from @QSTRING:client_addr: @port @NUMBER:port:@ ssh2
For details on using pattern databases to classify log messages, see Using pattern databases.
© ALL RIGHTS RESERVED. Termini di utilizzo Privacy Cookie Preference Center