Table of Contents
pdbtool — An application to test and convert syslog-ng pattern database rules
pdbtool [command] [options]
This manual page is only an abstract, for the complete documentation of syslog-ng and pdbtool, see the syslog-ng Documentation page.
The syslog-ng application can match the contents of the log messages to a database of predefined message patterns (also called patterndb). By comparing the messages to the known patterns, syslog-ng is able to identify the exact type of the messages, tag the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The functionality of the pattern database is similar to that of the logcheck project, but the syslog-ng approach is faster, scales better, and is much easier to maintain compared to the regular expressions of logcheck.
The pdbtool application is a utility that can be used to:
convert an older pattern database to the latest database format
merge pattern databases into a single file
automatically create pattern databases from a large amount of log messages
dump the RADIX tree built from the pattern database (or a part of it) to explore how the pattern matching works.
dictionary [options]
Lists every name-value pair that can be set by the rules of the pattern database.
List the tags instead of the names of the name-value pairs.
Name of the pattern database file to use.
List only the name-value pairs that can be set for the messages of the specified $PROGRAM application.
dump [options]
Display the RADIX tree built from the patterns. This shows how are the patterns represented in syslog-ng and it might also help to track down pattern-matching problems. The dump utility can dump the tree used for matching the PROGRAM or the MESSAGE parts.
Enable debug/diagnostic messages on stderr.
Name of the pattern database file to use.
Displays the RADIX tree built from the patterns belonging to the ${PROGRAM} application.
Display the ${PROGRAM} tree.
Enable verbose messages on stderr.
Example and sample output:
pdbtool dump -p patterndb.xml -P 'sshd'
'p'
'assword for'
@QSTRING:@
'from'
@QSTRING:@
'port '
@NUMBER:@ rule_id='fc49054e-75fd-11dd-9bba-001e6806451b'
' ssh' rule_id='fc55cf86-75fd-11dd-9bba-001e6806451b'
'2' rule_id='fc4b7982-75fd-11dd-9bba-001e6806451b'
'ublickey for'
@QSTRING:@
'from'
@QSTRING:@
'port '
@NUMBER:@ rule_id='fc4d377c-75fd-11dd-9bba-001e6806451b'
' ssh' rule_id='fc5441ac-75fd-11dd-9bba-001e6806451b'
'2' rule_id='fc44a9fe-75fd-11dd-9bba-001e6806451b'
match [options]
Use the match command to test the rules in a pattern database. The command tries to match the specified message against the patterns of the database, evaluates the parsers of the pattern, and also displays which part of the message was parsed successfully. The command returns with a 0 (success) or 1 (no match) return code and displays the following information:
the class assigned to the message (that is, system, violation, and so on),
the ID of the rule that matched the message, and
the values of the parsers (if there were parsers in the matching pattern).
The match command has the following options:
Color the terminal output to highlight the part of the message that was successfully parsed.
Enable debug/diagnostic messages on stderr.
Print the debugging information returned by the --debug-pattern option as comma-separated values.
Print debugging information about the pattern matching. See also the --debug-csv option.
Process the messages of the specified log file with the pattern database. This option allows to classify messages offline, and to apply the pattern database to already existing logfiles. To read the messages from the standard input (stdin), specify a hyphen (-) character instead of a filename.
Print only messages matching the specified syslog-ng filter expression.
The text of the log message to match (only the ${MESSAGE} part without the syslog headers).
Name of the pattern database file to use.
Name of the program to use, as contained in the ${PROGRAM} part of the syslog message.
A syslog-ng template expression that is used to format the output messages.
Enable verbose messages on stderr.
Example: The following command checks if the patterndb.xml file recognizes the Accepted publickey for myuser from 127.0.0.1 port 59357 ssh6 message:
pdbtool match -p patterndb.xml -P sshd -M "Accepted publickey for myuser from 127.0.0.1 port 59357 ssh6"
The following example applies the sshd.pdb pattern database file to the log messages stored in the /var/log/messages file, and displays only the messages that received a useracct tag.
pdbtool match -p sshd.pdb \ –file /var/log/messages \ –filter ‘tags(“usracct”);’
merge [options]
Use the merge command to combine separate pattern database files into a single file (pattern databases are usually stored in separate files per applications to simplify maintenance). If a file uses an older database format, it is automatically updated to the latest format (V3). See the syslog-ng Documentation page for details on the different pattern database versions.
Enable debug/diagnostic messages on stderr.
The directory that contains the pattern database XML files to be merged.
Specify filenames to be merged using a glob pattern, for example, using wildcards. For details on glob patterns, see man glob. This pattern is applied only to the filenames, and not on directory names.
Name of the output pattern database file.
Merge files from subdirectories as well.
Sort files into alphabetic order during the merge (first sort by filename, then by directory name).
Enable verbose messages on stderr.
Example:
pdbtool merge --recursive --directory /home/me/mypatterns/ --pdb /var/lib/syslog-ng/patterndb.xml
Currently it is not possible to convert a file without merging, so if you only want to convert an older pattern database file to the latest format, you have to copy it into an empty directory.
patternize [options]
Automatically create a pattern database from a log file containing a large number of log messages. The resulting pattern database is printed to the standard output (stdout). The pdbtool patternize command uses a data clustering technique to find similar log messages and replacing the differing parts with @ESTRING:: @ parsers. For details on pattern databases and message parsers, see the syslog-ng Documentation page. The patternize command is available only in syslog-ng OSE version 3.2 and later.
Enable debug/diagnostic messages on stderr.
The logfile containing the log messages to create patterns from. To receive the log messages from the standard input (stdin), use -.
Recursively iterate on the log lines to cover as many log messages with patterns as possible.
The number of example log messages to include in the pattern database for every pattern. Default value: 1
Do not parse the input file, treat every line as the message part of a log message.
Include a generated name in the parsers, for example, .dict.string1, .dict.string2, and so on.
A pattern is added to the output pattern database if at least the specified percentage of log messages from the input logfile match the pattern. For example, if the input logfile contains 1000 log messages and the --support=3.0 option is used, a pattern is created only if the pattern matches at least 3 percent of the log messages (that is, 30 log messages). If patternize does not create enough patterns, try to decrease the support value.
Default value: 4.0
Enable verbose messages on stderr.
Example:
pdbtool patternize --support=2.5 --file=/var/log/messages
test [options]
Use the test command to validate a pattern database XML file. Note that you must have the xmllint application installed. The test command is available only in syslog-ng OSE version 3.2 and later.
Enable coloring in terminal output.
Enable debug/diagnostic messages on stderr.
Print debugging information on non-matching patterns.
Test only the patterndb rule (specified by its rule id) against its example.
Validate a pattern database XML file.
Enable verbose messages on stderr.
Example:
pdbtool test --validate /home/me/mypatterndb.pdb
The syslog-ng Documentation page
For the detailed documentation of syslog-ng OSE see the syslog-ng Documentation page
If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.
For news and notifications about of syslog-ng, visit the syslog-ng blogs.
persist-tool — Display the content of the persist file
persist-tool [command] [options]
NOTE: The persist-tool application is distributed with the system logging application, and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at https://syslog-ng.com..
This manual page is only an abstract, for the complete documentation of syslog-ng, see https://syslog-ng.com..
The persist-tool application is a utility that can be used to dump the content of the persist file, and manipulate its content.
Persist-tool is a special tool for syslog-ng experts. Do use the tool unless you know exactly what you are doing. Misconfiguring it will result in irrecoverable damage to the persist file, without any warning.
Limitations:
Wildcard characters are not supported in file/directory names.
dump [options] [persist_file]
Use the dump command to print the current content of the persist file in JSON format to the console.
The dump command has the following options:
Display a brief help message.
Example:
persist-tool dump /opt/syslog-ng/var/syslog-ng.persist
The output looks like:
run_id = { "value": "00 00 00 00 0C 00 00 00 " }
host_id = { "value": "00 00 00 00 5F 49 2F 01 " }add [options] [input_file]
Use the add command to add or modify a specified state-entry in the persist file. The state-entry should be in the same format as the dump command displays it. If the given state-entry already exists, it will be updated. Otherwise, a new value will be added. If the given persist state is invalid, it will be skipped.
To use the add command: use persist-tool dump to print the content of the current persist file, and redirect it to a file. Edit the content of this file. Use persist-tool add with this file to modify the persist.
The add command has the following options:
Display a brief help message.
Required parameter. The directory where the persist file is located at. The name of the persist file stored in this directory must be syslog-ng.persist.
Optional parameter. The name of the persist file to generate. Default value: syslog-ng.persist.
Example:
/opt/syslog-ng/bin/persist-tool add dump_persist -o .
The valid output looks like:
log_reader_curpos(Application) OK affile_sd_curpos(/var/aaa.txt) OK
The invalid output looks like:
log_reader_curpos(Application) OK
wrong
FAILED (error: Invalid entry syntax)
affile_sd_curpos(/var/aaa.txt) OKFor the detailed documentation of syslog-ng OSE see The syslog-ng OSE Administrator Guide
If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.
For news and notifications about of syslog-ng, visit the syslog-ng blogs.
Copyright 2000-2019 One Identity. Published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version is always available at https://www.syslog-ng.com.
Table of Contents
syslog-ng-ctl — Display message statistics and enable verbose, debug and trace modes in syslog-ng Open Source Edition
syslog-ng-ctl [command] [options]
NOTE: The syslog-ng-ctl application is distributed with the syslog-ng Open Source Edition system logging application, and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at syslog-ng page.
This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ng Documentation page.
The syslog-ng-ctl application is a utility that can be used to:
enable/disable various syslog-ng messages for troubleshooting
display statistics about the processed messages
handling password-protected private keys
display the currently running configuration of syslog-ng OSE
reload the configuration of syslog-ng OSE.
command [options]
Use the syslog-ng-ctl <command> --set=on command to display verbose, trace, or debug messages. If you are trying to solve configuration problems, the verbose (and occasionally trace) messages are usually sufficient. Debug messages are needed mostly for finding software errors. After solving the problem, do not forget to turn these messages off using the syslog-ng-ctl <command> --set=off. Note that enabling debug messages does not enable verbose and trace messages.
Use syslog-ng-ctl <command> without any parameters to display whether the particular type of messages are enabled or not.
If you need to use a non-standard control socket to access syslog-ng, use the syslog-ng-ctl <command> --set=on --control=<socket> command to specify the socket to use.
Print verbose messages. If syslog-ng was started with the --stderr or -e option, the messages will be sent to stderr. If not specified, syslog-ng will log such messages to its internal source.
Print trace messages of how messages are processed. If syslog-ng was started with the --stderr or -e option, the messages will be sent to stderr. If not specified, syslog-ng will log such messages to its internal source.
Print debug messages. If syslog-ng was started with the --stderr or -e option, the messages will be sent to stderr. If not specified, syslog-ng will log such messages to its internal source.
Example:
syslog-ng-ctl verbose --set=on
The syslog-ng OSE application stores various data, metrics, and statistics in a hash table. Every property has a name and a value. For example:
[syslog-ng]
|
|_[destinations]-[network]-[tcp]->[stats]->{received=12;dropped=2}
|
|_[sources]-[sql]-[stats]->{received=501;dropped=0}
You can query the nodes of this tree, and also use filters to select the information you need. A query is actually a path in the tree. You can also use the ? and * wildcards. For example:
Select every property: *
Select all dropped value from every stats node: *.stats.dropped
The nodes and properties available in the tree depend on your syslog-ng OSE configuration (that is, the sources, destinations, and other objects you have configured), and also on your stats-level() settings.
syslog-ng-ctl query list
Use the syslog-ng-ctl query list command to display the list of metrics that syslog-ng OSE collects about the processed messages. For details about the displayed metrics, see The syslog-ng Administrator Guide???.
An example output:
center.received.stats.processed center.queued.stats.processed destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.dropped destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.processed destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.queued destination.d_elastic.stats.processed source.s_tcp.stats.processed source.severity.7.stats.processed source.severity.0.stats.processed source.severity.1.stats.processed source.severity.2.stats.processed source.severity.3.stats.processed source.severity.4.stats.processed source.severity.5.stats.processed source.severity.6.stats.processed source.facility.7.stats.processed source.facility.16.stats.processed source.facility.8.stats.processed source.facility.17.stats.processed source.facility.9.stats.processed source.facility.18.stats.processed source.facility.19.stats.processed source.facility.20.stats.processed source.facility.0.stats.processed source.facility.21.stats.processed source.facility.1.stats.processed source.facility.10.stats.processed source.facility.22.stats.processed source.facility.2.stats.processed source.facility.11.stats.processed source.facility.23.stats.processed source.facility.3.stats.processed source.facility.12.stats.processed source.facility.4.stats.processed source.facility.13.stats.processed source.facility.5.stats.processed source.facility.14.stats.processed source.facility.6.stats.processed source.facility.15.stats.processed source.facility.other.stats.processed global.payload_reallocs.stats.processed global.msg_clones.stats.processed global.sdata_updates.stats.processed tag..source.s_tcp.stats.processed
The syslog-ng-ctl query list command has the following options:
Use --reset to set the selected counters to 0 after executing the query.
syslog-ng-ctl query get [options]
The syslog-ng-ctl query get <query> command lists the nodes that match the query, and their values.
For example, the "destination*" query lists the configured destinations, and the metrics related to each destination. An example output:
destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.dropped=0 destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.processed=0 destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.queued=0 destination.d_elastic.stats.processed=0
The syslog-ng-ctl query get command has the following options:
Add up the result of each matching node and return only a single number.
For example, the syslog-ng-ctl query get --sum "destination*.dropped" command displays the number of messages dropped by the syslog-ng OSE instance.
Use --reset to set the selected counters to 0 after executing the query.
stats [options]
Use the stats command to display statistics about the processed messages. For details about the displayed statistics, see The syslog-ng Administrator Guide???. The stats command has the following options:
Specify the socket to use to access syslog-ng. Only needed when using a non-standard socket.
Reset all statistics to zero, except for the queued counters. (The queued counters show the number of messages in the message queue of the destination driver, waiting to be sent to the destination.)
Example:
syslog-ng-ctl stats
An example output:
src.internal;s_all#0;;a;processed;6445 src.internal;s_all#0;;a;stamp;1268989330 destination;df_auth;;a;processed;404 destination;df_news_dot_notice;;a;processed;0 destination;df_news_dot_err;;a;processed;0 destination;d_ssb;;a;processed;7128 destination;df_uucp;;a;processed;0 source;s_all;;a;processed;7128 destination;df_mail;;a;processed;0 destination;df_user;;a;processed;1 destination;df_daemon;;a;processed;1 destination;df_debug;;a;processed;15 destination;df_messages;;a;processed;54 destination;dp_xconsole;;a;processed;671 dst.tcp;d_network#0;10.50.0.111:514;a;dropped;5080 dst.tcp;d_network#0;10.50.0.111:514;a;processed;7128 dst.tcp;d_network#0;10.50.0.111:514;a;queued;2048 destination;df_syslog;;a;processed;6724 destination;df_facility_dot_warn;;a;processed;0 destination;df_news_dot_crit;;a;processed;0 destination;df_lpr;;a;processed;0 destination;du_all;;a;processed;0 destination;df_facility_dot_info;;a;processed;0 center;;received;a;processed;0 destination;df_kern;;a;processed;70 center;;queued;a;processed;0 destination;df_facility_dot_err;;a;processed;0
syslog-ng-ctl credentials [options]
The syslog-ng-ctl credentials status command allows you to query the status of the private keys that syslog-ng OSE uses in the network() and syslog() drivers. You can also provide the passphrase for password-protected private keys using the syslog-ng-ctl credentials add command. For details on using password-protected keys, see The syslog-ng Administrator Guide .
syslog-ng-ctl credentials status [options]
The syslog-ng-ctl credentials status command allows you to query the status of the private keys that syslog-ng OSE uses in the network() and syslog() drivers. The command returns the list of private keys used, and their status. For example:
syslog-ng-ctl credentials status Secret store status: /home/user/ssl_test/client-1/client-encrypted.key SUCCESS
If the status of a key is PENDING, you must provide the passphrase for the key, otherwise syslog-ng OSE cannot use it. The sources and destinations that use these keys will not work until you provide the passwords. Other parts of the syslog-ng OSE configuration will be unaffected. You must provide the passphrase of the password-protected keys every time syslog-ng OSE is restarted.
The following log message also notifies you of PENDING passphrases:
Waiting for password; keyfile='private.key'
Specify the socket to use to access syslog-ng. Only needed when using a non-standard socket.
syslog-ng-ctl credentials add [options]
You can add the passphrase to a password-protected private key file using the following command. syslog-ng OSE will display a prompt for you to enter the passphrase. We recommend that you use this method.
syslog-ng-ctl credentials add --id=<path-to-the-key>
Alternatively, you can include the passphrase in the --secret parameter:
syslog-ng-ctl credentials add --id=<path-to-the-key> --secret=<passphrase-of-the-key>
Or you can pipe the passphrase to the syslog-ng-ctl command, for example:
echo "<passphrase-of-the-key>" | syslog-ng-ctl credentials add --id=<path-to-the-key>
Specify the socket to use to access syslog-ng. Only needed when using a non-standard socket.
The path to the password-protected private key file. This is the same path that you use in the key-file() option of the syslog-ng OSE configuration file.
The password or passphrase of the private key.
syslog-ng-ctl config [options]
Use the syslog-ng-ctl config command to display the configuration that syslog-ng OSE is currently running. Note by default, only the content of the main configuration file are displayed, included files are not resolved. To resolve included files and display the entire configuration, use the syslog-ng-ctl config --preprocessed command.
syslog-ng-ctl reload [options]
Use the syslog-ng-ctl reload command to reload the configuration file of syslog-ng OSE without having to restart the syslog-ng OSE application. The syslog-ng-ctl reload works like a SIGHUP.
The syslog-ng-ctl reload command returns 0 if the operation was successful, 1 otherwise.
The syslog-ng Documentation page
For the detailed documentation of syslog-ng OSE see the syslog-ng Documentation page
If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.
For news and notifications about of syslog-ng, visit the syslog-ng blogs.
Table of Contents
syslog-ng-debun — syslog-ng DEBUg buNdle generator
syslog-ng-debun [options]
NOTE: The syslog-ng-debun application is distributed with the syslog-ng OSE system logging application, and is usually part of the syslog-ng OSE package. The latest version of the syslog-ng OSE application is available at the syslog-ng page.
This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ng Documentation page.
The syslog-ng-debun tool collects and saves information about your syslog-ng OSE installation, making troubleshooting easier, especially if you ask help about your syslog-ng OSE related problem.
Run syslog-ng-debun. Using this option is required to actually execute the data collection with syslog-ng-debun. It is needed to prevent accidentally running syslog-ng-debun.
Display the help page.
Do not collect privacy-sensitive data, for example, process tree, fstab, and so on. If you use with -d, then the following parameters will be used for debug mode:-Fev
The directory where syslog-ng OSE is installed instead of /opt/syslog-ng.
Set the working directory, where the debug bundle will be saved. Default value: /tmp. The name of the created file is syslog.debun.${host}.${date}.${3-random-characters-or-pid}.tgz
Start syslog-ng OSE in debug mode, using the -Fedv --enable-core options.
Warning! Using this option under high message load may increase disk I/O during the debug, and the resulting debug bundle can be huge. To exit debug mode, press Enter.
Start syslog-ng OSE in debug mode, using the specified command-line options. To exit debug mode, press Enter. For details on the available options, see ???.
Run syslog-ng OSE in noninteractive debug mode for <seconds>, and automatically exit debug mode after the specified number of seconds.
Wait <seconds> seconds before starting debug mode.
Enable syscall tracing (strace -f or truss -f). Note that using -s itself does not enable debug mode, only traces the system calls of an already running syslog-ng OSE process. To trace system calls in debug mode, use both the -s and -d options.
Capturing packets requires a packet capture tool on the host. The syslog-ng-debun tool attempts to use tcpdump on most platforms, except for Solaris, where it uses snoop.
Capture packets only on the specified interface, for example, eth0.
Capture incoming packets using the following filter: port 514 or port 601 or port 53
Capture incoming packets using the specified filter.
Run syslog-ng OSE in noninteractive debug mode for <seconds>, and automatically exit debug mode after the specified number of seconds.
syslog-ng-debun -r
Create a simple debug bundle, collecting information about your environment, for example, list packages containing the word: syslog, ldd of your syslog-binary, and so on.
syslog-ng-debun -r -l
Similar to syslog-ng-debun -r, but without privacy-sensitive information. For example, the following is NOT collected: fstab, df output, mount info, ip / network interface configuration, DNS resolv info, and process tree.
syslog-ng-debun -r -d
Similar to syslog-ng-debun -r, but it also stops syslog-ng, then restarts it in debug mode (-Fedv --enable-core). To stop debug mode, press Enter. The output of the debug mode collected into a separate file, and also added to the debug bundle.
syslog-ng-debun -r -s
Trace the system calls (using strace or truss) of an already running syslog-ng OSE process.
syslog-ng-debun -r -d -s
Restart syslog-ng OSE in debug mode, and also trace the system calls (using strace or truss) of the syslog-ng OSE process.
syslog-ng-debun -r -p
Run packet capture (pcap) with the filter: port 514 or port 601 or port 53 Also waits for pressing Enter, like debug mode.
syslog-ng-debun -r -p -t 10
Noninteractive debug mode: Similar to syslog-ng-debun -r -p, but automatically exit after 10 seconds.
syslog-ng-debun -r -P "host 1.2.3.4" -D "-Fev --enable-core"
Change the packet-capturing filter from the default to host 1.2.3.4. Also change debugging parameters from the default to -Fev --enable-core. Since a timeout (-t) is not given, waits for pressing Enter.
syslog-ng-debun -r -p -d -w 5 -t 10
Collect pcap and debug mode output following this scenario:
Start packet capture with default parameters (-p)
Wait 5 seconds (-w 5)
Stop syslog-ng
Start syslog-ng in debug mode with default parameters (-d)
Wait 10 seconds (-t 10)
Stop syslog-ng debuging
Start syslog-ng
Stop packet capturing
For the detailed documentation of syslog-ng OSE see the syslog-ng Documentation page
If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.
For news and notifications about of syslog-ng, visit the syslog-ng blogs.
© 2024 One Identity LLC. ALL RIGHTS RESERVED. Terms of Use Privacy Cookie Preference Center