Table of Contents
lgstool — Inspect and validate the binary log files (logstores) created with syslog-ng Premium Edition
lgstool
[command] [options]
NOTE: The lgstool application is distributed with the syslog-ng Premium Edition system logging application, and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at the syslog-ng page.
This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ng Documentation page.
The lgstool application is a utility that can be used to:
Display and format the messages stored in logstore files
Display the record structure of logstore files
Process log messages from orphaned journal files and write them into logstore files
Follow (tail) messages arriving to a logstore file real-time
Validate the digital signature and timestamp of encrypted logstore files
cat
[options] [file]
Use the cat command to display the log messages stored in the logstore file. Log messages available in the journal file of the logstore (but not yet written to the logstore file itself) are displayed as well. The messages are printed to the standard output (stdout), so it is possible to use grep and other tools to find particular log messages, e.g., lgstool cat /var/log/messages.lgs |grep 192.168.1.1. Note that can also follow logstore files — for details on this feature, see the section called “The tail command”.
The cat command has the following options:
Print diagnostic and debugging messages to stderr.
Only print messages matching the specified syslog-ng PE filter. All possible macros, regular expressions and logical expressions can be specified in a filter.
Example 1. lgstool cat filter
lgstool cat -t 'host: ${HOST} program: ${PROGRAM} msg: ${MSG}\n' --filter='program("prg0000[0]")' /tmp/logstore-serialized.lgs
Display a brief help message.
Use the specified private key to decrypt encrypted logstore files.
Display only messages newer than the message specified.
Format the messages using the specified template.
Print verbose messages to stderr.
Display version information.
Example:
lgstool cat --key=mykey.pem mylogstore.lgs
inspect
[options] [file]
Use the inspect command to display structure of the logstore file. The following information is displayed:
cipher: The cipher algorithm used to encrypt the logstore file.
digest: The digest (hash) algorithm used.
encrypt: TRUE
if the logstore file is encrypted.
compress: TRUE
if the logstore file is compressed.
hmac: TRUE
if the logstore file includes HMAC (Hash-based Message Authentication Code) information for the chunks.
chunk_mac: The MAC (Message Authentication Code) of the chunk.
file_mac: The MAC (Message Authentication Code) of the chunk.
For timestamped logstore files, the following information is also displayed:
chunk_id: The ID of the chunk.
Version: The version of the logstore file format used.
Policy OID: The OID of the timestamping policy used in the timestamping request.
Hash Algorithm: The digest (hash) algorithm used to create the hash of the chunk.
Serial number: The serial number of the timestamp.
Timestamp: The date when the Timestamping Authority timestamped the chunk.
Accuracy: The accuracy of the timestamp.
Ordering: Indicates the status of the ordering field in the timestamping request.
Nonce: The nonce (a large random number with a high probability that it is generated by the client only once) included in the timestamping request (if any).
TSA: The Distinguished Name (DN) of the Timestamping Authority.
The inspect command has the following options:
Print diagnostic and debugging messages to stderr.
Display a brief help message.
Use the specified private key to decrypt encrypted logstore files.
Print verbose messages to stderr.
Display version information.
Example:
lgstool inspect --key=mykey.pem mylogstore.lgs
A sample output looks like this:
XFRM_INFO @941 cipher: aes-128-cbc digest: sha1 CHUNK 0@1079: [1 - 1000]: encrypt: TRUE compress: TRUE hmac: TRUE chunk_mac: e4d5d813979cf865d5ae4624f7aa98047123cd52 file_mac: 6600600ca5befb002a73b15be8f0ac04973d5936 TIMESTAMP @36481: chunk_id: 0 Status info: Status: Granted. Status description: unspecified Failure info: unspecified TST info: Version: 1 Policy OID: 1.2.3.4 Hash Algorithm: sha1 Message data: 0000 - 66 00 60 0c a5 be fb 00-2a 73 b1 5b e8 f0 ac 04 f.`.....*s.[.... 0010 - 97 3d 59 36 .=Y6 Serial number: 0x029A Time stamp: Mar 19 13:48:57 2010 GMT Accuracy: 0x01 seconds, 0x01F4 millis, 0x64 micros Ordering: no Nonce: 0xB613F55AEFFA6DC0 TSA: unspecified Extensions:
recover
[options] [file]
Do NOT use the lgstool recover command on logstore files that are actively used by syslog-ng PE. It might lead to data loss. Always stop syslog-ng PE first.
Use the recover command can process and correct broken logstore files. It can also process orphaned journal files and move their contents to the respective logstore file. Encrypted, compressed, and timestamped logstore files can be recovered as well — the private key of the logstore is not needed to recover encrypted logstore files (recovering the encrypted file does not give access to its contents). Note that the recover option is not available in the Windows-version of lgstool.
The lgstool application cannot fetch timestamps to the chunks (message blocks), so chunks recovered with lgstool are not timestamped (the internal timestamp of the syslog messages is included in the messages).
The recover command has the following options:
Set the level of compression when processing a journal file into a compressed logstore. Default value: 3
Print diagnostic and debugging messages to stderr.
Display a brief help message.
Print verbose messages to stderr.
Display version information.
Example:
lgstool recover mylogstore.lgs
tail
[options] [file]
Use the tail -f command to follow the contents of a logstore file like the traditional tail command does on Linux/UNIX systems. The messages are printed to the standard output (stdout). Contents of the journal file related to the logstore file are displayed as well.
The tail command has the following options.
Print diagnostic and debugging messages to stderr.
Display a brief help message.
Only print messages matching the specified syslog-ng PE filter. All possible macros, regular expressions and logical expressions can be specified in a filter.
Example 2. lgstool tail filter
lgstool tail -t 'host: ${HOST} program: ${PROGRAM} msg: ${MSG}\n' --filter='program("prg0000[0]")' /tmp/logstore-serialized.lgs
Follow mode: display messages as they arrive into the logstore.
Use the specified private key to decrypt encrypted logstore files.
Display the last N lines of the logstore file instead of the last 10. Alternatively, use +N
to display lines starting with the Nth.
Number of seconds to wait before displaying new messages in follow mode.
Format the messages using the specified template.
Print verbose messages to stderr.
Display version information.
Example:
lgstool tail -f -n=20 --key=mykey.pem mylogstore.lgs
validate
[options] [file]
Use the validate command to validate the signatures and timestamps of a logstore file. The validate command has the following options:
The directory that stores the certificates of the trusted Certificate Authorities. Use this option if the timestamps of your logstore files were signed with certificates belonging to different Certificate Authorities.
The type of the hash used for the CA certificates. The default value (md5
) is expected to change to sha1
in subsequent releases of syslog-ng PE.
A file that stores the certificate of the trusted Certificate Authority. Use this option if the timestamps of your logstore files were signed with a single certificate, or if every such certificate belongs to the same Certificate Authority.
The directory that stores the Certificate Revocation Lists of the trusted Certificate Authorities.
Print diagnostic and debugging messages to stderr.
Display a brief help message.
Use the specified private key to decrypt encrypted logstore files.
Consider the logstore file invalid unless the entire file is protected by a valid timestamp.
Use the ~/.rnd
file or the file specified in the $RANDFILE
environmental variable as seed. This is needed only on platforms that do not have a /dev/random
device (for example, Solaris) and the entropy gathering daemon egd
application is not installed on the system.
Consider the logstore file invalid unless the timestamps are signed by the specified Timestamping Authority. Specify the Distinguished Name (DN) of the Timestamping Authority.
Print verbose messages to stderr.
Display version information.
By default, the lgstool validate command checks only the checksum of the file. Use the --require-ts
option to validate the timestamps as well. The digital signature of the timestamps is checked only if the --ca-dir
or the --ca-file
parameter is set.
Example:
lgstool validate --key=mykey.pem --ca-file=mycacert.pem --ts-name=MYTSA mylogstore.lgs
reindex
[options] [file]
The reindex command is an experimental, currently unsupported tool. Do not attempt to use it unless your syslog-ng PE support team explicitly instructs you to do so.
For the detailed documentation of syslog-ng PE see The syslog-ng PE 7 Administrator Guide
If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.
For news and notifications about of syslog-ng, visit the syslog-ng blogs.
Copyright 2000-2019One Identity. Published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version is always available at https://www.syslog-ng.com.
Table of Contents
loggen — Generate syslog messages at a specified rate
loggen
[options]target
[port]
NOTE: The loggen application is distributed with the syslog-ng system logging application, and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at the syslog-ng page.
This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ng Documentation page.
The loggen application is tool to test and stress-test your syslog server and the connection to the server. It can send syslog messages to the server at a specified rate, using a number of connection types and protocols, including TCP, UDP, and unix domain sockets. The messages can be generated automatically (repeating the PADD
string over and over), or read from a file or the standard input. The following is a sample generated message:
<38>2017-04-05T12:16:46 localhost prg00000[1234]: seq: 0000000000, thread: 0000, runid: 1491387406, stamp: 2017-04-05T12:16:46 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
When loggen finishes sending the messages, it displays the following statistics:
average rate: Average rate the messages were sent in messages/second.
count: The total number of messages sent.
time: The time required to send the messages in seconds.
average message size: The average size of the sent messages in bytes.
bandwidth: The average bandwidth used for sending the messages in kilobytes/second.
Number of connections loggen will use to send messages to the destination. This option is usable only when using TCP or TLS connections to the destination. Default value: 1
The loggen utility waits until every connection is established before starting to send messages. See also the --idle-connections
option.
Send the statistics of the sent messages to stdout as CSV. This can be used for plotting the message rate.
Use datagram socket (UDP or unix-dgram) to send the messages to the target. Requires the --inet
option as well.
Do not parse the lines read from the input files, send them as received.
Display a brief help message.
Number of idle connections loggen will establish to the destination. Note that loggen will not send any messages on idle connections, but the connection is kept open using keep-alive messages. This option is usable only when using TCP or TLS connections to the destination. See also the --active-connections
option. Default value: 0
Use the TCP (by default) or UDP (when used together with the --dgram
option) protocol to send the messages to the target.
The number of seconds loggen will run. Default value: 10
Note that when the --interval
and --number
are used together, loggen will send messages until the period set in --interval
expires or the amount of messages set in --number
is reached, whichever happens first.
Specify the destination using its IPv6 address. Note that the destination must have a real IPv6 address.
Read the file specified in --read-file
option in loop: loggen will start reading from the beginning of the file when it reaches the end of the file.
Number of messages to generate.
Note that when the --interval
and --number
are used together, loggen will send messages until the period set in --interval
expires or the amount of messages set in --number
is reached, whichever happens first.
Do not use the framing of the IETF-syslog protocol style, even if the --syslog-proto
option is set.
Output statistics only when the execution of loggen is finished. If not set, the statistics are displayed every second.
Keep sending logs indefinitely, without time limit.
The number of messages generated per second for every active connection. Default value: 1000
If you want to change the message rate while loggen is running, send SIGUSR1 to double the message rate, or SIGUSR2 to halve it:
kill -USR1 <loggen-pid> kill -USR2 <loggen-pid>Read the messages from a file and send them to the target. See also the --skip-tokens
option.
Specify -
as the input file to read messages from the standard input (stdio). Note that when reading messages from the standard input, loggen can only use a single thread. The -R -
parameters must be placed at end of command, like: loggen 127.0.0.1 1061 --read-file -
Send the argument of the --sdata
option as the SDATA part of IETF-syslog (RFC5424 formatted) messages. Use it together with the --syslog-proto
option. For example: --sdata "[test name=\"value\"]
The size of a syslog message in bytes. Default value: 256. Minimum value: 127 bytes, maximum value: 8192 bytes.
Skip the specified number of space-separated tokens (words) at the beginning of every line. For example, if the messages in the file look like foo bar message
, --skip-tokens 2
skips the foo bar
part of the line, and sends only the message
part. Works only when used together with the --read-file
parameter. Default value: 0
Use a stream socket (TCP or unix-stream) to send the messages to the target.
Use the new IETF-syslog message format as specified in RFC5424. By default, loggen uses the legacy BSD-syslog message format (as described in RFC3164). See also the --no-framing
option.
Use a UNIX domain socket to send the messages to the target.
Use an SSL-encrypted channel to send the messages to the target. Note that it is not possible to check the certificate of the target, or to perform mutual authentication.
Display version number of syslog-ng.
The following command generates 100 messages per second for ten minutes, and sends them to port 2010 of the localhost via TCP. Each message is 300 bytes long.
loggen --size 300 --rate 100 --interval 600 127.0.0.1 2010
The following command is similar to the one above, but uses the UDP protocol.
loggen --inet --dgram --size 300 --rate 100 --interval 600 127.0.0.1 2010
Send a single message on TCP6 to the ::1
IPv6 address, port 1061
:
loggen --ipv6 --number 1 ::1 1061
Send a single message on UDP6 to the ::1
IPv6 address, port 1061
:
loggen --ipv6 --dgram --number 1 ::1 1061
Send a single message using a unix domain-socket:
loggen --unix --stream --number 1 </path/to/socket>
Read messages from the standard input (stdio) and send them to the localhost:
loggen 127.0.0.1 1061 --read-file -
For the detailed documentation of syslog-ng PE see The syslog-ng PE 7 Administrator Guide
If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.
For news and notifications about of syslog-ng, visit the syslog-ng blogs.
Copyright 2000-2019One Identity. Published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version is always available at https://www.syslog-ng.com.
Table of Contents
pdbtool — An application to test and convert syslog-ng pattern database rules
pdbtool
[command] [options]
This manual page is only an abstract, for the complete documentation of syslog-ng and pdbtool, see the syslog-ng Documentation page.
The syslog-ng application can match the contents of the log messages to a database of predefined message patterns (also called patterndb). By comparing the messages to the known patterns, syslog-ng is able to identify the exact type of the messages, tag the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The functionality of the pattern database is similar to that of the logcheck project, but the syslog-ng approach is faster, scales better, and is much easier to maintain compared to the regular expressions of logcheck.
The pdbtool application is a utility that can be used to:
convert an older pattern database to the latest database format
merge pattern databases into a single file
automatically create pattern databases from a large amount of log messages
dump the RADIX tree built from the pattern database (or a part of it) to explore how the pattern matching works.
dictionary
[options]
Lists every name-value pair that can be set by the rules of the pattern database.
List the tags instead of the names of the name-value pairs.
Name of the pattern database file to use.
List only the name-value pairs that can be set for the messages of the specified $PROGRAM
application.
dump
[options]
Display the RADIX tree built from the patterns. This shows how are the patterns represented in syslog-ng and it might also help to track down pattern-matching problems. The dump utility can dump the tree used for matching the PROGRAM or the MSG parts.
Enable debug/diagnostic messages on stderr.
Name of the pattern database file to use.
Displays the RADIX tree built from the patterns belonging to the ${PROGRAM}
application.
Display the ${PROGRAM}
tree.
Enable verbose messages on stderr.
Example and sample output:
pdbtool dump -p patterndb.xml -P 'sshd'
'p' 'assword for' @QSTRING:@ 'from' @QSTRING:@ 'port ' @NUMBER:@ rule_id='fc49054e-75fd-11dd-9bba-001e6806451b' ' ssh' rule_id='fc55cf86-75fd-11dd-9bba-001e6806451b' '2' rule_id='fc4b7982-75fd-11dd-9bba-001e6806451b' 'ublickey for' @QSTRING:@ 'from' @QSTRING:@ 'port ' @NUMBER:@ rule_id='fc4d377c-75fd-11dd-9bba-001e6806451b' ' ssh' rule_id='fc5441ac-75fd-11dd-9bba-001e6806451b' '2' rule_id='fc44a9fe-75fd-11dd-9bba-001e6806451b'
match
[options]
Use the match command to test the rules in a pattern database. The command tries to match the specified message against the patterns of the database, evaluates the parsers of the pattern, and also displays which part of the message was parsed successfully. The command returns with a 0
(success) or 1
(no match) return code and displays the following information:
the class assigned to the message (that is, system, violation, and so on),
the ID of the rule that matched the message, and
the values of the parsers (if there were parsers in the matching pattern).
The match command has the following options:
Color the terminal output to highlight the part of the message that was successfully parsed.
Enable debug/diagnostic messages on stderr.
Print the debugging information returned by the --debug-pattern
option as comma-separated values.
Print debugging information about the pattern matching. See also the --debug-csv
option.
Process the messages of the specified log file with the pattern database. This option allows to classify messages offline, and to apply the pattern database to already existing logfiles. To read the messages from the standard input (stdin), specify a hyphen (-
) character instead of a filename.
Print only messages matching the specified syslog-ng filter expression.
The text of the log message to match (only the ${MESSAGE}
part without the syslog headers).
Name of the pattern database file to use.
Name of the program to use, as contained in the ${PROGRAM}
part of the syslog message.
A syslog-ng template expression that is used to format the output messages.
Enable verbose messages on stderr.
Example: The following command checks if the patterndb.xml
file recognizes the Accepted publickey for myuser from 127.0.0.1 port 59357 ssh6
message:
pdbtool match -p patterndb.xml -P sshd -M "Accepted publickey for myuser from 127.0.0.1 port 59357 ssh6"
The following example applies the sshd.pdb
pattern database file to the log messages stored in the /var/log/messages
file, and displays only the messages that received a useracct
tag.
pdbtool match -p sshd.pdb \ –file /var/log/messages \ –filter ‘tags(“usracct”);’
merge
[options]
Use the merge command to combine separate pattern database files into a single file (pattern databases are usually stored in separate files per applications to simplify maintenance). If a file uses an older database format, it is automatically updated to the latest format (V3). See the The syslog-ng Administrator Guide for details on the different pattern database versions.
Enable debug/diagnostic messages on stderr.
The directory that contains the pattern database XML files to be merged.
Specify filenames to be merged using a glob pattern, for example, using wildcards. For details on glob patterns, see man glob. This pattern is applied only to the filenames, and not on directory names.
Name of the output pattern database file.
Merge files from subdirectories as well.
Enable verbose messages on stderr.
Example:
pdbtool merge --recursive --directory /home/me/mypatterns/ --pdb /var/lib/syslog-ng/patterndb.xml
Currently it is not possible to convert a file without merging, so if you only want to convert an older pattern database file to the latest format, you have to copy it into an empty directory.
patternize
[options]
Automatically create a pattern database from a log file containing a large number of log messages. The resulting pattern database is printed to the standard output (stdout). The pdbtool patternize command uses a data clustering technique to find similar log messages and replacing the differing parts with @ESTRING:: @
parsers. For details on pattern databases and message parsers, see the The syslog-ng Administrator Guide. The patternize command is available only in syslog-ng PE version 3.2 and later.
Enable debug/diagnostic messages on stderr.
The logfile containing the log messages to create patterns from. To receive the log messages from the standard input (stdin), use -
.
Recursively iterate on the log lines to cover as many log messages with patterns as possible.
The number of example log messages to include in the pattern database for every pattern. Default value: 1
Do not parse the input file, treat every line as the message part of a log message.
Include a generated name in the parsers, for example, .dict.string1
, .dict.string2
, and so on.
A pattern is added to the output pattern database if at least the specified percentage of log messages from the input logfile match the pattern. For example, if the input logfile contains 1000 log messages and the --support=3.0
option is used, a pattern is created only if the pattern matches at least 3 percent of the log messages (that is, 30 log messages). If patternize does not create enough patterns, try to decrease the support value.
Default value: 4.0
Enable verbose messages on stderr.
Example:
pdbtool patternize --support=2.5 --file=/var/log/messages
test
[options]
Use the test command to validate a pattern database XML file. Note that you must have the xmllint application installed. The test command is available only in syslog-ng PE version 3.2 and later.
Enable coloring in terminal output.
Enable debug/diagnostic messages on stderr.
Print debugging information on non-matching patterns.
Test only the patterndb rule (specified by its rule id) against its example.
Validate a pattern database XML file.
Enable verbose messages on stderr.
Example:
pdbtool test --validate /home/me/mypatterndb.pdb
The syslog-ng Administrator Guide
For the detailed documentation of syslog-ng PE see The syslog-ng PE 7 Administrator Guide
If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.
For news and notifications about of syslog-ng, visit the syslog-ng blogs.
Copyright 2000-2019One Identity. Published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version is always available at https://www.syslog-ng.com.
persist-tool — Display the content of the persist file
persist-tool
[command] [options]
NOTE: The persist-tool application is distributed with the system logging application, and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at https://syslog-ng.com..
This manual page is only an abstract, for the complete documentation of syslog-ng, see https://syslog-ng.com..
The persist-tool application is a utility that can be used to dump the content of the persist file, and manipulate its content.
Persist-tool is a special tool for syslog-ng experts. Do use the tool unless you know exactly what you are doing. Misconfiguring it will result in irrecoverable damage to the persist file, without any warning.
Limitations:
The persist-state functions can be used only with syslog-ng PE 5 LTS style persist file (SLP4). Older persist files are not supported.
Wildcard characters are not supported in file/directory names.
dump
[options] [persist_file]
Use the dump command to print the current content of the persist file in JSON format to the console.
The dump command has the following options:
Display a brief help message.
Example:
persist-tool dump /opt/syslog-ng/var/syslog-ng.persist
The output looks like:
run_id = { "value": "00 00 00 00 0C 00 00 00 " } host_id = { "value": "00 00 00 00 5F 49 2F 01 " }
add
[options] [input_file]
Use the add command to add or modify a specified state-entry in the persist file. The state-entry should be in the same format as the dump command displays it. If the given state-entry already exists, it will be updated. Otherwise, a new value will be added. If the given persist state is invalid, it will be skipped.
To use the add command: use persist-tool dump to print the content of the current persist file, and redirect it to a file. Edit the content of this file. Use persist-tool add with this file to modify the persist.
The add command has the following options:
Display a brief help message.
Required parameter. The directory where the persist file is located at. The name of the persist file stored in this directory must be syslog-ng.persist
.
Optional parameter. The name of the persist file to generate. Default value: syslog-ng.persist
.
Example:
/opt/syslog-ng/bin/persist-tool add dump_persist -o .
The valid output looks like:
log_reader_curpos(Application) OK affile_sd_curpos(/var/aaa.txt) OK
The invalid output looks like:
log_reader_curpos(Application) OK wrong FAILED (error: Invalid entry syntax) affile_sd_curpos(/var/aaa.txt) OK
For the detailed documentation of syslog-ng PE see The syslog-ng PE 7 Administrator Guide
If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.
For news and notifications about of syslog-ng, visit the syslog-ng blogs.
Copyright 2000-2019 One Identity. Published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version is always available at https://www.syslog-ng.com.
© 2022 One Identity LLC. ALL RIGHTS RESERVED. Feedback Terms of Use Privacy