Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.14 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka: Publishing messages to Apache Kafka logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations smtp: Generating SMTP messages (e-mail) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql: Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages

manpages

This chapter collects the manual pages of syslog-ng PE and other related applications that are usually distributed and packaged together with the syslog-ng Premium Edition application.

dqtool.1


Table of Contents

dqtool— Display the contents of a disk-buffer file created with syslog-ng Premium Edition
Name

dqtool — Display the contents of a disk-buffer file created with syslog-ng Premium Edition

Synopsis

dqtool [command] [options]

Description

NOTE: The dqtool application is distributed with the syslog-ng Premium Edition system logging application, and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at the syslog-ng page.

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ng Documentation page.

The dqtool application is a utility that can be used to display and format the messages stored in a disk-buffer file.

The cat command

cat [options] [file]

Use the cat command to display the log messages stored in the disk-buffer (also called disk-queue) file, and also information from the header of the disk queue file. The messages are printed to the standard output (stdout), so it is possible to use grep and other tools to find particular log messages, e.g., dqtool cat /var/log/messages.qf |grep 192.168.1.1.

The cat command has the following options:

--debug or -d

Print diagnostic and debugging messages to stderr.

--help or -h

Display a brief help message.

--template=<template> or -t

Format the messages using the specified template.

--verbose or -v

Print verbose messages to stderr.

--version or -V

Display version information.

Example:

        ./dqtool cat ../var/syslog-ng-00000.qf

The output looks like:

        Disk-buffer state loaded; filename='../var/syslog-ng-00000.qf', qout_length='65', qbacklog_length='0', qoverflow_length='9205', qdisk_length='0'
Mar  3 10:52:05 tristram localprg[1234]: seq: 0000011630, runid: 1267609923, stamp: 2010-03-03T10:52:05 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
Mar  3 10:52:05 tristram localprg[1234]: seq: 0000011631, runid: 1267609923, stamp: 2010-03-03T10:52:05 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
Files

/opt/syslog-ng/bin/dqtool

See also

syslog-ng.conf(5)

syslog-ng(8)

Note

For the detailed documentation of syslog-ng PE see The syslog-ng PE 7 Administrator Guide

If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng blogs.

Author

This manual page was written by the One Identity Documentation Team.

Copyright

Copyright 2000-2019 One Identity. Published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version is always available at https://www.syslog-ng.com.

lgstool.1


Table of Contents

lgstool— Inspect and validate the binary log files (logstores) created with syslog-ng Premium Edition
Name

lgstool — Inspect and validate the binary log files (logstores) created with syslog-ng Premium Edition

Synopsis

lgstool [command] [options]

Description

NOTE: The lgstool application is distributed with the syslog-ng Premium Edition system logging application, and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at the syslog-ng page.

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ng Documentation page.

The lgstool application is a utility that can be used to:

  • Display and format the messages stored in logstore files

  • Display the record structure of logstore files

  • Process log messages from orphaned journal files and write them into logstore files

  • Follow (tail) messages arriving to a logstore file real-time

  • Validate the digital signature and timestamp of encrypted logstore files

The cat command

cat [options] [file]

Use the cat command to display the log messages stored in the logstore file. Log messages available in the journal file of the logstore (but not yet written to the logstore file itself) are displayed as well. The messages are printed to the standard output (stdout), so it is possible to use grep and other tools to find particular log messages, e.g., lgstool cat /var/log/messages.lgs |grep 192.168.1.1. Note that can also follow logstore files — for details on this feature, see the section called “The tail command”.

The cat command has the following options:

--debug or -d

Print diagnostic and debugging messages to stderr.

--filter<expression> or -i

Only print messages matching the specified syslog-ng PE filter. All possible macros, regular expressions and logical expressions can be specified in a filter.

Example 1. lgstool cat filter

                  lgstool cat  -t 'host: ${HOST} program: ${PROGRAM} msg: ${MSG}\n' --filter='program("prg0000[0]")' /tmp/logstore-serialized.lgs

--help or -h

Display a brief help message.

--key=<keyfile> or -k

Use the specified private key to decrypt encrypted logstore files.

--seek=<ID> or -s

Display only messages newer than the message specified.

--template=<template> or -t

Format the messages using the specified template.

--verbose or -v

Print verbose messages to stderr.

--version or -V

Display version information.

Example:

        lgstool cat --key=mykey.pem mylogstore.lgs
The inspect command

inspect [options] [file]

Use the inspect command to display structure of the logstore file. The following information is displayed:

  • cipher: The cipher algorithm used to encrypt the logstore file.

  • digest: The digest (hash) algorithm used.

  • encrypt: TRUE if the logstore file is encrypted.

  • compress: TRUE if the logstore file is compressed.

  • hmac: TRUE if the logstore file includes HMAC (Hash-based Message Authentication Code) information for the chunks.

  • chunk_mac: The MAC (Message Authentication Code) of the chunk.

  • file_mac: The MAC (Message Authentication Code) of the chunk.

For timestamped logstore files, the following information is also displayed:

  • chunk_id: The ID of the chunk.

  • Version: The version of the logstore file format used.

  • Policy OID: The OID of the timestamping policy used in the timestamping request.

  • Hash Algorithm: The digest (hash) algorithm used to create the hash of the chunk.

  • Serial number: The serial number of the timestamp.

  • Timestamp: The date when the Timestamping Authority timestamped the chunk.

  • Accuracy: The accuracy of the timestamp.

  • Ordering: Indicates the status of the ordering field in the timestamping request.

  • Nonce: The nonce (a large random number with a high probability that it is generated by the client only once) included in the timestamping request (if any).

  • TSA: The Distinguished Name (DN) of the Timestamping Authority.

The inspect command has the following options:

--debug or -d

Print diagnostic and debugging messages to stderr.

--help or -h

Display a brief help message.

--key=<keyfile> or -k

Use the specified private key to decrypt encrypted logstore files.

--verbose or -v

Print verbose messages to stderr.

--version or -V

Display version information.

Example:

        lgstool inspect --key=mykey.pem mylogstore.lgs

A sample output looks like this:

XFRM_INFO @941
    cipher: aes-128-cbc
    digest: sha1
CHUNK 0@1079: [1 - 1000]:
    encrypt: TRUE
    compress: TRUE
    hmac: TRUE
    chunk_mac: e4d5d813979cf865d5ae4624f7aa98047123cd52
    file_mac: 6600600ca5befb002a73b15be8f0ac04973d5936
TIMESTAMP @36481:
    chunk_id: 0
    Status info:
    Status: Granted.
    Status description: unspecified
    Failure info: unspecified
    TST info:
    Version: 1
    Policy OID: 1.2.3.4
    Hash Algorithm: sha1
    Message data:
        0000 - 66 00 60 0c a5 be fb 00-2a 73 b1 5b e8 f0 ac 04 f.`.....*s.[....
        0010 - 97 3d 59 36                                       .=Y6
    Serial number: 0x029A
    Time stamp: Mar 19 13:48:57 2010 GMT
    Accuracy: 0x01 seconds, 0x01F4 millis, 0x64 micros
    Ordering: no
    Nonce: 0xB613F55AEFFA6DC0
    TSA: unspecified
    Extensions:
The recover command

recover [options] [file]

Warning

Do NOT use the lgstool recover command on logstore files that are actively used by syslog-ng PE. It might lead to data loss. Always stop syslog-ng PE first.

Use the recover command can process and correct broken logstore files. It can also process orphaned journal files and move their contents to the respective logstore file. Encrypted, compressed, and timestamped logstore files can be recovered as well — the private key of the logstore is not needed to recover encrypted logstore files (recovering the encrypted file does not give access to its contents). Note that the recover option is not available in the Windows-version of lgstool.

Warning

The lgstool application cannot fetch timestamps to the chunks (message blocks), so chunks recovered with lgstool are not timestamped (the internal timestamp of the syslog messages is included in the messages).

The recover command has the following options:

--compress-level or -c

Set the level of compression when processing a journal file into a compressed logstore. Default value: 3

--debug or -d

Print diagnostic and debugging messages to stderr.

--help or -h

Display a brief help message.

--verbose or -v

Print verbose messages to stderr.

--version or -V

Display version information.

Example:

lgstool recover mylogstore.lgs
The tail command

tail [options] [file]

Use the tail -f command to follow the contents of a logstore file like the traditional tail command does on Linux/UNIX systems. The messages are printed to the standard output (stdout). Contents of the journal file related to the logstore file are displayed as well.

The tail command has the following options.

--debug or -d

Print diagnostic and debugging messages to stderr.

--help or -h

Display a brief help message.

--filter=<expression> or -i

Only print messages matching the specified syslog-ng PE filter. All possible macros, regular expressions and logical expressions can be specified in a filter.

Example 2. lgstool tail filter

                  lgstool tail  -t 'host: ${HOST} program: ${PROGRAM} msg: ${MSG}\n' --filter='program("prg0000[0]")' /tmp/logstore-serialized.lgs

--follow or -f

Follow mode: display messages as they arrive into the logstore.

--key=<keyfile> or -k

Use the specified private key to decrypt encrypted logstore files.

--lines=<N> or -n

Display the last N lines of the logstore file instead of the last 10. Alternatively, use +N to display lines starting with the Nth.

--sleep_interval=<seconds> or -s

Number of seconds to wait before displaying new messages in follow mode.

--template=<template> or -t

Format the messages using the specified template.

--verbose or -v

Print verbose messages to stderr.

--version or -V

Display version information.

Example:

        lgstool tail -f -n=20 --key=mykey.pem mylogstore.lgs
The validate command

validate [options] [file]

Use the validate command to validate the signatures and timestamps of a logstore file. The validate command has the following options:

--ca-dir=<directory> or -C

The directory that stores the certificates of the trusted Certificate Authorities. Use this option if the timestamps of your logstore files were signed with certificates belonging to different Certificate Authorities.

--ca-dir-layout=<md5|sha1>

The type of the hash used for the CA certificates. The default value (md5) is expected to change to sha1 in subsequent releases of syslog-ng PE.

--ca-file=<file> or -P

A file that stores the certificate of the trusted Certificate Authority. Use this option if the timestamps of your logstore files were signed with a single certificate, or if every such certificate belongs to the same Certificate Authority.

--crl-dir=<directory> or -R

The directory that stores the Certificate Revocation Lists of the trusted Certificate Authorities.

--debug or -d

Print diagnostic and debugging messages to stderr.

--help or -h

Display a brief help message.

--key=<keyfile> or -k

Use the specified private key to decrypt encrypted logstore files.

--require-ts or -T

Consider the logstore file invalid unless the entire file is protected by a valid timestamp.

--seed or -S

Use the ~/.rnd file or the file specified in the $RANDFILE environmental variable as seed. This is needed only on platforms that do not have a /dev/random device (for example, Solaris) and the entropy gathering daemon egd application is not installed on the system.

--ts-name=<name> or -D

Consider the logstore file invalid unless the timestamps are signed by the specified Timestamping Authority. Specify the Distinguished Name (DN) of the Timestamping Authority.

--verbose or -v

Print verbose messages to stderr.

--version or -V

Display version information.

By default, the lgstool validate command checks only the checksum of the file. Use the --require-ts option to validate the timestamps as well. The digital signature of the timestamps is checked only if the --ca-dir or the --ca-file parameter is set.

Example:

        lgstool validate --key=mykey.pem --ca-file=mycacert.pem --ts-name=MYTSA mylogstore.lgs
The reindex command

reindex [options] [file]

The reindex command is an experimental, currently unsupported tool. Do not attempt to use it unless your syslog-ng PE support team explicitly instructs you to do so.

Files

/opt/syslog-ng/bin/lgstool

See also

syslog-ng.conf(5)

syslog-ng(8)

Note

For the detailed documentation of syslog-ng PE see The syslog-ng PE 7 Administrator Guide

If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng blogs.

Author

This manual page was written by the One Identity Documentation Team.

Copyright

Copyright 2000-2019One Identity. Published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version is always available at https://www.syslog-ng.com.

loggen.1


Table of Contents

loggen— Generate syslog messages at a specified rate
Name

loggen — Generate syslog messages at a specified rate

Synopsis

loggen [options]
target [port]

Description

NOTE: The loggen application is distributed with the syslog-ng system logging application, and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at the syslog-ng page.

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ng Documentation page.

The loggen application is tool to test and stress-test your syslog server and the connection to the server. It can send syslog messages to the server at a specified rate, using a number of connection types and protocols, including TCP, UDP, and unix domain sockets. The messages can be generated automatically (repeating the PADDstring over and over), or read from a file or the standard input. The following is a sample generated message:

        <38>2017-04-05T12:16:46 localhost prg00000[1234]: seq: 0000000000, thread: 0000, runid: 1491387406, stamp: 2017-04-05T12:16:46 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD

When loggen finishes sending the messages, it displays the following statistics:

  • average rate: Average rate the messages were sent in messages/second.

  • count: The total number of messages sent.

  • time: The time required to send the messages in seconds.

  • average message size: The average size of the sent messages in bytes.

  • bandwidth: The average bandwidth used for sending the messages in kilobytes/second.

Options
--active-connections <number-of-connections>

Number of connections loggen will use to send messages to the destination. This option is usable only when using TCP or TLS connections to the destination. Default value: 1

The loggen utility waits until every connection is established before starting to send messages. See also the --idle-connections option.

--csv or -C

Send the statistics of the sent messages to stdout as CSV. This can be used for plotting the message rate.

--dgram or -D

Use datagram socket (UDP or unix-dgram) to send the messages to the target. Requires the --inet option as well.

--dont-parse or -d

Do not parse the lines read from the input files, send them as received.

--help or -h

Display a brief help message.

--idle-connections <number-of-connections>

Number of idle connections loggen will establish to the destination. Note that loggen will not send any messages on idle connections, but the connection is kept open using keep-alive messages. This option is usable only when using TCP or TLS connections to the destination. See also the --active-connections option. Default value: 0

--inet or -i

Use the TCP (by default) or UDP (when used together with the --dgram option) protocol to send the messages to the target.

--interval <seconds> or -I <seconds>

The number of seconds loggen will run. Default value: 10

Note

Note that when the --interval and --number are used together, loggen will send messages until the period set in --interval expires or the amount of messages set in --number is reached, whichever happens first.

--ipv6 or -6

Specify the destination using its IPv6 address. Note that the destination must have a real IPv6 address.

--loop-reading or -l

Read the file specified in --read-file option in loop: loggen will start reading from the beginning of the file when it reaches the end of the file.

--number <number-of-messages> or -n <number-of-messages>

Number of messages to generate.

Note

Note that when the --interval and --number are used together, loggen will send messages until the period set in --interval expires or the amount of messages set in --number is reached, whichever happens first.

--no-framing or -F

Do not use the framing of the IETF-syslog protocol style, even if the --syslog-proto option is set.

--quiet or -Q

Output statistics only when the execution of loggen is finished. If not set, the statistics are displayed every second.

--permanent or -T

Keep sending logs indefinitely, without time limit.

--rate <message/second> or -r <message/second>

The number of messages generated per second for every active connection. Default value: 1000

--read-file <filename> or -R <filename>

Read the messages from a file and send them to the target. See also the --skip-tokens option.

Specify - as the input file to read messages from the standard input (stdio). Note that when reading messages from the standard input, loggen can only use a single thread. The -R - parameters must be placed at end of command, like: loggen 127.0.0.1 1061 --read-file -

--sdata <data-to-send> or -p <data-to-send>

Send the argument of the --sdata option as the SDATA part of IETF-syslog (RFC5424 formatted) messages. Use it together with the --syslog-proto option. For example: --sdata "[test name=\"value\"]

--size <message-size> or -s <message-size>

The size of a syslog message in bytes. Default value: 256. Minimum value: 127 bytes, maximum value: 8192 bytes.

--skip-tokens <number>

Skip the specified number of space-separated tokens (words) at the beginning of every line. For example, if the messages in the file look like foo bar message, --skip-tokens 2 skips the foo bar part of the line, and sends only the message part. Works only when used together with the --read-file parameter. Default value: 0

--stream or -S

Use a stream socket (TCP or unix-stream) to send the messages to the target.

--syslog-proto or -P

Use the new IETF-syslog message format as specified in RFC5424. By default, loggen uses the legacy BSD-syslog message format (as described in RFC3164). See also the --no-framing option.

--unix </path/to/socket> or -x </path/to/socket>

Use a UNIX domain socket to send the messages to the target.

--use-ssl or -U

Use an SSL-encrypted channel to send the messages to the target. Note that it is not possible to check the certificate of the target, or to perform mutual authentication.

--version or -V

Display version number of syslog-ng.

Examples

The following command generates 100 messages per second for ten minutes, and sends them to port 2010 of the localhost via TCP. Each message is 300 bytes long.

        loggen --size 300 --rate 100 --interval 600 127.0.0.1 2010

The following command is similar to the one above, but uses the UDP protocol.

        loggen --inet --dgram --size 300 --rate 100 --interval 600 127.0.0.1 2010

Send a single message on TCP6 to the ::1 IPv6 address, port 1061:

        loggen --ipv6 --number 1 ::1 1061

Send a single message on UDP6 to the ::1 IPv6 address, port 1061:

        loggen --ipv6 --dgram --number 1 ::1 1061

Send a single message using a unix domain-socket:

        loggen --unix --stream --number 1 </path/to/socket>

Read messages from the standard input (stdio) and send them to the localhost:

        loggen 127.0.0.1 1061 --read-file -
Files

/opt/syslog-ng/bin/loggen

See also

syslog-ng.conf(5)

Note

For the detailed documentation of syslog-ng PE see The syslog-ng PE 7 Administrator Guide

If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng blogs.

Author

This manual page was written by the One Identity Documentation Team.

Copyright

Copyright 2000-2019One Identity. Published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version is always available at https://www.syslog-ng.com.

Related Documents