How to get information of disk-buffer files (4287099)

Notes:

• This guide will use the default installation path /opt/syslog-ng in the commands and syslog-ng files.
• The syslog-ng persist file format is different in syslog-ng PE 6 and PE 7 so the commands may differ for the two versions.

Useful information about disk-buffers

• disk-buffer() can be configured for a remote destination in the destination() statement. More information: Administration guide
• By default, syslog-ng creates disk-buffer files under /opt/syslog-ng/var directory, unless dir() option is set in disk-buffer().
• The filenames are generated automatically by syslog-ng with the extensions .qf  for normal disk-buffer and .rqf for reliable disk-buffer.
• The disk-buffer file stores processed log messages in the format in which they would have been sent out to the destination, but doesn't store information about the destination.

Getting the list of disk-buffer files

Syslog-ng stores information about disk-buffers in the persist file.
The IP:PORT or DNS:PORT of the destinations and the name of the disk-buffer file.

syslog-ng PE 6

syslog-ng PE 7

/opt/syslog-ng/bin/persist-tool dump /opt/syslog-ng/var/syslog-ng.persist | awk -F '["=]' '/(qfile\(|\.queue)/ { gsub(/[ \t]+/, "", $5); gsub(/^[0-9A-Fa-f]{8}/, "", $5); "echo "$5"|xxd -r -p"|& getline QUEUE; printf("%s ==> %s\n",$1,QUEUE)}'  

Example output:

afsocket_dd_qfile(stream,10.21.10.20:601)  ==> /opt/syslog-ng/var/syslog-ng-00000.rqf

Note: If you get "xxd: command not found" error, install vim-common package in your system.

Getting status information of disk-buffer files

Command syntax

/opt/syslog-ng/bin/dqtool info DISK_QUEUE_FILE

Examples

• Empty normal disk-buffer file, all *_length is zero.

/opt/syslog-ng/bin/dqtool info /opt/syslog-ng/var/syslog-ng-00000.qf 

Disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.qf', qout_length='0', qbacklog_length='0', qoverflow_length='0', qdisk_length='0'

• Non-empty reliable disk-buffer queue file.

One-liner command to get the state of disk-buffer files in default directory

for qfile in /opt/syslog-ng/var/*.?(r)qf ; do /opt/syslog-ng/bin/dqtool info $qfile 2>&1 ; done | cut -f2 -d ";" | awk -F \' '/_length/ { if ($4 > 0) { print "\nNON-EMPTY DISK-BUFFER\t"$0; } else { print "\nEmpty disk-buffer\t"$0 } }'

Printing the content of disk-buffer files

Command syntax

/opt/syslog-ng/bin/dqtool cat DISK_QUEUE_FILE

Example (Short output)

/opt/syslog-ng/bin/dqtool cat /opt/syslog-ng/var/syslog-ng-00000.rqf

Reliable disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.rqf', queue_length='2952', size='-437712'
Jul 31 12:33:48.226 10.21.10.10 2019-07-31T12:33:36 localhost prg00000[1234]: seq: 0000000838, thread: 0000, runid: 1564569216, stamp: 2019-07-31T12:33:36 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
...

Orphan disk-buffer files

Syslog-ng creates a new disk-buffer file instead of using the old one in certain situations, for example, after modifying the disk-buffer configuration or losing the persist information.

Get the list of disk-buffer files from the persist file (see above) and compare it with the content of the disk-buffer files' saving directory.

Example

Disk-buffer file list from persist file:

afsocket_dd_qfile(stream,10.21.10.112:514) = { "queue_file": "/opt/syslog-ng/var/syslog-ng-00001.rqf" }

Disk-buffer files' saving directory content:

The disk-buffer files syslog-ng-00000.qf and syslog-ng-00000.rqf don't exist in the persist file, so they are orphan.