-
Title
How to process messages from an orphan disk-buffer file using a separated syslog-ng instance -
Description
This guide describes how to read messages from an orphan disk buffer files using a separated syslog-ng process beside the running instance. -
Cause
Syslog-ng creates a new disk-buffer file instead of using the old one in certain situations, for example, after modifying the disk-buffer configuration or losing the persist information.
The old disk-buffer files became orphan, syslog-ng will not send out the messages from them.
-
Resolution
Warning
- During the resolution a separated syslog-ng instance will be started with its temporary files beside the running one.
Incorrect startup command and configuration may cause issues for the already running instance. - Some steps of the resolution are a bit different for syslog-ng PE 6 and for PE 7.
- The disk-buffer file stores processed log messages in the format in which they would have been sent out to the destination.
- The disk-buffer file doesn't store information about the destination.
Resolution
1. Identify and note down the orphan disk-buffer files. See the knowledge article "How to get information of disk-buffer files" for help.
It is important to know the type of the disk-buffer, that can be normal (.qf) or reliable (.rqf).
In the example the /opt/syslog-ng/var/syslog-ng-00005.rqf orphan reliable disk-buffer file is used.
2. Determine the destination of the logs. The content of the disk-buffer may help, check the above knowledge article.
In the example the destination 10.21.10.20 with the standard network() port 514 is used.3. Create a directory for the temporary instance. In the example the /tmp/qdisk directory is used.
Warning! Make sure that in the directory there is sufficient disk space, at least as much as the size of the orphan disk-buffer file.
mkdir /tmp/qdisk
If different temporary directory is used, create a symbolic link to that with 'ln -s /path/to/tempdir /tmp/qdisk'.
That way you can use the commands from the guide, otherwise change the temporary directory in the example commands and file names.4. Create the configuration file /tmp/qdisk/qdisk.conf for the temporary instance with the following content.
Example for syslog-ng PE 7. For syslog-ng PE 6 change '@version:7.0' to '@version:6.0'.
@version:7.0
@include "scl.conf"options {
keep-hostname(yes);
keep-timestamp(yes);
};destination d_destination {
# ADD YOUR DESTINATION HERE};
log {
destination(d_destination);
};5. Add your destination statement with disk-buffer() to the configuration file. You can copy the destination statement from your running syslog-ng's configuration.
Warning! Add dir() option and set disk-buffer file's destination directory, that is the temporary directory (/tmp/qdisk).Example:
network("10.21.10.20"
disk-buffer(
disk-buf-size(1048576)
reliable(yes)
dir(/tmp/qdisk/)
);6. Start the temporary syslog-ng instance in the foreground.
syslog-ng -Fe -f /tmp/qdisk/qdisk.conf -R /tmp/qdisk/qdisk.persist -c /tmp/qdisk/qdisk.ctl
Syslog-ng will log to the console, so you will see if any error happens during the startup.
The following output shows that an empty disk-buffer file is created and the connection to the remote destination is established.
Follow-mode file source not found, deferring open; filename='/no_such_file_or.dir'
Reliable disk-buffer state saved; filename='/tmp/qdisk/syslog-ng-00000.rqf', qdisk_length='0'
No server license found, running in client mode;
syslog-ng starting up; version='6.0.16', cfg-fingerprint='eaa03b9efb88b87d7c1b0ce7efd042ed8ac0c013', cfg-nonce-ndx='0', cfg-signature='c0327a7f7e6418ce0399a75089377dfb662bb072'
FIPS information; FIPS-mode='disabled'
Syslog connection established; fd='7', server='AF_INET(10.21.10.20:514)', local='AF_INET(0.0.0.0:0)'7. Press CTRL+C to stop syslog-ng.
8. Overwrite the empty disk-buffer file with the orphan disk-buffer file.
mv /opt/syslog-ng/var/syslog-ng-00005.rqf /tmp/qdisk/syslog-ng-00000.rqf
9. Start syslog-ng using the command used in step 6.
10. Open another terminal and check the progress with one of the following ways.
- Checking the number of stored logs in the disk-buffer. (The last number from the output.)
/opt/syslog-ng/sbin/syslog-ng-ctl stats -c /tmp/qdisk/qdisk.ctl | grep 'dst.*queued'
When the disk-buffer file is empty the last number is 0.
dst.syslog;d_remote_srv#0;tcp,10.21.10.20:514;a;queued;0
- Checking the status of the disk-buffer file.
/opt/syslog-ng/bin/dqtool info /tmp/qdisk/syslog-ng-00000.rqf
Empty disk-buffer file example:
Reliable disk-buffer state loaded; filename='/tmp/qdisk/syslog-ng-00000.rqf', queue_length='0', size='0'
11. Press CTRL+C to stop syslog-ng.
12. Check the state of the orphan disk-buffer file. See the knowledge article "How to get information of disk-buffer files" for help.
13. If you have more orphan disk-buffer files, repeat the steps 8-12 for each of them.
14. Remove the temporary directory.
rm -rf /tmp/qdisk
- During the resolution a separated syslog-ng instance will be started with its temporary files beside the running one.