-
Title
Search UI stops showing newly received logs after system time changes or is changed backwards -
Description
If the system time suddenly changes to a later date, then that causes problems in the messages.store and related index files on the affected days for each logstore that received messages. By design, the indexer expects time to flow in only one forward moving direction, thus it can only process logs with monotonously increasing timestamps.The possible reasons for this time change could be either of the following:
1. NTP related issue/change2. Manual time change on SSB (not supported)3. Timezone change on SSB (there is a warning if this is attempted, timezone changing is not supported)
Once this happens, messages do not appear in the expected timestamp order in the messages.store file:$ /opt/syslog-ng/bin/lgstool cat ./messages.store...Jan 20 14:36:49.975 myhost.example.com sed: ...Jan 20 22:39:45.285 myhost.example.com cat: ......Jan 20 22:39:54.817 myhost.example.com cat: ...Jan 20 14:40:14.982 myhost.example.com cat: ......
In this case errors like these appear in the logs:2022-01-20T16:50:33+08:00 ssb01 index-logspace1[00000]: An old message appeared out of order; last_received='135246606', current='34015131', time_index_file='not accessible'
Jan 20 15:19:59.000 ssb01 index-local[597]: Error while writing time index file: timestamp is inconsistent; timestamp='1642688399', last_updated_timestamp='1642717030', filename='/opt/ssb/var/logspace/id/20996373924899c4736c584/2022/01-20/time.index' -
Cause
System time suddenly changed to a later date for some reason, which caused new logs to be received with earlier timestamps than those which had already been indexed by the indexer. -
Resolution
The logstore files of the affected day(s) when the system time changed backwards have to be moved from their default directory, as logs with inconsistent timestamps prevent SSB from functioning optimally.The procedure of this is the following:
Please note that the commands in the below steps should be executed on boot firmware!
1. Stop syslog-ng (Basic Settings > System > Service Control: Disable syslog traffic and indexing)
2. Choose an external drive/share or a different directory on the SSB itself where the logs can be moved to.
3. Confirm how much space is needed with the command:# du -sm /mnt/firmware/opt/ssb/var/logspace/id/* | awk 'BEGIN{sum=0} {sum=sum+$1} END{print sum"MB"}'
4. Once a place to move the files to is chosen and it is confirmed that there is enough space on it, enter that directory:# cd /to/where/we/want/to/move/the/files
5. Use the movescript.sh script to perform the file moving for every logspace.The script will create a similar directory hierarchy like the one used below /mnt/firmware/opt/ssb/var/logspace/id/ in the current working directory.Please make sure you are in the correct directory before executing the script! The command 'pwd' can confirm what is the current working directory.Create a file (example: movescript.sh) - make sure to REPLACE the example date in the script with the actual date of the issue appearance!The file's contents should be:
-----------------------------------------#!/bin/bashDAY="2022/01-20"for id in $(ls /mnt/firmware/opt/ssb/var/logspace/id/) ; domkdir -vp "${id}/${DAY}"echo "Moving files into ${id}/${DAY}/ ..."echo ""mv /mnt/firmware/opt/ssb/var/logspace/id/${id}/${DAY}/* ${id}/${DAY}/done-----------------------------------------
NOTE: The DAY variable ( /2022/01-20/ ) in the script should be adjusted to the desired date!
6. Add execute permission to the script:# chmod u+x movescript.sh
7. Execute the script:# ./movescript.sh
8. Confirm whether every file was moved for the affected day - make sure to REPLACE the example date:# ls -l /mnt/firmware/opt/ssb/var/logspace/id/*/2022/01-20/*
9. Start syslog-ng in the GUI (Basic Settings > System > Service Control: Enable syslog traffic and indexing)
After this procedure, the logs that have been moved can only be read via the command line tool, they will not appear on the GUI anymore and cannot be searched there.Delete the logs manually from the external drive/share/directory you moved them to once they are not needed any longer.