-
Title
WEC requires regular restart to maintain logflow -
Description
The number of WEC logs received by syslog-ng gradually decrease over time, causing potential log loss.
Restarting WEC only works as a temporary solution, as the number of logs coming through may rise and stay at a much higher level for a period of time, but then it decreases again.
-
Cause
The open files (fd) limit is reached.
This error can be confirmed through the WEC logs if debug-logging is enabled for the WEC instance. Details on enabling debug-logging can be found here:
https://support.oneidentity.com/syslog-ng-premium-edition/kb/315853/troubleshooting-syslog-ng-wec
-
Resolution
Please check the current soft fd limit and hard fd limit set, and increase the numbers as needed.
How to check the fd of all WEC processes and the system-wide limit:Global fd numbers:cat /proc/sys/fs/file-nrcat /proc/sys/fs/file-maxPrint hard and soft fd limits:ulimit -Hnulimit -SnGet the WEC limits and opened files:for i in $(ps axf|awk '/wec/{print $1}');do lsof -p ${i} >wec.${i}.lsof;cat /proc/${i}/limits >wec.${i}.limits;ls -l /proc/${i}/fd/ >wec.${i}.fd;done
Example output of a WEC process:
Limit Soft Limit Hard Limit UnitsMax cpu time unlimited unlimited secondsMax file size unlimited unlimited bytesMax data size unlimited unlimited bytesMax stack size 8388608 unlimited bytesMax core file size 0 unlimited bytesMax resident set unlimited unlimited bytesMax processes 15531 15531 processesMax open files 1024 4096 filesMax locked memory 16777216 16777216 bytesMax address space unlimited unlimited bytesMax file locks unlimited unlimited locksMax pending signals 15531 15531 signalsMax msgqueue size 819200 819200 bytesMax nice priority 0 0Max realtime priority 0 0Max realtime timeout unlimited unlimited us