-
Title
Malformed log messages may cause performance issues.High CPU. Number of dynamic cluster limit has been reached or syslogng Dynamic Clusters Maximum Reached. -
Description
Malformed log messages can cause performance issues, which can affect Syslog-ng Store Box (SSB) functions like searching or reporting. High CPU.
SSB may send an alert with the title "syslogng Dynamic Clusters Maximum Reached" or Number of dynamic cluster limit has been reached
The following error messages from local logspace may indicate malformed log messages.
ERROR (root@localhost) Too long statistics processing iteration detected; iteration_duration='959'
ERROR (root@localhost) Skipped 7937 rrd updates due to limit, first skipped id is unknown
Log statistics processing limit has been reached; max_number_of_processed_statistics='20000'
Number of dynamic cluster limit has been reached.
-
Cause
SSB collects statistics and metrics about the messages received and delivered.Malformed log messages produce incorrect statistics which may cause performance issues. Some of log senders do not seem to follow the proper syslog message format, and that leads to the misparsing of incoming logs, which then in turn pollutes the internal statistics of syslog-ng. If the size of the statistics reaches a certain limit, then the alert is generated.
Cause 1: The client sends the logs in other syslog format than the server expects.Cause 2: The client sends the logs in a non-syslog format.Cause 3: The log message has an additional syslog header. -
Resolution
In order to remediate this situation, the procedure would need to be the following:
- determine which hosts send the wrongly formatted logs
- determine which source the logs arrive through
- reconfigure that source on the SSB to have the no-parse option turned on
Checking for the existence of malformed logs
The most significant symptom of a malformed log is the incorrect "Program" field.
To check for such program names, execute the following command in the CORE shell.
/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^src.program
Any unknown or strange program name indicate malformed logs.src.program;;\xff\xff\xff\xff;d;processed;3
src.program;;^B^B^H;d;processed;3
src.program;;8HKCEJ;d;processed;3
src.program;;1022;d;processed;2
src.program;;Aug;d;processed;3348565
Finding misparsed logs in SSB
The easiest way to find mispared logs is to use the Search interface of SSB.
NOTE: It is only available if program field indexing is enabled in logspace configuration. (See "Indexed fields" at Log --> Logspaces)
From Syslog-ng statistics it is not possible to tell in which logspace the logs are. Therefore all logspaces have to be checked one by one.
a. Navigate to Search --> Logspaces
b. Select a logspace
c. Select a greater time range (1 week recommended)
d. Click the chart icon beside "Program" in the table header
e. "Statistics for "Program" column" frame should be appeared with the statistics chart of program names
f. Find an incorrect program name in the list and click on it to add it as a search expression. See example incorrect program name examples marked on above image.
g. Click 'Search' to show the log messages with the selected program name.
Find the source hosts
Check the logs to find out the sender host. Basically there are 3 options.
a. The "Host" field shows the original sender host.
b. The "Host" field shows a relay host.
Check the log message itself for clues to identify the original sender host or application.
c. The "Host" field shows an incorrect or unknown host.
Check the log message itself for clues to identify the original sender host or application.
RESOLUTION 1:
1 - Check the sender host and configure the log format corresponding to the format configured in the SSB source.
How to configure BSD-syslog and IETF-syslog message formats in Syslog-ng Store Box
How to configure BSD-syslog and IETF-syslog message formats in Syslog-ng Premium Edittion
RESOLUTION 2:
1 - Select "Do not parse" Syslog flag in SSB source where the logs are received.
Note: This solution will disable the parsing of syslog messages as well. If syslog formatted messages are received on the same source, it is recommended to create two separate sources for the syslog and non-syslog messages and reconfigure the clients accordingly.
RESOLUTION 3:
1 - Check the SSB source if "Do not parse" syslog flag is set.
- In case of yes, disable it.
Note: This solution will enable the parsing of all messages. If non-syslog formatted messages are received on the same source, it is recommended to create two separate sources for the syslog and non-syslog messages and reconfigure the clients accordingly. - In case of no, probably there is a relay before the SSB which puts the additional syslog header. Enable parsing on the relay.
-
Additional Information