Message format in Windows Event Collector for syslog-ng PE
Message format in Windows Event Collector for syslog-ng PE
The Windows Event Collector for syslog-ng Premium Edition (syslog-ng PE) is supported for Windows 8 or newer platforms. Starting with Windows 7, event logging is XML-based, meaning that event log messages reach WEC in XML format. When these are forwarded to the syslog-ng PE server, syslog-ng PE parses them into key-value pairs using the XML parser.
Once event log data is available in syslog-ng PE, you have the flexibility to modify and format data any way you want, using macros and rewrite rules.
Note that while event log data as processed by the WEC tool may differ from the data collected and made available by the syslog-ng Agent for Windows, the Windows Event Collector tool provides you with greater freedom and flexibility when it comes to manipulating your raw data.
Flow control
The Windows Event Collector (WEC) tool applies flow control to minimize event log loss.
WEC regularly (in every second) polls the datagram socket that will receive the Windows events to check whether it exists already. Once the socket has been created (syslog-ng PE has started up), WEC connects to the socket and accepts the incoming connections from the Windows hosts. If the datagram socket does not exist, WEC refuses the incoming connections.
If the socket exists (syslog-ng PE is running) but syslog-ng PE does not read the Unix datagram socket, WEC fills up the kernel buffer of the datagram socket and stores queuesize amounts of log messages in the memory. When all buffers are full, WEC stops reading messages from the HTTP connections to prevent message loss.
The buffer size of a datagram socket is determined by certain Linux kernel parameters: the value of rmem_* (max/default) and the count of net.unix.max_dgram_qlen.
Reliability
WEC flags a message as delivered once it has put the message in the socket buffer. If syslog-ng PE does not read the socket for some reason (for example, because it is still flow-controlled) and syslog-ng PE is stopped, the contents of this socket (that is, the messages that are in the kernel buffer, unread by syslog-ng PE) will be lost.
This is why in cases when a restart is necessary, it is recommended to stop the Windows Event Collector and syslog-ng PE in the following order:
-
Windows Event Collector
-
syslog-ng PE
While it is not guaranteed that syslog-ng PE has read all sockets by the time you stop it, at least you can maximize the chances that it has.
Performance
Performance is dependent on the number of event log messages that the Windows hosts send to Windows Event Collector (WEC) and the capabilities of the XML parser.
Our performance measurements indicate that syslog-ng Premium Edition (syslog-ng PE)'s XML parser is capable of parsing 15,000-20,000 events/second. The exact capacity of the XML parser depends on the complexity of the Windows log messages, as well as the performance of the hardware that syslog-ng PE and WEC are running on. When the limit of 15,000-20,000 events/seconds is reached, a workaround is recommended.
As the value set in the batchsizelimit parameter is treated only as a recommendation by the Windows hosts, there is no direct way to control the amount of messages arriving from the event source computers. For more information, see batchsizelimit in the subscriptions option in Configuring Windows Event Collector.
A possible workaround is to launch multiple WEC servers and create multiple windowsevent() sources in syslog-ng PE. That way, you can distribute your Windows hosts across multiple WEC and syslog-ng PE servers, decreasing the load on individual servers.
To run multiple WEC services per syslog-ng PE service, you need to create your own init script. This is because the init script that comes with WEC enables you to run only a single WEC service per syslog-ng PE service.
Limitations
The Windows Event Collector (WEC) for syslog-ng Premium Edition (syslog-ng PE) currently has the following limitations:
-
Only source-initiated push subscriptions are supported (Windows hosts connect to the WEC server).
An advantage of this, however, is that this requires less firewall rules.
-
The compression of events is not supported.
-
The batchsizelimit and batchtimeoutlimit options are not enforced on the Windows host side: Windows is handling these values only as a recommendation.
For more information, see batchsizelimit and batchtimeoutlimit in the subscriptions option in Configuring Windows Event Collector.
-
WEC cannot work in different authentication modes at once: either Kerberos authentication, or the certificate-based authentication is configured.
-
Kerberos authentication does not work in a WEC cluster deployment.
-
There is a known issue. After several reconnects (if WEC is restarted quickly), the remote sender can stop forwarding the logs for a certain period of time. In this case, restarting the Windows RM service can help.
This issue can also occur between two Windows machines. It has been reported to Microsoft and is awaiting resolution.
