Troubleshooting Windows Event Collector
When you experience issues while using Windows Event Collector (WEC), run WEC in debug mode to get detailed log messages.
-
Set the log level to debug:
log:
level: "debug"
-
Start WEC.
At every refresh interval, the following debug messages should be displayed:
DEBUG subscriptionEndpoint {"clientAddress": "..."}
DEBUG actionHandler {"messageID": "...", "action": "http://schemas.xmlsoap.org/ws/2004/09/enumeration/Enumerate"}
DEBUG enumerate
This means that the client has connected and requested the subscription list.
-
If you cannot see these messages within the refresh interval, you should check the following channels in the client's event viewer:
Some common error codes and their explanations:
-
5004: A channel specified in the query XML does not exist or cannot be read on the Windows client. This can be caused by the "Network Service" not having permission to read the security log.
Add the "Network Service" account to the Event Log Readers group, and restart the computer for changes to take effect.
-
15008: The query XML of the subscription is invalid.
-
995 (HTTP error 12186): The "Network Service" does not have permission to read the client certificate.
-
HTTP error 403: If everything is set correctly, then it might be possible that a proxy is set and the forwarder tries to connect to the proxy instead of WEC.
TIP: Sometimes proxy settings are not displayed in any GUI window. Check them using netsh winhttp show proxy. To reset proxy settings, use netsh winhttp reset proxy.
Communication is not established between WEC and Windows clients
If the communication between Windows Event Collector (WEC) and the Windows clients is not established, check the following:
-
In the firewall settings between Windows clients and Linux systems, the required port must be opened and allowed.
-
The date and time must be synchronized between the Windows clients and the Linux host.
-
The correct SPN must be mapped with the generated domain user. Check it with:
setspn.exe -L <DOMAIN_NETBIOS_NAME_in_UPPER_CASE>\<domain_user>
For example:
setspn.exe -L TESTDOMAIN\root
-
The copied keytab file on the WEC host must contain the correct SPN. To check it, run the following command from the command line. As a pre-requisite, the klist tool must be installed.
klist -kte <path_of_keytab_file>
For more information on the klist command, see klist(1) - Linux man page.
-
Enable the verbose level Kerberos event debugging on the Windows host, and check the system event logs.
For more information, see How to enable Kerberos event logging/docs.microsoft.com.
-
The Windows Remote Management service must be enabled on the Windows clients.
-
Check the following Windows event logs for errors:
-
Check the WEC debug log for errors.
WEC does not receive logs from previously authenticated Windows clients
If WEC does not receive logs from previously authenticated Windows clients and the WEC debug log shows a similar error message, the current Key Version Number (KVNO) of the associated principal in Active Directory is higher than the KVNO in the keytab file. For security reasons, in such a case, the Kerberos authentication is not established between WEC and Windows.
ERROR KRB-Event gssapi/gssapi.go:90 Error in gss_api_wec_auth
...
ERROR KRB-Event gssapi/gssapi.go:69 Request ticket server [Principal from keytab] not found in keytab (ticket kvno 22) {"idx": 0}
...
ERROR wecserver/wecserver.go:221 Kerberos authentication failed {"client": "[Subscription Info for client]", "error": "Error doKerberosAuth(clientID:[Subscription Info for client]): Error authenticating client: "}
Windows is responsible for the KVNO value. The KVNO can be changed by Active Directory, for example, when the password of the associated domain user is changed.
Check if the KVNOs are synchronized:
-
On Linux, get the KVNO from keytab. Run:
klist -kte <keytab_path>
-
On Windows, get the KVNO from keytab. Run:
ktpass -in <keytab_path>
-
On Windows, get the associated KVNO for the principal from Active Directory. Run:
get-aduser <domain_user> -property msDS-KeyVersionNumber
-
If the keytab files are not yet synchronized, re-generate the keytab file and copy it to the Linux host.
For more information, see Configuring Kerberos authentication on Windows Server (DC) hosts.
-
(Optional) If the problem persists and Kerberos authentication fails in WEC, but the Key Version Numbers match in AD and in the keytab file, restart the Windows clients.
Invalid keytab file
If the keytab file is invalid, a similar error message is listed in the WEC debug log after each post from the Windows clients:
ERROR KRB-Subscription gssapi/gssapi.go:33 Error in gss_api_wec_init
...
ERROR KRB-Subscription gssapi/gssapi.go:69 Unsupported key table format version number
...
-
Check the keytab file by running the following command in the command line. As a pre-requisite, the klist tool must be installed.
klist -kte <path_of_keytab_file>
For more information on the klist command, see klist(1) - Linux man page.
-
Correct the keytab file configuration for WEC.
For more information, see Configuring Kerberos authentication on Windows Server (DC) hosts and Configuring Kerberos authentication on Linux hosts.
WEC configuration examples
The certificate-based authentication and the Kerberos authentication require a different configuration:
This section provides a Windows Event Collector (WEC) configuration example for using certificate-based authentication between Windows Event Forwarding (WEF) and WEC.
Example: Configuring WEC if certificate-based authentication is used
The configuration example lists the following:
-
Server, port, key, and certificate-related settings
-
Log detail level (info) and the path where the log file is created ("/opt/syslog-ng/var/wec.log")
-
Event destination-related settings that specify how the event logs are stored
-
The subscription to the specified computers ("windowsdc.mydomain.com" and "*.trusteddomain.com") in the "ExampleDefaultSubscription" domain
-
Subscriptions-related parameters (such as connectionretry: 60.0 and batchtimeoutlimit seconds): 900.000
server: "wec.mydomain"
port: 5986
keyfile: "/opt/syslog-ng/etc/server.key"
certfile: "/opt/syslog-ng/etc/server.crt"
cadir: "/opt/syslog-ng/etc/cadir"
log:
level: "info"
file: "/opt/syslog-ng/var/wec.log"
eventdestination:
unixdatagram: "/opt/syslog-ng/var/run/wec.sock"
subscriptions:
- name: "ExampleDefaultSubscription"
computers:
- "windowsdc.mydomain.com"
- "*.trusteddomain.com"
contentformat: "RenderedText"
heartbeats: 900.000
connectionretry: 60.0
batchtimeoutlimit: 900.000
queries: |
<QueryList>
<Query Id="0">
<Select Path="Application">*</Select>
<Select Path="Security">*</Select>
<Select Path="System">*</Select>
</Query>
</QueryList>
For more information on the parameters and the configurable values, see Configuring Windows Event Collector.
