Voltar
-
Título
How to Troubleshoot Missing Logs. -
Descrição
This purpose of the article is a help an administrator troubleshoot for missing events in an environment where events are generated on a source device, sent to a syslog-ng PE relay or server and forwarded on to syslog-ng Store Box (SSB).
This is not a step by step troubleshooting process but a series of pointers on where to look for information and in some cases to rule devices out of the troubleshooting process.
-
Resolução
For the purposes of this article- Log messages are referred to as ‘events’
- The device which sends logs to the relay is referred to as the ‘sender’.
- Identify one sender, by ip address & hostname, that has the problem.
This may be-
- A Linux host which has syslog-ng PE already installed on it.
- A Microsoft Windows host with the syslog-ng PE already installed on it.
- A Microsoft Windows host configured to send events using the Windows Event Collector (WEC)
- Another device using third party software to send events to the relay.
- If the sender is a Linux/Unix device, create a local log file and see if the events get sent there as well as to the relay. See the section Creating A Temporary Log file below.
Examine the syslog-ng.conf file to see if it is correctly configured.
Check the statistics on the sender to see if events are being sent to the relay from this sender. See the section Checking Statistics below. And check the statistics on the relay to see if events are being received from this sender.
Refer to the administration guide for help with syslog-ng PE configuration settings.
https://support.oneidentity.com/syslog-ng-premium-edition/technical-documents
Check that the source on the relay is correctly configured to receive events from the sender.- If the Windows Agent is being used to send events to the relay check the configuration of the agent and that the events that you are expecting to receive are actually being generated on the sender. See the “syslog-ng Premium Edition 6.0.22 Administrator Guide for syslog-ng Agent for Windows” for details on how to configure the agent.
https://support.oneidentity.com/syslog-ng-premium-edition/6.0.22/technical-documents
Check that the source on the relay is correctly configured to receive events from the sender.- If WEC is being used to send events from the sender to the relay check the yaml file on the relay to see if the configuration matches (or does not match) any missing events.
Check that the filters are correct.
If certificates are being used check that they are correct and in date.
If kerberos is being used make sure the keytab is correct and that the time on the relay is in sync with the time in the Key Distribution Center (KDC)
Verify that the subscription is correctly configured on the sender. See the “Windows Event Collector Administration Guide” for details
https://support.oneidentity.com/syslog-ng-premium-edition/technical-documents
Check that the source on the relay is correctly configured to receive events from the sender.- If the sender uses Third Party Software then examine the configuration and check on the relay if events are being received from this sender.
Check that the source on the relay is correctly configured to receive events from the sender.
Please note that if this is causing the problem it is third party software and as such not supported by One Identity Technical Support.Check that the source on the relay is correctly configured to receive events from the sender.- If multiple senders are not sending logs to the SSB then check certificates, firewalls and that the correct ports are open between the senders, the relay & the SSB.
Check that the source on the relay is correctly configured to receive events from the sender.- Once you have ruled out the sender, check the configuration on the relay, the source, destination and logpath. For WEC examine the yaml file
Refer to the administration guide for help with configuration settings.
https://support.oneidentity.com/syslog-ng-premium-edition/technical-documents
Check if the relay is receiving the events by creating a local log file and see if the events get sent there as well as to the SSB. See the section Creating A Temporary Log file below.
Examine the syslog-ng.conf to see if it is correctly configured. Refer to the administration guide for configuration settings.
Check that the source on the SSB is correctly configured to receive events from the relay.- If the sender and the relay have been ruled out that just leaves the SSB.
Check the statistics on the SSB to see if it is receiving events from the relay.
Examine the source and destination configuration to see if it is configured correctly
Look for filters, parsers, rewrites, templates etc which may discard events before reaching their destination.
Checking Statistics:To see if the relay and / or the SSB is receiving the events.
In the relay and SSB debug bundles check the statistics file called "syslogngstat.out" or "syslog.stats.<number>"
In it, search for logs that are supressed, dropped and/or processed from the source that the sender reports into.
This can also be done from the command line (use core shell in SSB) by running the following commands and filtering using grep as in the following example.
# /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep <source> | grep suppressed
Creating a Temporary Log File on the RelayTo rule in or out a syslog-ng source, relay or server create a temporary destination on the device for the events coming from the sender. i.e:destination d_file_TEST { file("/tmp/message_${HOST}.log") ; };-
-
Create a log path or modify an existing log path to send the events to the new destination as well as the SSB
-
Once the events start going to the file, check the contents of that file to see if the missing events are present.
-
If they are not present, then it is likely that the problem is on the device or on one of the preceding devices such as the sender.
-
If they are present in the file but not in the relay or SSB then it is likely that the either the relay or the SSB is the problem.
-
Please note that if the relay is unlicensed, then a license will be required in order create a local destination file or logstore. Please contact your One Identity Account Manager for details.
If, after the above investigation, the issue persists please send in the following.- Details of the troubleshooting steps taken to date.
- Debug bundles from the sender, the relay and the SSB
- Details of the issue including sample events which may be missing or mis-configured.
Debug Bundles:- Windows Client:
Please gather a debug bundle from the Windows client. The following KB article details how to do this. Please run the command with the WEC switch enabled. See the following KB article for more details.
https://support.oneidentity.com/syslog-ng-premium-edition/kb/261676/how-to-create-a-syslog-ng-debug-bundle-archive-on-windows-operating-system- Syslog-ng PE Relay:
Send me a debug bundle AND the internal logs from the Syslog-ng PE relay.
The following KB article details how to do this. See the section called “Standard method of generating debug bundle”
https://support.oneidentity.com/syslog-ng-premium-edition/kb/261675/how-to-create-a-syslog-ng-debug-bundle-on-linux-or-unix-operating-system-261675-- SSB:
Send me a debug bundle from the SSB.
To create a debug bundle, navigate from the SSB Web UI to Basic settings | Troubleshooting | System debug | Collect and save current system state info
https://support.oneidentity.com/syslog-ng-store-box/kb/264360
Configuration:- Please see the SSB admin Guide for details on how to configure the settings on the SSB
- The Syslog-ng PE admin guide for details on how to configure the relay
- The WEC admin guide is
- And if using a syslog-ng PE agent to send events to the relay the admin guide(s) can be found here.
For configuration issues it is recommended, that you get help from One Identity Professional Services.
They can sort out the configuration problems but more importantly they can provide training to key individuals so that you can become self-reliant and much more proficient in your management of your overall syslog-ng / SSB environment.
Ref: https://support.oneidentity.com/syslog-ng-store-box/professional-services