Chat now with support
Chat with Support

syslog-ng Premium Edition 6.0.21 - Administrator Guide for syslog-ng Agent for Windows

Chapter 5. Configuring message sources

The syslog-ng Agent for Windows application can read messages from eventlog containers and text files. The following sections explain how to configure these message sources.

Eventlog sources

The syslog-ng Agent for Windows application can collect messages from the standard Windows eventlog containers, as well as from custom containers. The agent automatically forwards the messages from three standard eventlog containers (Application, Security, System). To enable or disable these sources, or to add custom eventlog containers, complete the following steps:

 NOTE:

The syslog-ng Agent for Windows sends its own log messages into the Application eventlog container.

The agent stores the ID of the last message sent to the destination server, so if the agent is not operating for a time (for example it is restarted ), then it starts reading messages from the last stored message ID, sending out all the new messages.

 Caution:

If an eventlog container becomes corrupt, the agent will stop processing the event source. A log message (Eventlog file is corrupt) is sent directly to the log server to notify about the error.

 Caution:

Hazard of data loss! It is not recommended to setup archiving for the event container. It is possible to lose logs if there are non-processed events in the event container when the archiving is started. Windows closes and renames the event container and starts a new one regardless of any reading applications.

To prevent this, enable overwrite events when needed mode in the Windows Event Viewer with the following conditions:

  • The messages are not generated faster than the agent's processing speed.

  • There is enough window between the first and the last events for planned agent stops. Ensure that new events will not overwrite the event last read by the agent during agent stop.

Procedure 5.1. Managing eventlog sources

Figure 5.1. Managing eventlog sources

Managing eventlog sources

Steps: 

  1. Start the configuration interface of the syslog-ng Agent for Windows application.

  2. Select syslog-ng Agent Settings > Eventlog Sources, and double-click on Event Containers.

    • To disable sending messages from an eventlog container, deselect the checkbox before the name of the container.

    • To modify the log facility associated with the messages of the container, select the container, click Edit, and select the log facility to use in the Log Facility field.

  4. Select Apply, then OK. To activate the changes, restart the syslog-ng Agent service.

Procedure 5.2. Adding eventlog sources

Purpose: 

To forward the messages from an eventlog container to your central log server, complete the following steps.

Prerequisites: 

You need to know the name of the eventlog container. If you do not know the name of the container, see Procedure 5.3, “Determining the name of a custom eventlog container” or Procedure 5.4, “Determining the name of a custom eventlog container on Windows XP, or Server 2003”.

Steps: 

  1. Start the configuration interface of the syslog-ng Agent for Windows application.

  2. Select syslog-ng Agent Settings > Eventlog Sources, and double-click on Event Containers.

  3. Click Add, and enter the name of the container into the Event Container Name field. You can use the * and ? wildcard characters in the name of the container. That way you can handle multiple eventlog containers in a single source.

    If you use wildcards in the name of the eventlog container, note the following points:

    • If none of the existing eventlog containers match the pattern, the syslog-ng Agent will send a warning message into the debug log. For details on enabling debug logs, see the section called “Debugging syslog-ng Agent”.

    • The syslog-ng Agent application checks for new eventlog containers only when it starts or restarts. If a new eventlog container is created with a name that matches the pattern of an eventlog source, restart the syslog-ng Agent service.

      		 Caution:

      Hazard of data loss! If you use wildcards in multiple eventlog source names, make sure that only one pattern matches every container name. If two eventlog sources match the same container, syslog-ng Agent might ignore the messages of the eventlog container.

  4. Click Apply, then OK. To activate the changes, restart the syslog-ng Agent service.

    Expected result: 

    The syslog-ng Agent application starts sending new messages from the newly added eventlog container. Note that the syslog-ng Agent will send existing messages from the eventlog container only if you have selected the Read Old Records option.

Procedure 5.3. Determining the name of a custom eventlog container

Purpose: 

To determine the name of a custom eventlog container, complete the following steps.

Steps: 

  1. Open the Event Viewer application.

  2. Select the custom container you are looking for (for example DNS Server).

  3. Right click on the container and select Properties.

  4. The name of the container is the name of the file (without the extension) displayed in the Logname field (for example for C:\WINDOWS\system32\winevt\Logs\Security.evtx it is Security).

  5. Use this name as the name of the custom eventlog container during the procedure described in Procedure 5.1, “Managing eventlog sources”.

    		 NOTE:

    Some containers are not real containers, but show selected messages collected from multiple containers. To forward such messages to the syslog-ng server, you have to find out which real containers are displayed in the container, and add them to the configuration of the syslog-ng Agent.

    Some containers have the %4 characters in their names. When adding these to the syslog-ng Agent, replace %4 with the / (slash) character. For example write microsoft-windows-bits-client/analytic instead of microsoft-windows-bits-client%4analytic.

    If you are sending old messages to the server as well, the syslog-ng Agent will not send the very first message stored in the container. This is a bug in the Windows API.

Procedure 5.4. Determining the name of a custom eventlog container on Windows XP, or Server 2003

Purpose: 

To determine the name of a custom eventlog container on Windows XP, or Server 2003, complete the following steps.

Steps: 

  1. On the client host select Start > Run > regedit.

  2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\. The custom containers are listed here. For example, the following are valid container names: DFS Replication, File Replication Service, DNS Server.

  3. Use this name as the name of the custom eventlog container during the procedure described in Procedure 5.1, “Managing eventlog sources”.

Procedure 5.5. Managing file sources

Purpose: 

The syslog-ng Agent for Windows application can collect log messages from text files. It can process messages spanning multiple lines, and supports the use of wildcards (*, ?) in filenames to be able to follow log files that are automatically rotated. Note that every line of the file that ends with a newline character is considered a separate message. However, if a file contains only a single line that does not end with a newline character, syslog-ng Agent will not process the line.

To configure file sources, complete the following steps:

 Caution:

Files used as file sources must reside locally on the host the syslog-ng Agent application is running on. Files located on network shares are not supported, because the syslog-ng Agent for Windows application is running as a local service and does not have the privileges to access network shares.

 Caution:

If an application deletes a log file, the application must ensure that syslog-ng Agent had enough time to forward the messages from the file to the central server to avoid losing messages.

Example 5.1. Collecting the logs of multiple applications from a single folder

If two applications log into the same folder (for example C:\logs), you have to create two file sources. For example, if the name of the log files is application1-*.log and application2-*.log, respectively, then create two file sources with the C:\logs Base Directory, but with different File Name Filter: application1-*.log and application2-*.log, respectively.

If other applications log into the C:\logs folder, add a separate expression for each application.

By default, the syslog-ng Agent will send every message to the server that arrives into any of the monitored log files.


Figure 5.2. Managing file sources

Managing file sources

Figure 5.3. Sources properties

Sources properties

Steps: 

  1. Start the configuration interface of the syslog-ng Agent for Windows application.

  2. Select syslog-ng Agent Settings > File Sources, double-click on Sources, and check the Enable option.

  3. Select Add > Browse, and select the folder containing the log files in the Base Directory field. Select or enter the name and extension of the log files in the File Name Filter field. Wildcards can be used. The syslog-ng Agent will forward log messages from every file that is located in this folder and has a name that matches the filter expression.

    		 Caution:

    If you use wildcards in multiple file sources, make sure that the files and folders that match the wildcards do not overlap. That is, every file and folder should belong to only one file source. Monitoring a file from multiple wildcard sources can lead to data loss.

    		 Caution:

    Files used as file sources must reside locally on the host the syslog-ng Agent application is running on. Files located on network shares are not supported, because the syslog-ng Agent for Windows application is running as a local service and does not have the privileges to access network shares.

    		 TIP:

    When specifying the Base Directory, you can use the environment variables of Windows, for example %WINDIR%, %SYSTEMROOT%, %PROGRAMFILES%, and so on.

    		 Caution:

    Note that when managing members of a domain, the selected path must be available on the domain members, for example C:\logs must be available on the client hosts and not on the domain controller.

    • To send messages from the files located in the subfolders of the folder set as Base Directory, select the Recursive option.

    • To change the log facility or the log severity associated to the file source, select the desired facility or priority from the Log Facility or Log Severity fields, respectively.

      		 NOTE:

      Significant changes to the settings of a file source can cause the syslog-ng Agent to resend the entire contents of the matching files. This means that log messages already sent earlier to the syslog-ng server may be resent and thus duplicated in the server logs. Configuration changes that can result in such behavior are:

      • changing the Base Directory,

      • changing filter options,

      • changing the Recursive option.

  5. Optional Step: By default, the syslog-ng Agent application starts sending messages from the beginning of the file. If you only want to send the messages that are newly added to the file, deselect the Read Old Records option.

    		 NOTE:

    Be careful when Read Old Records is disabled. If a new file(s) is created while syslog-ng Agent is stopped, the content of this file will not be forwarded, only the new records. To avoid message loss, never disable Read Old Records in the configuration.

  6. Optional Step: By default, the operating system notifies the syslog-ng Agent application when an application modifies a logfile. However, in some cases this does not happen, because the file-monitoring API of Windows does not notice that the file has changed, for example, when monitoring logfiles of the Windows DHCP service.

    In such case, select the Force Directory Polling option. Note that enabling this option decreases the performance of syslog-ng Agent if you monitor lots of logfiles.

  7. By default, the syslog-ng Agent application assumes that the source files are encoded using the default windows ANSI code page, specific to the locale of the host. If the files have a different encoding, select it from the File Encoding field. Note that the log messages are sent to the destinations using UTF-8 encoding.

  8. If a log messages in the log file consists of multiple lines, that is, the log messages contain newline characters, configure syslog-ng Agent to process the related lines as a single message.

    The syslog-ng Agent application can automatically handle Apache Tomcat Catalina and Oracle SQL log messages. To process such messages, select the name of the application from the Multiple Lines > Application field. Note that the timestamp of Tomcat log messages depends on the locale of the host. The syslog-ng Agent for Windows application automatically removes the last CRLF control character from multi-line messages.

    To process multi-line log messages of a different application, complete the following steps.

    1. Select Multiple Lines > Application > Custom, and set the Multiple Lines > Prefix and optionally the Multiple Lines > Garbage fields.

    2. Specify a string or regular expression that matches the beginning of the log messages in the Multiple Lines > Prefix field. If the Prefix option is set, the syslog-ng Agent ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message.

      		 NOTE:

      Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages.

    3. Use the Multiple Lines > Garbage option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the Garbage option is set, the syslog-ng Agent ignores lines between the line matching the Garbage expression and the next line matching Prefix.

      When receiving multi-line messages from a source when the Garbage option is set but no matching line is received between two lines that match Prefix, the syslog-ng Agent application will continue to process the incoming lines as a single message until a line matching Garbage is received.

      		 Caution:

      If the Garbage option is set, the syslog-ng Agent application discards lines between the line matching the Garbage and the next line matching Prefix expressions.

    4. Optional Step: After creating and testing a custom pattern, please consider sending your pattern to One Identity so we can include it in a future version of syslog-ng Agent. To share your pattern with One Identity and other syslog-ng Agent users, click Multiple Lines > Send custom pattern to BalaBit. Your e-mail application will open, with an e-mail containing the application name and the pattern.

  9. Select Apply, then OK. To activate the changes, restart the syslog-ng Agent service.

Procedure 5.6. Managing the internal source

Purpose: 

All messages generated internally by syslog-ng Agent for Windows application use the internal source. The syslog-ng Agent for Windows application can forward messages originating from the internal source to certain destinations. To configure the internal source, complete the following steps:

Steps: 

  1. Select syslog-ng Agent Settings and double-click on Global Settings.

  2. Enable Global Settings.

  3. Navigate to Internal Messages.

  4. Select the internal message types to forward to the Application event container, or to Remote destinations (meaning all servers that are configured as normal TCP destinations). The message types correspond to the respective message severities. The default setting is internal error and warning messages forwarded to Application event container, and info messages forwarded to Remote destinations.

    Only the selected message types will be forwarded.

    		 Caution:

    If the same message types are selected for both the Application event container and the Remote destinations, and the application event container is also a source, messages can be duplicated.

    		 NOTE:

    These options will be inherited from GPOs (Group Policy Objects). For details, see the section called “Domain versus local settings”. They can also be exported/imported from an XML configuration also.

  5. Click Apply.

Procedure 5.7. Configuring global settings

Purpose: 

The syslog-ng Agent for Windows application has some global settings that can apply to both eventlog and file sources. To configure the global settings, complete the following procedure:

Steps: 

  1. Start the configuration interface of the syslog-ng Agent for Windows application.

  2. Select syslog-ng Agent Settings and double-click on Global Settings.

  3. Set the default log facility associated to the messages.

  4. By default, the filters and regular expressions (see Chapter 7, Filtering messages) used in the message filters are case-sensitive. To make them case-insensitive, select the Regular Expressions Ignore Case or the Filters Ignore Case options, or both.

    		 NOTE:

    The Regular Expressions Ignore Case option makes the Message Contents filter case-insensitive for both file and eventlog sources. The Filters Ignore Case option makes the Computers, Sources and Categories, and the Users filter case-insensitive.

  5. Select Apply, then OK. To activate the changes, restart the syslog-ng Agent service.

Procedure 5.8. Configuring the hostname format

Purpose: 

The syslog-ng Agent for Windows application can send the hostname macro in different format types (FQDN or short hostname), depending on the domain membership of the host, and the source of the message (eventlog or file). The hostname settings will affect all logs originating from file sources, eventlog sources, as well as MARK messages and internal messages of syslog-ng Agent, for example, start/stop messages.

To prevent using two host licenses from a trusted source, use the same hostname type in every outgoing message.

To determine the hostname, syslog-ng Agent queries the short hostname of the machine at startup, and then attempts to resolve it from the DNS server to receive the FQDN. If DNS resolution is not possible, the hostname will be the short hostname.

 NOTE:

The syslog-ng Agent will never rewrite hostnames.

To configure the hostname format globally, complete the following steps:

Steps: 

  1. Select syslog-ng Agent Settings and double-click on Global Settings.

  2. Enable Global Settings.

  3. Navigate to Hostname.

  4. Select the hostname type to use globally.

    • To use only the short hostname in the $HOST macro of the outgoing message, select Use only hostname. This is the default setting.

      • In case of file sources, MARK messages and internal messages of syslog-ng Agent the outgoing hostname will be the short hostname of the machine.

      • In case of eventlog sources, the hostname will be the short hostname of the event message (for example mypc), or syslog-ng Agent will cut the domain name from the FQDN and use the short hostname part (for example mypc.mycompany.local becomes mypc).

    • To use FQDN (hostname.domain_name) in the $HOST macro of the outgoing message, select Use FQDN.

      • In case of file sources, MARK messages and internal messages of syslog-ng Agent, the hostname will be the FQDN of the machine.

        		 NOTE:

        If there is no DNS server, or the DNS server cannot resolve the hostname, only the simple hostname of the machine will be used.

      • In case of eventlog sources, if the hostname of event message is already an FQDN, syslog-ng Agent will use it as the hostname (for example mypc.mycompany.local will be used as such). If this is not an FQDN, syslog-ng Agent will try to resolve this hostname and use the received FQDN as hostname (for example mypc becomes mypc.mycompany.local).

        		 NOTE:

        If there is no DNS server, or the DNS server cannot resolve the hostname, only the short hostname of the event message will be used.

    • To use a custom domain name that will be appended after the short hostname to receive the FQDN, select Use hostname with custom domain name and enter the domain name to append to the short hostname in the field below. This option affects every outgoing message: eventlog sources, file sources, MARK messages and internal messages of syslog-ng Agent.

      • If the hostname is a short hostname, the custom domain name will be appended after the hostname (for example mypc becomes mypc.customcompany.local).

      • If the hostname is an FQDN, the domain name part will be replaced with the custom domain name (for example if the FQDN in the forwarded message is mypc.mycompany.local and the custom domain name is customcompany.local, the hostname in the outgoing message becomes mypc.customcompany.local).

    		 NOTE:

    The hostname still can be different in the outgoing messages if in the eventlog message, the hostname in the event is different from the machine hostname:

    • In case of a forwarded eventlog: the original machine hostname will be the hostname.

    • The machine hostname is different from what the DNS server provides (if there is a DNS server and it can resolve the hostname).

  5. To use lower-case characters in every hostname, enable Convert to lower-case. This is enabled by default. When disabled, mixed lower-case and upper-case characters (if there is any) will be used in hostnames. This option affects every outgoing message: eventlog sources, file sources, MARK messages and internal messages of syslog-ng Agent.

  6. Click Apply.

Procedure 5.9. Disabling sources and filters globally

Purpose: 

Filters and sources can be disabled globally as well. Disabling filters or sources means that the syslog-ng Agent ignores the disabled settings: that is, if the file sources are disabled, the agent does not send the messages from the files to the server. For details, see the following procedure.

Steps: 

  1. Start the configuration interface of the syslog-ng Agent for Windows application.

    • To disable eventlog sources, select syslog-ng Agent Settings, right-click on Eventlog Sources, then select Properties > Disable.

    • To disable file sources, select syslog-ng Agent Settings, right-click on File Sources, then select Properties > Disable.

    • To disable eventlog filters, select syslog-ng Agent Settings > Destinations, right-click on Global Event Filters, then select Properties > Disable.

    • To disable file filters, select syslog-ng Agent Settings > Destinations, right-click on Global File Filters, then select Properties > Disable.

  3. Select Apply, then OK. To activate the changes, restart the syslog-ng Agent service.

Chapter 6. Using TLS-encrypted connections with syslog-ng Agent

When the syslog-ng Premium Edition (syslog-ng PE) server is configured to use mutual authentication, it requests a certificate from the syslog-ng PE clients. The syslog-ng Agent for Windows application can automatically show the requested certificate to the server when the connection is established, provided it is available in the Personal Certificates store of your Local Computer (MMC > Certificates > Computer Account > Local Computer > Personal Certificates).

To import this certificate, use the Certificate Import Wizard. For details, see Importing certificates with the Microsoft Management Console.

NOTE: The syslog-ng Agent for Windows application only supports this certificate import method for the Windows Certificate Store authentication method.

NOTE: If a certificate revocation list (CRL) is available in the Local Computer > Personal Certificates store, syslog-ng Agent for Windows verifies that the certificate of the syslog-ng PE server is not on this list.

Authentication method options for TLS-encryption

While configuring your server, you have two options to use TLS-encryption:

Figure 1: syslog-ng Agent Settings > Local Settings > Destinations > Add New Server > Server Property: Authentication methods available for using TLS encryption when configuring your syslog-ng PE server.

NOTE: When using TLS-encryption on your server, consider the following:

  • It is not possible to configure both File-based certificates, and Windows Certificate Store as an authentication method for newly added destinations.

  • For imported configurations that may already be configured for Windows Certificate Store authentication method, it is possible to configure File-based certificates, but the File-based certificates authentication method overwrites the Windows Certificate Store method.

  • When using File-based certificate as an authentication method, uploading a CA file is required.

NOTE: When using TLS-encryption on your server, consider the following:

  • The syslog-ng Agent for Windows application supports using the File-based certificate as an authentication method both for TLS version 1.1 or lower, and for TLS version 1.2 or higher. One Identity recommends using this option instead of the legacy Windows Certificate Store option, which only supports TLS versions 1.1 or lower.

  • The syslog-ng Agent for Windows application only supports using the Windows Certificate Store as an authentication method for TLS version 1.1 or lower. One Identity does not recommend using this legacy option.

Prerequisites

  1. For using file-based certificates as an authentication method:

    • Valid X.509 certificates, in PEM format.

    • Valid non-encrypted keys, in PEM format.

    • The CA file, the Client Certificate, and the Client Key must exist in their respective configured certificate paths before starting syslog-ng Agent for Windows, and must be properly distributed.

  2. For using Windows Certificate Store as an authentication method:

    • To use the Windows Certificate Store as your authentication method, you must have certificates available in the Personal Certificates store of your Local Computer (MMC > Certificates > Computer Account > Local Computer > Personal Certificates).

      To import these certificates, you can use the Certificate Import Wizard. For details, see Importing certificates with the Microsoft Management Console.

Limitations

Using TLS-encryption with syslog-ng Agent for Windows has the current limitations:

  • For the File-based certificate authentication method, the certificates must be valid X.509 certificates, in PEM format.

  • For the File-based certificate authentication method, the keys must be valid non-encrypted keys, in PEM format.

  • For the File-based certificate authentication method, the configured new files (that is, the CA-file, the client certificate, and the client key) are not distributed by syslog-ng Agent for Windows, but they must exist in the configured certificate path before starting syslog-ng Agent for Windows. The end-user is responsible for ensuring that the required files exist in the configured certificate paths.

Using the File-based certificates authentication method

When using TLS-encryption with syslog-ng Agent for Windows, using file-based certificates as an authentication method is one of your options.

Figure 2: syslog-ng Agent Settings > Local Settings > Destinations > Add New Server > Server Property

NOTE: The following are the responsibility of the user:

  • The user is responsible for ensuring that the certificates are valid X.509 certificates, in PEM format.

  • The user is responsible for ensuring that the keys are valid non-encrypted keys, in PEM format.

  • The configured new files (that is, the CA-file, the Client Certificate, and the Client Key) are not distributed by syslog-ng Agent for Windows, but they must exist in the configured certificate path before starting syslog-ng Agent for Windows. The end-user is responsible for ensuring that the required files exist in the configured certificate paths.

NOTE: When using TLS-encryption on your server, consider the following:

  • The syslog-ng Agent for Windows application supports using the File-based certificate as an authentication method both for TLS version 1.1 or lower, and for TLS version 1.2 or higher. One Identity recommends using this option instead of the legacy Windows Certificate Store option, which only supports TLS versions 1.1 or lower.

  • The syslog-ng Agent for Windows application only supports using the Windows Certificate Store as an authentication method for TLS version 1.1 or lower. One Identity does not recommend using this legacy option.

To configure using File-based certificates as your authentication method,

  1. Navigate to syslog-ng Agent Settings > Local Settings > Destinations > Add New Server > Server Property.

  2. Enable Use TLS encryption.

  3. Select File-based certificates.

    Figure 3: syslog-ng Agent Settings > Local Settings > Destinations > Add New Server > Server Property: Using File-based certificates when using TLS encryption with syslog-ng Agent for Windows

    NOTE: When using TLS-encryption on your server, consider the following:

    • It is not possible to configure both File-based certificates, and Windows Certificate Store as an authentication method for newly added destinations.

    • For imported configurations that may already be configured for Windows Certificate Store authentication method, it is possible to configure File-based certificates, but the File-based certificates authentication method overwrites the Windows Certificate Store method.

    • When using File-based certificate as an authentication method, uploading a CA file is required.

  4. To select your CA file from your local computer, click Select CA file.

    NOTE: Selecting a CA file is required while using the File-based certificates method.

  5. (Optional) To select your certificate and private key, click Select Certificate, and Select Private Key, then select the respective certificate and private key files from your Local Computer.

    NOTE: When selecting your certificate key and private key, consider that you cannot select only one of them. That is, if you select the Certificate Key, selecting the Private Key is also required. Similarly, if you select the Private Key, selecting the Certificate Key is also required.

  6. (Optional). To set compression-related preferences while using TLS-encryption, click Advanced Options and configure the server according to your preferences..

Configuring mutual authentication when using the File-based certificates authentication method

If the syslog-ng Premium Edition (syslog-ng PE) server requests authentication from the syslog-ng Agent for Windows, complete the configuration steps referring to selecting your certificate and private key when using the file-based certificates authentication method.

For details, see Using the File-based certificates authentication method.

NOTE: When using TLS-encryption on your server, consider the following:

  • The syslog-ng Agent for Windows application supports using the File-based certificate as an authentication method both for TLS version 1.1 or lower, and for TLS version 1.2 or higher. One Identity recommends using this option instead of the legacy Windows Certificate Store option, which only supports TLS versions 1.1 or lower.

  • The syslog-ng Agent for Windows application only supports using the Windows Certificate Store as an authentication method for TLS version 1.1 or lower. One Identity does not recommend using this legacy option.

For more information about using mutual authentication options when using the Windows Certificate Store authentication method, see Configuring mutual authentication when using the Windows Certificate Store authentication method.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating