Chapter 8. Customizing the message format
The format of the messages received from the eventlog and the file sources can be customized using templates. You can define separate message format for the eventlog and the file sources. If you have multiple destination servers configured, you can define separate templates for each server. When creating a template to customize the message format, you can use macros, all alphanumeric characters, and the following special characters: <>,():;-+/_.
By default, syslog-ng Agent uses the following templates to forward messages:
-
For the BSD protocol:
<${PRI}>${BSDDATE} ${HOST} ${MSGHDR}${MESSAGE} -
For messages read from the eventlog, the
$MESSAGEpart is${EVENT_USERNAME}: ${EVENT_NAME} ${EVENT_SOURCE}: [${EVENT_TYPE}] ${EVENT_MSG} (EventID ${EVENT_ID})for every protocol. -
For messages read from a file, the
$MESSAGEpart is$FILE_NAME: $FILE_CURRENT_POSITION/$FILE_SIZE: $FILE_MESSAGEfor every protocol.
Procedure 8.1. Customizing messages using templates
Purpose:
To create a template, complete the following procedure:
|
|
Caution:
These macros are available only in the syslog-ng Agent for Windows. To recognize Windows-specific elements of the log message (for example eventlog-related macros) on the syslog-ng server, you have to use parsers on the syslog-ng server. The parser must be configured to match the message format set in the syslog-ng Agent. |
Steps:
-
Start the configuration interface of the syslog-ng Agent for Windows application.
-
Select . Select your log server, and click .
-
To change the format of messages received from eventlog sources, type the message format you want to use into the field.
To change the format of messages received from file sources, type the message format you want to use into the field.
Do not forget to add the
$character before macros. For a complete list of the available macros, see the section called “Macros available in the syslog-ng Agent”.For example, to send the messages in the
DATE HOSTNAME MESSAGEformat, typeDate:$DATE Hostname:$HOST Logmessage:$MESSAGE.Note that the $MESSAGE macro contains not only the text of the log message, but also additional information received from the message source, such as the name of the eventlog container, or the file, as set in the eventlog-specific and file-specific templates.
NOTE: Templates are assigned to a single destination server, so it is possible to use different templates for different servers. However, a server and its failover servers always receive the same message.
Caution: If you have more than one destination servers configured (separate servers, not in failover mode), and you want to use the same template for every server, you must manually copy the template into the configuration of each server. Template modifications are not applied automatically to every server.
-
Click .
-
To activate the changes, restart the syslog-ng Agent service.
The syslog-ng Agent can send the syslog messages using either the ISO or the BSD timestamp format. It is recommended to use the ISO format, because it contains much more information than the BSD format.
Note that in the syslog-ng Agent, the macros without prefix (for example DATE) always refer to the receiving date of the message (for example R_DATE) when it arrived into the event log container, and are included only for compatibility reasons.
Macros available in the syslog-ng Agent
The following sections list the available macros:
|
|
Caution:
These macros are available only in the syslog-ng Agent for Windows. To recognize Windows-specific elements of the log message (for example eventlog-related macros) on the syslog-ng server, you have to use parsers on the syslog-ng server. The parser must be configured to match the message format set in the syslog-ng Agent. |
|
|
NOTE:
Note that if you use the Syslog protocol template (meaning that messages are sent using the IETF-syslog protocol), only the message part of the log message can be customized, the structure of the headers and other information is fixed by the protocol. |
Description:
-
At event container: Name of the application the message came from
-
At file as the name of creator (default value): syslog-ng-agent
Description: Name of the host sending the message. Hostnames are automatically converted to lowercase.
Description: The content of the message, including the text of the message and any file- or event-specific macros that are set for the source.
Description: Date of the message in BSD timestamp format (month/day/hour/minute/second, each expressed in two digits). This is the original syslog time stamp without year information, for example Jun 13 15:58:00. If possible, it is recommended to use ISODATE for timestamping.
Description: Date of the message using the BSD-syslog style timestamp format (month/day/hour/minute/second, each expressed in two digits). This is the original syslog time stamp without year information, for example: Jun 13 15:58:00.
Description: A nonstandard format for the date of the message using the same format as ${DATE}, but including the year as well, for example: 2006 Jun 13 15:58:00.
Description: Date of the message in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00. If possible, it is recommended to use ISODATE for timestamping. Note that the syslog-ng Agent cannot produce fractions of a second (for example milliseconds) in the timestamp.
Description: The month the message was sent as a decimal value, prefixed with a zero if smaller than 10.
Description: The English name of the month the message was sent, abbreviated to three characters (for example Jan, Feb, and so on).
Description: The time-zone as hour offset from GMT, for example: -07:00. In syslog-ng 1.6.x this used to be -0700 but as ${ISODATE} requires the colon it was added to ${TZOFFSET} as well.
Description: Standard UNIX timestamp, represented as the number of seconds since 1970-01-01T00:00:00.
Description: The number of received messages per container since the syslog-ng Agent for Windows was started.
Description: A unique ID for messages generated at reception time on the receiving host. It facilitates defining relationships between messages that are potentially distributed to different files on the same host, or different hosts.
Description: Importance level of the message represented as a number: 6 - Success, 5 - Informational, 4- Warning, or 3 - Error).
Description: Contains the entire message in XML format. This is an alias of the EVENT_MESSAGE_XML.
Description: Name of the Windows event log container (for example Application or Security).
Description: The position of the message from the beginning of the file in bytes.
Description: Importance level of the message represented as a number: 6 - Success, 5 - Informational, 4- Warning, or 3 - Error).
Chapter 9. Controlling the syslog-ng Agent services
During installation, syslog-ng Agent registers the syslog-ng Agent service that is started automatically when the host boots. To disable the automatic startup of the syslog-ng Agent use the interface. The service is running with the privileges of the NT AUTHORITY\SYSTEM user.
To manually start or stop the service use the interface, or navigate to . Note that in the latter case if User Access Control (UAC) is enabled, you need the Run as Administrator privilege to start or stop the syslog-ng Agent.
When the syslog-ng Agent service is started or stopped, it sends a syslog message to the central log server and an eventlog message to the Application eventlog container of the host.
|
|
NOTE:
It is possible to run the service with an administrator account that has "log on as service" rights (to set user rights, navigate to ). These settings are unsupported, use them only at your own risk. Also note that during the next upgrade procedure, these settings will be overwritten by factory default settings. |
The syslog-ng Agent for Windows application has the following command-line options:
|
|
NOTE:
Command-line options are case-insensitive. The options consist of a single letter introduced by either " |
|
|
NOTE:
Command-line options will only work with administrator permission. |
- /c
-
Start the syslog-ng Agent using the specified XML configuration file.
- /d
-
Start the syslog-ng Agent in debug mode.
- /h
-
Display a help message about the command-line options.
- /i
-
Install the syslog-ng Agent service into the services list.
- /r
-
Remove the syslog-ng Agent service from the services list.
- /v
-
Display version information.
- /x
-
Validate XML configuration file without importing it.
To use these options, select , navigate to the directory where the syslog-ng Agent is installed (for example cd C:\Windows\Program Files\BalaBit\syslog-ng Agent\), and execute the syslog-ng-agent.exe file with the required option.
Example 9.1. Using command line options
To start syslog-ng Agent in debug mode:
syslog-ng-agent.exe /d
To start syslog-ng Agent with XML configuration file:
syslog-ng-agent.exe /c C:\ConfigFiles\syslog-ng-agent-conf.xml
To register syslog-ng Agent as a service using XML configuration file:
syslog-ng-agent.exe /i C:\ConfigFiles\syslog-ng-agent-conf.xml
Chapter 10. Troubleshooting syslog-ng Agent for Windows
In case you experience problems with the syslog-ng Agent for Windows application, the following points can be of help.
|
|
NOTE:
The followings address only problems specific to the syslog-ng Agent, and assume that communication between the server and the client is otherwise possible (that is, the server is properly configured to receive messages and is available on the network, and name resolution is properly configured on the client). |
Configuration changes do not take effect:
Configuration changes take effect only after restarting the syslog-ng service or rebooting the system. Also restart the system after changing the timezone settings of the host, or importing a certificate that you want to use to authenticate the communication between the agent and the server. If the configuration of the agent has changed since the last restart, the syslog-ng Agent sends a message of the change, including the hmac-sha-1 hash of the new configuration.
Also note that if your clients are managed from a Domain Controller, configuration changes are not instantly downloaded to the client hosts, only at the time of the next group policy update. To update the configuration of a client host earlier, open a command prompt on the client host, and issue the gpupdate /force command.
After downloading the configuration from the Domain Controller, the syslog-ng Agent service is automatically restarted if the configuration has changed.
|
|
NOTE:
Certain domain settings that can affect the syslog-ng Agent are downloaded only when the machine is rebooted. For example, moving the computer from one group policy to another requires a reboot to have effect. |
The syslog-ng Agent does not send messages to the server:
Check the Application eventlog for messages of the syslog-ng Agent. In case of connection errors and certificate problems, the syslog-ng Agent sends error messages into the eventlog. Ensure that the destination address of the server is correctly set. If you use SSL encryption, verify that the certificate of the Certificate Authority of the server and that the certificate of the client are properly imported. If there are no error messages, check the logs on your log server: the syslog-ng Agent sends a MARK message every ten minutes even if there are no other messages to send (unless you have disabled MARK messages).
The syslog-ng Agent sends only MARK messages to the server:
Verify that you have configured the eventlog and file sources, and that they have not been disabled globally. If these settings are correct but the server still does not send any messages, temporarily disable all filters to see that they are not configured to ignore every message. When using filter, it is also recommended to check the global case-sensitivity settings.
The hostname used in the messages changes: If a host is sometimes logged in into a domain and sometimes it is not, its hostname might reflect this. To avoid this situation, select and double-click on . Enable , navigate to and select . This causes syslog-ng Agent to resolve its own hostname from DNS and use the resolved FQDN in the syslog messages. For details, see Procedure 5.8, “Configuring the hostname format”.
Command-line parameters are ignored:
Command-line parameters work only for administrators if User Account Control (UAC) is enabled. To execute syslog-ng Agent with command-line parameters, select , right-click on .
If you contact our Support Team about a problem with the syslog-ng Agent for Windows, execute the syslog-ng-agent -V command from the command line and include every version and platform information it displays in your support request.
CPU load is high: See the section called “Sending messages and CPU load”.
Losing messages from eventlog containers:
An eventlog container is a special file. The Agent reads this file, formats the messages and sends them to remote log server. Note that the eventlog container can be configured only to a certain size. If the container reaches that size, Windows writes the next message to the beginning of the file. As a result, if the agent is not running (or the destination server is unavailable) so long that the eventlog container is filled up, messages can be lost.
Logs are not forwarded instantly:
For the logs of certain applications (for example, Internet Information Services (IIS) for Windows Server), the syslog-ng Agent for Windows application does not forward the log messages in real time, only in batches after a certain amount of time. The cause of the problem is that the Windows operating system does not immediately flush its buffers to the file when an application sends a log message. The syslog-ng Agent for Windows application immediately starts sending the log messages when they become available in the log file.
The syslog-ng Agent application can send messages to the server when the Windows Scheduler provides resources to the syslog-ng Agent. When there are many unsent log messages in the log sources, and there is no other significant activity on the host, syslog-ng will start to send the messages to the server, possibly increasing the CPU load to 100%. After all messages have been sent, or if another application requires the resources, the CPU load decreases back to normal.
|
|
TIP:
To avoid the initial large load on the CPU, limit the rate of message sending temporarily. You can remove the limit after the old messages have been sent. For details, see Procedure 4.2, “Limiting the rate of messages”. |
When relaying the messages from multiple sources, the syslog-ng Agent sends one message at a time from each source. That way a single source with a large log traffic does not block other log sources.