Customizing the message format
The format of the messages received from the eventlog and the file sources can be customized using templates. You can define separate message format for the eventlog and the file sources. If you have multiple destination servers configured, you can define separate templates for each server. When creating a template to customize the message format, you can use macros, all alphanumeric characters, and the following special characters: <>,():;-+/_.
By default, syslog-ng Agent uses the following templates to forward messages:
-
For the BSD protocol: <${PRI}>${BSDDATE} ${HOST} ${MSGHDR}${MESSAGE}
-
For messages read from the eventlog, the $MESSAGE part is ${EVENT_USERNAME}: ${EVENT_NAME} ${EVENT_SOURCE}: [${EVENT_TYPE}] ${EVENT_MSG} (EventID ${EVENT_ID}) for every protocol.
-
For messages read from a file, the $MESSAGE part is $FILE_NAME: $FILE_CURRENT_POSITION/$FILE_SIZE: $FILE_MESSAGE for every protocol.
Procedure 8.1. Customizing messages using templates
Purpose:
To create a template, complete the following procedure:
|
|
Caution:
These macros are available only in the syslog-ng Agent for Windows. To recognize Windows-specific elements of the log message (for example eventlog-related macros) on the syslog-ng server, you have to use parsers on the syslog-ng server. The parser must be configured to match the message format set in the syslog-ng Agent. |
Steps:
-
Start the configuration interface of the syslog-ng Agent for Windows application.
-
Select . Select your log server, and click .
-
To change the format of messages received from eventlog sources, type the message format you want to use into the field.
To change the format of messages received from file sources, type the message format you want to use into the field.
Do not forget to add the $ character before macros. For a complete list of the available macros, see the section called “Macros available in the syslog-ng Agent”.
For example, to send the messages in the DATE HOSTNAME MESSAGE format, type Date:$DATE Hostname:$HOST Logmessage:$MESSAGE.
Note that the $MESSAGE macro contains not only the text of the log message, but also additional information received from the message source, such as the name of the eventlog container, or the file, as set in the eventlog-specific and file-specific templates.
|
|
NOTE:
Templates are assigned to a single destination server, so it is possible to use different templates for different servers. However, a server and its failover servers always receive the same message. |
|
|
Caution:
If you have more than one destination servers configured (separate servers, not in failover mode), and you want to use the same template for every server, you must manually copy the template into the configuration of each server. Template modifications are not applied automatically to every server. |
-
Click .
-
To activate the changes, restart the syslog-ng Agent service.
Customizing the timestamp used by the syslog-ng Agent
The syslog-ng Agent can send the syslog messages using either the ISO or the BSD timestamp format. It is recommended to use the ISO format, because it contains much more information than the BSD format.
Note that in the syslog-ng Agent, the macros without prefix (for example DATE) always refer to the receiving date of the message (for example R_DATE) when it arrived into the event log container, and are included only for compatibility reasons.
|
|
Caution:
If a remote host is logging into the event log of the local host that is running syslog-ng Agent for Windows, both hosts have to be in the same timezone, because the event log message does not include the timezone information of the sender host. Otherwise, the date of the messages received from the remote host will be incorrect. |
The following sections list the available macros:
|
|
Caution:
These macros are available only in the syslog-ng Agent for Windows. To recognize Windows-specific elements of the log message (for example eventlog-related macros) on the syslog-ng server, you have to use parsers on the syslog-ng server. The parser must be configured to match the message format set in the syslog-ng Agent. |
|
|
NOTE:
Note that if you use the Syslog protocol template (meaning that messages are sent using the IETF-syslog protocol), only the message part of the log message can be customized, the structure of the headers and other information is fixed by the protocol. |
Protocol-related macros of the syslog-ng Agent
Description: Name of the host sending the message. Hostnames are automatically converted to lowercase.
Description: The content of the message, including the text of the message and any file- or event-specific macros that are set for the source.
Description: An alias for the MESSAGE macro.
Description: The name and the PID of the program that sent the log message in PROGRAM[PID]: format. Includes a trailing whitespace. Note that the macro returns an empty value if both the PROGRAM and PID fields of the message are empty.
Description: Priority header of the message, storing the facility and the level of the message.
Description: PID of the application the message came from.
Time-related macros of the syslog-ng Agent
BSDDATE, R_BSDDATE, S_BSDDATE
Description: Date of the message in BSD timestamp format (month/day/hour/minute/second, each expressed in two digits). This is the original syslog time stamp without year information, for example Jun 13 15:58:00. If possible, it is recommended to use ISODATE for timestamping.
Description: Date of the message using the BSD-syslog style timestamp format (month/day/hour/minute/second, each expressed in two digits). This is the original syslog time stamp without year information, for example: Jun 13 15:58:00.
Description: The day the message was sent.
FULLDATE, R_FULLDATE, S_FULLDATE
Description: A nonstandard format for the date of the message using the same format as ${DATE}, but including the year as well, for example: 2006 Jun 13 15:58:00.
Description: The hour of day the message was sent.
ISODATE, R_ISODATE, S_ISODATE
Description: Date of the message in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00. If possible, it is recommended to use ISODATE for timestamping. Note that the syslog-ng Agent cannot produce fractions of a second (for example milliseconds) in the timestamp.
Description: The minute the message was sent.
Description: The month the message was sent as a decimal value, prefixed with a zero if smaller than 10.
MONTHNAME, R_MONTHNAME, S_MONTHNAME
Description: The English name of the month the message was sent, abbreviated to three characters (for example Jan, Feb, and so on).
Description: The second the message was sent.
TZOFFSET, R_TZOFFSET, S_TZOFFSET
Description: The time-zone as hour offset from GMT, for example: -07:00. In syslog-ng 1.6.x this used to be -0700 but as ${ISODATE} requires the colon it was added to ${TZOFFSET} as well.
UNIXTIME, R_UNIXTIME, S_UNIXTIME
Description: Standard UNIX timestamp, represented as the number of seconds since 1970-01-01T00:00:00.
Description: The year the message was sent.
Description: The week number of the year, prefixed with a zero for the first nine week of the year. (The first Monday in the year marks the first week.)
WEEKDAY, R_WEEKDAY, S_WEEKDAY
Description: The 3-letter name of the day of week the message was sent, for example Thu.
Eventlog-related macros of the syslog-ng Agent
Description: The category of the event.
Description: Empty macro, does not contain any data.
Description: The number of received messages per container since the syslog-ng Agent for Windows was started.
Description: The facility that sent the message.
Description: A unique ID for messages generated at reception time on the receiving host. It facilitates defining relationships between messages that are potentially distributed to different files on the same host, or different hosts.
Description: The name of the host that sent the log message.
Description: The identification number of the event.
Description: Importance level of the message represented as a number: 6 - Success, 5 - Informational, 4- Warning, or 3 - Error).
Description: The content of the message.
Description: Contains the entire message in XML format.
Description: The content of the message. This is an alias of the EVENT_MESSAGE.
Description: Contains the entire message in XML format. This is an alias of the EVENT_MESSAGE_XML.
Description: Name of the Windows event log container (for example Application or Security).
Description: Name of the service that generated the log message.
Description: The record number of the event in the event log.
Description: The security identification number of the event.
Description: The security identification number resolved into name. One of the following: User, Group, Domain, AliasWellKnownGroup, DeletedAccount, Invalid, Unknown, Computer.
Description: The application that created the message.
Description: The task category of the event.
Description: The importance level of the message in text format.
Description: The user running the application that created the message.
File-related macros of the syslog-ng Agent
Description: The position of the message from the beginning of the file in bytes.
Description: The facility that sent the message.
Description: Importance level of the message represented as a number: 6 - Success, 5 - Informational, 4- Warning, or 3 - Error).
Description: The content of the message.
Description: The content of the message. This is an alias of the FILE_MESSAGE macro.
Description: Name of the log file (including its path) from where the syslog-ng PE received the message.
Description: The current size of the file in bytes.
Controlling the syslog-ng Agent services
During installation, syslog-ng Agent registers the syslog-ng Agent service that is started automatically when the host boots. To disable the automatic startup of the syslog-ng Agent use the interface. The service is running with the privileges of the NT AUTHORITY\SYSTEM user.
To manually start or stop the service use the interface, or navigate to . Note that in the latter case if User Access Control (UAC) is enabled, you need the Run as Administrator privilege to start or stop the syslog-ng Agent.
When the syslog-ng Agent service is started or stopped, it sends a syslog message to the central log server and an eventlog message to the Application eventlog container of the host.
|
|
Caution:
If you change the timezone setting of the host while the syslog-ng Agent is running, you have to restart the syslog-ng Agent. Otherwise, it will not receive the updated timezone information and the date of the events will be incorrect. |
|
|
NOTE:
It is possible to run the service with an administrator account that has "log on as service" rights (to set user rights, navigate to ). These settings are unsupported, use them only at your own risk. Also note that during the next upgrade procedure, these settings will be overwritten by factory default settings. |
The syslog-ng Agent for Windows application has the following command-line options:
|
|
NOTE:
Command-line options are case-insensitive. The options consist of a single letter introduced by either "-" or "/". |
|
|
NOTE:
Command-line options will only work with administrator permission. |
- /c
-
Start the syslog-ng Agent using the specified XML configuration file.
- /d
-
Start the syslog-ng Agent in debug mode.
- /h
-
Display a help message about the command-line options.
- /i
-
Install the syslog-ng Agent service into the services list.
- /r
-
Remove the syslog-ng Agent service from the services list.
- /v
-
Display version information.
- /x
-
Validate XML configuration file without importing it.
To use these options, select , navigate to the directory where the syslog-ng Agent is installed (for example cd C:\Windows\Program Files\BalaBit\syslog-ng Agent\), and execute the syslog-ng-agent.exe file with the required option.
Example 9.1. Using command line options
To start syslog-ng Agent in debug mode:
syslog-ng-agent.exe /d
To start syslog-ng Agent with XML configuration file:
syslog-ng-agent.exe /c C:\ConfigFiles\syslog-ng-agent-conf.xml
To register syslog-ng Agent as a service using XML configuration file:
syslog-ng-agent.exe /i C:\ConfigFiles\syslog-ng-agent-conf.xml
Troubleshooting syslog-ng Agent for Windows
In case you experience problems with the syslog-ng Agent for Windows application, the following points can be of help.
|
|
NOTE:
The followings address only problems specific to the syslog-ng Agent, and assume that communication between the server and the client is otherwise possible (that is, the server is properly configured to receive messages and is available on the network, and name resolution is properly configured on the client). |
Configuration changes do not take effect:
Configuration changes take effect only after restarting the syslog-ng service or rebooting the system. Also restart the system after changing the timezone settings of the host, or importing a certificate that you want to use to authenticate the communication between the agent and the server. If the configuration of the agent has changed since the last restart, the syslog-ng Agent sends a message of the change, including the hmac-sha-1 hash of the new configuration.
Also note that if your clients are managed from a Domain Controller, configuration changes are not instantly downloaded to the client hosts, only at the time of the next group policy update. To update the configuration of a client host earlier, open a command prompt on the client host, and issue the gpupdate /force command.
After downloading the configuration from the Domain Controller, the syslog-ng Agent service is automatically restarted if the configuration has changed.
|
|
NOTE:
Certain domain settings that can affect the syslog-ng Agent are downloaded only when the machine is rebooted. For example, moving the computer from one group policy to another requires a reboot to have effect. |
The syslog-ng Agent does not send messages to the server:
Check the Application eventlog for messages of the syslog-ng Agent. In case of connection errors and certificate problems, the syslog-ng Agent sends error messages into the eventlog. Ensure that the destination address of the server is correctly set. If you use SSL encryption, verify that the certificate of the Certificate Authority of the server and that the certificate of the client are properly imported. If there are no error messages, check the logs on your log server: the syslog-ng Agent sends a MARK message every ten minutes even if there are no other messages to send (unless you have disabled MARK messages).
The syslog-ng Agent sends only MARK messages to the server:
Verify that you have configured the eventlog and file sources, and that they have not been disabled globally. If these settings are correct but the server still does not send any messages, temporarily disable all filters to see that they are not configured to ignore every message. When using filter, it is also recommended to check the global case-sensitivity settings.
The hostname used in the messages changes: If a host is sometimes logged in into a domain and sometimes it is not, its hostname might reflect this. To avoid this situation, select and double-click on . Enable , navigate to and select . This causes syslog-ng Agent to resolve its own hostname from DNS and use the resolved FQDN in the syslog messages. For details, see Procedure 5.8, “Configuring the hostname format”.
Command-line parameters are ignored:
Command-line parameters work only for administrators if User Account Control (UAC) is enabled. To execute syslog-ng Agent with command-line parameters, select , right-click on .
If you contact our Support Team about a problem with the syslog-ng Agent for Windows, execute the syslog-ng-agent -V command from the command line and include every version and platform information it displays in your support request.
CPU load is high: See the section called “Sending messages and CPU load”.
Losing messages from eventlog containers:
An eventlog container is a special file. The Agent reads this file, formats the messages and sends them to remote log server. Note that the eventlog container can be configured only to a certain size. If the container reaches that size, Windows writes the next message to the beginning of the file. As a result, if the agent is not running (or the destination server is unavailable) so long that the eventlog container is filled up, messages can be lost.
Logs are not forwarded instantly:
For the logs of certain applications (for example, Internet Information Services (IIS) for Windows Server), the syslog-ng Agent for Windows application does not forward the log messages in real time, only in batches after a certain amount of time. The cause of the problem is that the Windows operating system does not immediately flush its buffers to the file when an application sends a log message. The syslog-ng Agent for Windows application immediately starts sending the log messages when they become available in the log file.
Sending messages and CPU load
The syslog-ng Agent application can send messages to the server when the Windows Scheduler provides resources to the syslog-ng Agent. When there are many unsent log messages in the log sources, and there is no other significant activity on the host, syslog-ng will start to send the messages to the server, possibly increasing the CPU load to 100%. After all messages have been sent, or if another application requires the resources, the CPU load decreases back to normal.
When relaying the messages from multiple sources, the syslog-ng Agent sends one message at a time from each source. That way a single source with a large log traffic does not block other log sources.
