syslog-ng Premium Edition 6.0.17 - Administration Guide

Target audience and prerequisites Products covered in this guide Typographical conventions About this document

What syslog-ng is not Why is syslog-ng needed? What is new in syslog-ng Premium Edition 6 LTS? Who uses syslog-ng? Supported platforms

Chapter 2. The concepts of syslog-ng

Logging with syslog-ng Modes of operation Global objects Timezones and daylight saving Versions and releases of syslog-ng PE Licensing GPL and LGPL licenses High availability support The structure of a log message Message representation in syslog-ng PE Structuring macros, metadata, and other value-pairs Things to consider when forwarding messages between syslog-ng PE hosts NFS file system for log files

Chapter 3. Installing syslog-ng

Security-enhanced Linux: grsecurity, SELinux Installing syslog-ng using the .run installer Installing syslog-ng PE using .pkg installer Installing syslog-ng without user-interaction on Windows Upgrading syslog-ng PE Uninstalling syslog-ng PE

Chapter 4. The syslog-ng PE quick-start guide Chapter 5. The syslog-ng PE configuration file

The configuration syntax in detail Notes about the configuration syntax Global and environmental variables Logging configuration changes Modules in syslog-ng PE Managing complex syslog-ng configurations

Chapter 6. Collecting log messages — sources and source drivers

Chapter 7. Sending and storing log messages — destinations and destination drivers

Sending messages directly to Elasticsearch version 2.0 or higher Storing messages in plain-text files Storing messages on the Hadoop Distributed File System (HDFS) Publishing messages to Apache Kafka Storing messages in encrypted files Storing messages in a MongoDB database Sending messages to a remote log server using the RFC3164 protocol (network() driver) Sending messages to named pipes Sending messages to external applications Generating SMTP messages (e-mail) from logs Sending SNMP traps Storing messages in an SQL database Sending messages to a remote log server using the IETF-syslog protocol Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Sending messages to UNIX domain sockets Sending messages to a user terminal — usertty() destination

Chapter 8. Routing messages: log paths, reliability, and filters

Managing incoming and outgoing messages with flow-control Using disk-based and memory buffering Client-side failover Filters Dropping messages

Chapter 9. Global options of syslog-ng PE

Global options

Chapter 10. TLS-encrypted message transfer

Encrypting log messages with TLS Mutual authentication using TLS TLS options

Chapter 11. FIPS-compliant syslog-ng

Limitations of the FIPS-compliant syslog-ng PE Legal Notice of FIPS Compliance of Syslog-ng Premium Edition

Chapter 12. Reliable Log Transfer Protocol™

RLTP™ options Examples for using RLTP™

Chapter 13. Reliability and minimizing the loss of log messages

Flow control, no disk buffering, no RLTP™ Flow control, normal disk buffering, no RLTP™ Flow control, reliable disk buffering, no RLTP™ Flow control, reliable disk buffering, RLTP™ Deciding which loss prevention mechanism to apply

Chapter 14. Manipulating messages

Modifying messages Regular expressions

Chapter 15. Parsing and segmenting structured messages

Parsing key=value pairs The JSON parser

Chapter 16. Processing message content with a pattern database

Using pattern databases Correlating log messages Triggering actions for identified messages Creating pattern databases

Chapter 17. Statistics and metrics of syslog-ng Chapter 18. Multithreading and scaling in syslog-ng PE

Configuring multithreading Optimizing multithreaded performance

Chapter 19. Troubleshooting syslog-ng

Collecting debugging information with strace, truss, or tusc Stopping syslog-ng Reporting bugs and finding help Recover data from orphaned diskbuffer files

Chapter 20. Best practices and examples

Handling large message load Using name resolution in syslog-ng Configuring log rotation

Installing syslog-ng PE using .pkg installer

Chapter 3. Installing syslog-ng > Installing syslog-ng PE using .pkg installer

Installing syslog-ng PE using .pkg installer

The syslog-ng PE application can be installed with user-interaction or in silent mode.

The generic syslog service of the operating system is stopped and deregistered automatically by the installer before installation.

Caution:

Remove any other syslog service before starting the installation. Failing to do so might have unpredictable results.

The syslog-ng PE application is compatible with the standard Solaris logrotate mechanism by default.

To install syslog-ng PE with user-interaction, complete Procedure 3.6, “Installing syslog-ng PE with user-interaction”.
To install syslog-ng PE without any user-interaction, complete Procedure 3.7, “Installing syslog-ng PE without user-interaction”.
To install syslog-ng PE from a transformed PKG package, complete Procedure 3.8, “Installing syslog-ng PE from a transformed PKG package”.

Procedure 3.6. Installing syslog-ng PE with user-interaction

Purpose:

The syslog-ng PE application can be installed in an interactive way.

Steps:

Unpack the package with the following command:

gunzip syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg.gz

Run the following command:

pkgadd -d syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg BBsyslng

Answer the questions with y or n. Besides these, the following commands are available: ? displays the help, q exits the installer. The installer generates a default configuration file for syslog-ng PE based on the answers (if an old configuration file is not used):
- the path of the license file
- whether an old configuration file is to be used (if the answer is yes, the rest of the questions will be skipped)
- if a new configuration file is requested, whether syslog-ng PE will send or receive logs over the network
The installer finishes the installation and the replaces the default syslog service with syslog-ng PE.

Procedure 3.7. Installing syslog-ng PE without user-interaction

Purpose:

The syslog-ng PE application can be installed in silent mode without any user-interaction by specifying the required parameters from the command line. Answers to every question of the installer can be set in advance using command-line parameters.

Steps:

Generate a response file that contains the answers to the questions.

pkgask -r <responsefile> -d syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg BBsyslng

For example:

pkgask -r response_file -d syslog-ng-premium-edition-4.2.1-solaris10-amd64.pkg BBsyslng

Caution:

The pkgask will fail if the response file already exists.

Run the pkgadd command with the added response file answering y to all questions.

yes | pkgadd -r <responsefile> -d syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg BBsyslng

Caution:

pkgask runs the request script of the package under the current condition of the operating system. The installation might fail if something changes (another syslog-ng PE package has been installed, configuration file has been removed), because the response file's contents will be invalid.

Procedure 3.8. Installing syslog-ng PE from a transformed PKG package

Purpose:

The syslog-ng PE application can be installed from a normal file structure. To perform this, transform the .pkg package.

Steps:

To transform the .pkg package, execute the following command:
```
pkgtrans syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg <outdirectory>
```
In this case, the .pkg package will be extracted to the <outdirectory> directory.
Example 3.1. Extracting syslog-ng PE from a transformed PKG package
```
pkgtrans syslog-ng-premium-edition-4.2.0-solaris-9-sparc.pkg /tmp/out
```
This will extract the files to the /tmp/out/BBsyslng directory.
To install syslog-ng PE, execute the following command:
```
pkgadd <options> -d <outdirectory> BBsyslng
```
In this case, <options> stand for the pkgadd options, for example -r <responsefile>.

Installing syslog-ng without user-interaction on Windows

Chapter 3. Installing syslog-ng > Installing syslog-ng without user-interaction on Windows

The syslog-ng Premium Edition application can be installed in silent mode as well, without requiring any user interaction. The various installer options can be specified as command-line options. Using the /S option is required. The following options are available:

/A

Start the syslog-ng PE service (/A=yes) when the installation is finished. The syslog-ng PE service will start only if a valid configuration file is installed as well.

NOTE:

Only one syslog-ng can be installed as service.

/D=<path>

Install the syslog-ng PE into the specified folder (%INSTALLDIR%).

Caution:

If you use the /D option, make sure that this is the last option in the command-line. For example: syslog-ng-<version>-setup.exe /S /D=c:\syslog-ng\

/F=<path>

Enter the path to the syslog-ng PE configuration file. The installer will copy the specified configuration file to %INSTALLDIR%\etc\syslog-ng.conf.

Caution:

The syslog-ng PE application will not use the specified file directly, but copy it to %INSTALLDIR%\etc\syslog-ng.conf. To modify the configuration of syslog-ng PE, edit the %INSTALLDIR%\etc\syslog-ng.conf file. By default, syslog-ng PE is installed into the C:\Program Files\syslog-ng directory.

/L=<path>

Enter the path to the license file. The installer will copy the specified license file to %INSTALLDIR%\etc\license.txt.

/M

Add entries about syslog-ng PE to the Start menu.

/P

Only unpack syslog-ng PE into the specified destination folder. If this option is enabled, (/P=yes), the installer will ignore other options (except for /D that specifies the destination folderh).

/R

/S

Start the installer in silent mode. This option is required for the silent installation.

Upgrading syslog-ng PE

Chapter 3. Installing syslog-ng > Upgrading syslog-ng PE

Upgrading syslog-ng PE

This section describes the possible upgrade paths of syslog-ng PE.

To see the possible upgrade paths, see the section called “Upgrading from previous syslog-ng PE versions to 6 LTS”.
To upgrade an existing syslog-ng PE installation, see Procedure 3.11, “Upgrading to syslog-ng PE 6 LTS”.

Upgrading from previous syslog-ng PE versions to 6 LTS

Upgrading is supported from the following syslog-ng PE versions:

syslog-ng PE 5 LTS (5.0.x)
syslog-ng PE 5 F6 (5.6.x)

To upgrade an existing syslog-ng PE installation, see Procedure 3.11, “Upgrading to syslog-ng PE 6 LTS”.

Procedure 3.11. Upgrading to syslog-ng PE 6 LTS

Purpose:

To upgrade to syslog-ng PE 6 LTS, complete the following steps:

Steps:

Download the new installer package. Use the same package type as you used for the installation (for example, use the .run package for the upgrade if you have originally installed syslog-ng PE using a .run installer).
Install syslog-ng PE and check the warnings. Upgrade the respective parts of your configuration if needed.
Set the version of the configuration file to 6.0.