Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.29 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng PE The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files google-pubsub: collecting messages from the Google Pub/Sub messaging service wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mssql, oracle, sql: collecting messages from an SQL database network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP without Java kafka(): Publishing messages to Apache Kafka (Java implementation) (DEPRECATED) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel(): Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql(): Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

Upgrading syslog-ng PE to other package versions

This scenario is not supported and will fail with the following error messages.

Upgrading from platform-specific package to .run
Upgrading from rpm package to .run package

Unsupported. Installation stops and the following error message is displayed:

Incompatible syslog-ng package already installed
Upgrading from deb package to .run package

Unsupported. Installation stops and the following error message is displayed:

Incompatible syslog-ng package already installed
Upgrading from pkg package to .run package

Unsupported. Installation stops and the following error message is displayed:

Incompatible syslog-ng package already installed to <syslog-ng path>
Upgrading from .run to a platform-specific package

This scenario is not supported and will fail with the following error messages. To replace a .run package with a platform-specific package, create a backup of your configuration and persist files, uninstall the .run package using the --purge option, then install the platform-specific package.

Upgrading from .run package to rpm package

Unsupported. Installation stops and the following error message is displayed:

Incompatible standalone (.run) installer of syslog-ng Premium Edition

 Caution:

Hazard of data loss! Installing rpm package syslog-ng PE on AIX platform is possible even if the upgrade conditions are not met, since the rpm package installs before checking the upgrade conditions and therefore no error message is displayed. This might result in overwriting the old configuration file.

Upgrading from .run package to deb package

Unsupported. Installation stops and the following error message is displayed:

Errors were encountered while processing
Upgrading from .run package to pkg package

Unsupported. Installation stops and the following error message is displayed:

Please remove the conflicting package before installing this package. Installation aborted.

Upgrading from syslog-ng PE to syslog-ng OSE

Upgrading from syslog-ng PE to syslog-ng OSE is unsupported since it counts as downgrading.

Upgrade from syslog-ng OSE to syslog-ng PE

If you wish to upgrade your existing syslog-ng OSE installation to syslog-ng PE, there are a number of considerations to keep in mind. This section highlights the main differences between syslog-ng OSE and syslog-ng PE that are useful to know before performing an upgrade. It also provides you with step-by-step instructions on how to do the upgrade.

Feature differences

syslog-ng OSE includes certain features that are highly experimental, require special external dependencies, or are important only to a very limited set of users. While syslog-ng PE is built from the same code base, it includes only a subset of syslog-ng OSE features. Those that are well tested and represent commercial value. These features are commercially supported as they are covered by automated end-to-end tests, which make sure that they not only compile but work correctly on many different platforms.

This means that your syslog-ng OSE installation may contain features that are not part of syslog-ng PE, or if they are, they may not have been tested.

Packaging

The packaging of syslog-ng OSE and syslog-ng PE also differ greatly.

With syslog-ng OSE, distribution packages do not bundle dependencies and only include features for which dependencies are available within the distribution. Packaging is modular to make sure that you install only a minimal set of extra dependencies. In addition, the naming and content of subpackages varies between distributions, and there are also unofficial syslog-ng OSE packages enabling more features than available in official distribution packages.

In the case of syslog-ng PE, all dependencies are included in a single package either in a distribution specific format (rpm or deb) or in a generic .run installer.

Upgrading from syslog-ng OSE to syslog-ng PE

The cleanest way to upgrade from syslog-ng OSE to syslog-ng PE is to remove the syslog-ng OSE package from the system. This way you can avoid the packaging conflicts and feature differences.

In the example procedure provided here, we describe an upgrade of syslog-ng OSE version 3.12 from unofficial repositories running on Red Hat Enterprise Linux 7.4 to syslog-ng PE version 7.0.4. The process should work in a fairly similar way when using other OS or syslog-ng versions.

To upgrade from syslog-ng OSE to syslog-ng PE

  1. Remove syslog-ng OSE.

    The following instructions assume that the user is in the /root directory.

    1. Unless you have not touched the syslog-ng configuration at all, make a backup of syslog-ng.conf first. Copy the contents of /etc/syslog-ng to a directory under /root (or where you can find it), so you have a backup you can work from later:

      cp -R /etc/syslog-ng sngose

    2. Remove the syslog-ng package and dependent subpackages:

      yum erase syslog-ng

    3. Remove the /etc/syslog-ng directory:

      rm -fr /etc/syslog-ng

      		 Caution:

      Check the output of yum carefully. If there are any applications listed other than syslog-ng and subpackages, remove syslog-ng using rpm -e –€”nodeps, so dependent packages are not removed.

  2. Install syslog-ng PE.

    The following instructions assume that the syslog-ng PE rpm package is available in the current directory. You can install syslog-ng PE using the following command:

    [root@localhost ~]# rpm -Uvh syslog-ng-premium-edition-compact-7.0.5-1.rhel7.x86_64.rpm
Preparing...                          ################################# [100%]
Trying to stop syslog services on Linux, using systemd services.
Updating / installing...
   1:syslog-ng-premium-edition-compact################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/syslog-ng.service to /usr/lib/systemd/system/syslog-ng.service.
[root@localhost ~]#

  3. Merge configurations.

    The configuration file of the freshly installed syslog-ng PE is available under /opt/syslog-ng/etc/syslog-ng.conf. Start by making a backup of it.

    The next steps largely depend on the particulars of your previous syslog-ng OSE configuration and what you want to achieve:

    1. Append your old OSE configuration to /opt/syslog-ng/etc/syslog-ng.conf.

    2. Edit out redundant configuration parts, for example, a version declaration.

    3. Edit out those configuration parts that refer to features unavailable in syslog-ng PE, such as the Riemann destination.

      If you try to start syslog-ng PE with an unknown feature enabled, it fails with a similar error message (in the example, it is the Riemann destination that is causing the error):

      /opt/syslog-ng/sbin/syslog-ng -s
Error parsing destination, destination plugin riemann not found in /opt/syslog-ng/etc/syslog-ng.conf at line 41, column 2:

    riemann(
    ^^^^^^^

    4. Syntax check your configuration using the -s option of syslog-ng. Make sure that you use the full path to syslog-ng PE, or add it to the PATH:

      /opt/syslog-nb/sbin/syslog-ng -s

    5. If no errors are found, stop syslog-ng:

      systemctl stop syslog-ng

    6. Try to start syslog-ng from the command line in the foreground using the -F option, so you can see any errors:

      /opt/syslog-ng/sbin/syslog-ng -F

      Some common error messages and explanations:

      • syslog-ng OSE uses s_sys for references to local system sources, while syslog-ng PE uses s_local. Remember to rename such references, otherwise a similar error message will be displayed:

        [2017-10-03T14:04:18.968550] Error resolving reference; content='source', name='s_sys', location='/opt/syslog-ng/etc/syslog-ng.conf:86:2'

      • Some features of syslog-ng PE require a license file to be present. In the example shown here, a Java plugin failed to initialize due to a missing license:

        [2017-10-03T14:07:05.894534] syslog-ng running in client/relay mode, cannot initialize plugin; plugin name='java'
[2017-10-03T14:07:05.894560] Error initializing message pipeline; plugin name='java', location='#buffer:2:3'

      Once you have made sure that your configuration works fine, you do not have to start syslog-ng in the foreground anymore.

    7. Stop syslog-ng using Ctrl-C.

    8. Start syslog-ng as a service using systemctl start syslog-ng.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating