Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.34 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng PE The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files google-pubsub: collecting messages from the Google Pub/Sub messaging service wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mssql, oracle, sql: collecting messages from an SQL database network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6,udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2>: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files google_bigquery(): Sending logs to a Google BigQuery table google_bigquery_managedaccount(): Sending logs to a Google BigQuery table authenticated by Google Cloud managed service account google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service google_pubsub-managedaccount(): Sending logs to the Google Cloud Pub/Sub messaging service authenticated by Google Cloud managed service account hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka(): Publishing messages to Apache Kafka (Java implementation) (DEPRECATED) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel(): Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql(): Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transport Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

Upgrading from syslog-ng PE to syslog-ng OSE

Upgrading from syslog-ng PE to syslog-ng OSE is unsupported since it counts as downgrading.

Upgrade from syslog-ng OSE to syslog-ng PE

If you wish to upgrade your existing syslog-ng OSE installation to syslog-ng PE, there are a number of considerations to keep in mind. This section highlights the main differences between syslog-ng OSE and syslog-ng PE that are useful to know before performing an upgrade. It also provides you with step-by-step instructions on how to do the upgrade.

Feature differences

syslog-ng OSE includes certain features that are highly experimental, require special external dependencies, or are important only to a very limited set of users. While syslog-ng PE is built from the same code base, it includes only a subset of syslog-ng OSE features. Those that are well tested and represent commercial value. These features are commercially supported as they are covered by automated end-to-end tests, which make sure that they not only compile but work correctly on many different platforms.

This means that your syslog-ng OSE installation may contain features that are not part of syslog-ng PE, or if they are, they may not have been tested.

Packaging

The packaging of syslog-ng OSE and syslog-ng PE also differ greatly.

With syslog-ng OSE, distribution packages do not bundle dependencies and only include features for which dependencies are available within the distribution. Packaging is modular to make sure that you install only a minimal set of extra dependencies. In addition, the naming and content of subpackages varies between distributions, and there are also unofficial syslog-ng OSE packages enabling more features than available in official distribution packages.

In the case of syslog-ng PE, all dependencies are included in a single package either in a distribution specific format (rpm or deb) or in a generic .run installer.

Upgrading from syslog-ng OSE to syslog-ng PE

The cleanest way to upgrade from syslog-ng OSE to syslog-ng PE is to remove the syslog-ng OSE package from the system. This way you can avoid the packaging conflicts and feature differences.

In the example procedure provided here, we describe an upgrade of syslog-ng OSE version 3.12 from unofficial repositories running on Red Hat Enterprise Linux 7.4 to syslog-ng PE version 7.0.4. The process should work in a fairly similar way when using other OS or syslog-ng versions.

To upgrade from syslog-ng OSE to syslog-ng PE

  1. Remove syslog-ng OSE.

    The following instructions assume that the user is in the /root directory.

    1. Unless you have not touched the syslog-ng configuration at all, make a backup of syslog-ng.conf first. Copy the contents of /etc/syslog-ng to a directory under /root (or where you can find it), so you have a backup you can work from later:

      cp -R /etc/syslog-ng sngose
    2. Remove the syslog-ng package and dependent subpackages:

      yum erase syslog-ng
    3. Remove the /etc/syslog-ng directory:

      rm -fr /etc/syslog-ng

      Caution:

      Check the output of yum carefully. If there are any applications listed other than syslog-ng and subpackages, remove syslog-ng using rpm -e –€”nodeps, so dependent packages are not removed.

  2. Install syslog-ng PE.

    The following instructions assume that the syslog-ng PE rpm package is available in the current directory. You can install syslog-ng PE using the following command:

    [root@localhost ~]# rpm -Uvh syslog-ng-premium-edition-compact-7.0.5-1.rhel7.x86_64.rpm
    Preparing...                          ################################# [100%]
    Trying to stop syslog services on Linux, using systemd services.
    Updating / installing...
       1:syslog-ng-premium-edition-compact################################# [100%]
    Created symlink from /etc/systemd/system/multi-user.target.wants/syslog-ng.service to /usr/lib/systemd/system/syslog-ng.service.
    [root@localhost ~]#
  3. Merge configurations.

    The configuration file of the freshly installed syslog-ng PE is available under /opt/syslog-ng/etc/syslog-ng.conf. Start by making a backup of it.

    The next steps largely depend on the particulars of your previous syslog-ng OSE configuration and what you want to achieve:

    1. Append your old OSE configuration to /opt/syslog-ng/etc/syslog-ng.conf.

    2. Edit out redundant configuration parts, for example, a version declaration.

    3. Edit out those configuration parts that refer to features unavailable in syslog-ng PE, such as the Riemann destination.

      If you try to start syslog-ng PE with an unknown feature enabled, it fails with a similar error message (in the example, it is the Riemann destination that is causing the error):

      /opt/syslog-ng/sbin/syslog-ng -s
      Error parsing destination, destination plugin riemann not found in /opt/syslog-ng/etc/syslog-ng.conf at line 41, column 2:
      
          riemann(
          ^^^^^^^
    4. Syntax check your configuration using the -s option of syslog-ng. Make sure that you use the full path to syslog-ng PE, or add it to the PATH:

      /opt/syslog-nb/sbin/syslog-ng -s
    5. If no errors are found, stop syslog-ng:

      systemctl stop syslog-ng
    6. Try to start syslog-ng from the command line in the foreground using the -F option, so you can see any errors:

      /opt/syslog-ng/sbin/syslog-ng -F

      Some common error messages and explanations:

      • syslog-ng OSE uses s_sys for references to local system sources, while syslog-ng PE uses s_local. Remember to rename such references, otherwise a similar error message will be displayed:

        [2017-10-03T14:04:18.968550] Error resolving reference; content='source', name='s_sys', location='/opt/syslog-ng/etc/syslog-ng.conf:86:2'
      • Some features of syslog-ng PE require a license file to be present. In the example shown here, a Java plugin failed to initialize due to a missing license:

        [2017-10-03T14:07:05.894534] syslog-ng running in client/relay mode, cannot initialize plugin; plugin name='java'
        [2017-10-03T14:07:05.894560] Error initializing message pipeline; plugin name='java', location='#buffer:2:3'

      Once you have made sure that your configuration works fine, you do not have to start syslog-ng in the foreground anymore.

    7. Stop syslog-ng using Ctrl-C.

    8. Start syslog-ng as a service using systemctl start syslog-ng.

Upgrading from complete syslog-ng PE to client setup version of syslog-ng PE

The installer displays the following message if you try to upgrade from complete syslog-ng PE to client setup syslog-ng PE with .run package.

This version of syslog-ng Premium Edition doesn't support storing messages in SQL servers, while the installed one did.
Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating