Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.29 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng PE The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files google-pubsub: collecting messages from the Google Pub/Sub messaging service wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mssql, oracle, sql: collecting messages from an SQL database network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP without Java kafka(): Publishing messages to Apache Kafka (Java implementation) (DEPRECATED) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel(): Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql(): Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transfer Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

Upgrading from complete syslog-ng PE to client setup version of syslog-ng PE

The installer displays the following message if you try to upgrade from complete syslog-ng PE to client setup syslog-ng PE with .run package.

This version of syslog-ng Premium Edition doesn't support storing messages in SQL servers, while the installed one did.

Upgrading the sql() source of syslog-ng PE

This section describes how you can upgrade your sql() source between syslog-ng Premium Edition (syslog-ng PE) 6 LTS and syslog-ng PE 7 LTS.

For more information about the mssql(), oracle(), or sql() sources, see mssql, oracle, sql: collecting messages from an SQL database.

To upgrade the sql() source between syslog-ng PE 6 LTS and syslog-ng PE 7 LTS

  1. Extract the value of last_read_uid from the persist file by using persist tool.

    For more information on using the persist tool, see the The syslog-ng manual pages.

  2. Add the extracted last_read_uid value to the syslog-ng PE configuration in a start-uid(value) format.
Example: adding the value of last_read_uid from the persist file into the configuration as start-uid(value)

You can add the value of last_read_uid that you extracted from the persist file into your configuration as start-uid(value)with the following method:

$ /opt/syslog-ng/bin/persist-tool dump /opt/syslog-ng/var/syslog-ng.persist
hostid = { "value": "65 D5 E1 06" }

freetds,mssql-host,1433,test_database,test_table = { "version": 0, "big_endian": false, "last_read_id": "2" }

run_id = { "value": "01 00 00 00" }

In the example above, the value is "2". The parameter in the configuration should look like start-uid("2").

Differences in configuration

This section describes the sql() source-related differences in configuration between syslog-ng PE 6 LTS and syslog-ng PE 7 LTS, including the features not ported from syslog-ng PE 6 LTS and the features that changed in usage since syslog-ng PE 6 LTS.

For more information about the mssql(), oracle(), or sql() sources, see mssql, oracle, sql: collecting messages from an SQL database.

Features not ported
  • Only MSSQL and Oracle are supported in syslog-ng PE 7 LTS. MySQL and PostGreSQL users cannot upgrade to syslog-ng PE 7 LTS.
  • archive-query(): It needs to be removed from configuration, because it is not supported in syslog-ng PE 7 LTS. Users need to find another way to cleanup old records.
  • read-old-records(): This option is not supported in 7 LTS, as its functionality has been replaced by the start-uid() option. If you follow To upgrade the sql() source between syslog-ng PE 6 LTS and syslog-ng PE 7 LTS, your configuration will have a start-uid() in the appropriate configuration level and no specific action will be necessary.
Changes in features

  • default-facility()

    The default value is user instead of local0.

    To achieve original behavior, you have to set default-facility(local0) explicitly in your configuration.

  • default-priority()

    The default value is notice instead of info.

    In syslog-ng PE 7 LTS, default-priority has been renamed to default-severity(). For compatibility reasons, the original default-priority() option can be still used. However, One Identity recommends replacing it with default-severity() instead.

    To achieve original behavior, you have to set default-severity(info) explicitly in your configuration.

  • follow-freq()

    The default value of follow-freq() has changed to 60 seconds (1 minute). The original default value was 10 seconds in syslog-ng PE 6 LTS.

    In syslog-ng PE 7 LTS, if time-reopen() is set, then syslog-ng PE will use the value you set. If neither time-reopen(), nor follow-freq() is set, the default value is 60 seconds.

    To achieve original behavior, you have to set follow-freq(10) explicitly in your configuration.

  • host-template()

    In syslog-ng PE 6 LTS, it was mandatory to set keep-hostname(yes) for host-template() to work. In syslog-ng PE 7 LTS, syslog-ng PE will automatically set keep-hostname(yes) even if you do not set it in your configuration. If you set host-template(), but also set keep-hostname(no), syslog-ng PE warns you about it, but otherwise ignores the keep-hostname() setting.

    In 6 LTS, syslog-ng PE allowed empty host-template() options in configuration. In syslog-ng PE 7 LTS, the template is mandatory in host-template(template). The syslog-ng PE application will not start with such configuration. As a result, you have to delete empty host-template() options from your configuration.

  • message-template()

    In 6 LTS, syslog-ng PE allowed empty message-template() options in configuration. In syslog-ng PE 7 LTS, the template is mandatory in message-template(template). The syslog-ng PE application will not start with such configuration. As a result, you have to delete empty message-template() options from your configuration.

  • program-template()

    In syslog-ng PE6 LTS, allowed empty program-template() options in configuration. In syslog-ng PE 7 LTS, the template is mandatory in program-template(template). The syslog-ng PE application will not start with such configuration. As a result, you have to delete empty program-template() options from your configuration.

  • prefix()

    You have to add a trailing dot to the prefix(). For example, if prefix(".sql") was provided in the configuration, the same option should look like prefix(".sql.") in syslog-ng PE 7 LTS.

    If the prefix does not end with a trailing dot, syslog-ng PE 7 LTS will add it automatically, and emit a warning about it.

    The default value has been changed to .sql. to reflect this change.

  • read-old-records()

    • read-old-records(yes) works the same way as in syslog-ng PE 6 LTS.

    • read-old-records(no): In syslog-ng PE 6 LTS, whenever syslog-ng PE starts, the application will only read those entries that are created after start. This means between restarts or reload, message loss is possible. In 7 LTS, syslog-ng PE will store the id of the message that was fetched last time in the persist file, even in case of read-old-records(no). When syslog-ng PE starts, it will continue from that id. This means that read-old-records() only takes effect when syslog-ng PE first starts. After restarts or reloads, read-old-records() is ignored.

Uninstalling syslog-ng PE

If you need to uninstall syslog-ng PE for some reason, you have the following options:

  • If you have installed syslog-ng PE from a .deb package: Execute the dpkg -r syslog-ng-premium-edition command to remove syslog-ng, or the dpkg -P syslog-ng-premium-edition command to remove syslog-ng PE and the configuration files as well. Note that removing syslog-ng PE does not restore the syslog daemon used before syslog-ng.

  • If you have installed syslog-ng PE from an .rpm package: Execute the rpm -e syslog-ng-premium-edition command to remove syslog-ng PE. Note that removing syslog-ng PE does not restore the syslog daemon used before syslog-ng PE.

  • If you have installed syslog-ng PE from a .pkg package: Execute the pkgrm BBsyslng command to remove syslog-ng PE. Note that removing syslog-ng PE does not restore the syslog daemon used before syslog-ng.

    For automatic uninstall (answering y to all questions): Execute the yes | pkgrm BBsyslng command.

    The following files have to be deleted manually:

    • <syslog-ng path>/etc/syslog-ng.conf

    • <syslog-ng path>/var/syslog-ng.persist

    • <syslog-ng path>/var/syslog-ng-00000.qf

    • anything else under the <syslog-ng path>/var directory

  • If you have installed syslog-ng PE using the .run installer: Execute the uninstall.sh script located at /opt/syslog-ng/bin/uninstall.sh. The uninstall script will automatically restore the syslog daemon used before installing syslog-ng. To completely remove syslog-ng PE, including the configuration files, use the uninstall.sh --purge command.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating