Chat now with support
Chat with Support

syslog-ng Premium Edition 7.0.34 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng PE The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files google-pubsub: collecting messages from the Google Pub/Sub messaging service wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mssql, oracle, sql: collecting messages from an SQL database network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6,udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2>: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files google_bigquery(): Sending logs to a Google BigQuery table google_bigquery_managedaccount(): Sending logs to a Google BigQuery table authenticated by Google Cloud managed service account google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service google_pubsub-managedaccount(): Sending logs to the Google Cloud Pub/Sub messaging service authenticated by Google Cloud managed service account hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka(): Publishing messages to Apache Kafka (Java implementation) (DEPRECATED) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel(): Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql(): Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transport Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

Upgrading the sql() source of syslog-ng PE

This section describes how you can upgrade your sql() source between syslog-ng Premium Edition (syslog-ng PE) 6 LTS and syslog-ng PE 7 LTS.

For more information about the mssql(), oracle(), or sql() sources, see mssql, oracle, sql: collecting messages from an SQL database.

To upgrade the sql() source between syslog-ng PE 6 LTS and syslog-ng PE 7 LTS

  1. Extract the value of last_read_uid from the persist file by using persist tool.

    For more information on using the persist tool, see the The syslog-ng manual pages.

  2. Add the extracted last_read_uid value to the syslog-ng PE configuration in a start-uid(value) format.
Example: adding the value of last_read_uid from the persist file into the configuration as start-uid(value)

You can add the value of last_read_uid that you extracted from the persist file into your configuration as start-uid(value)with the following method:

$ /opt/syslog-ng/bin/persist-tool dump /opt/syslog-ng/var/syslog-ng.persist
hostid = { "value": "65 D5 E1 06" }

freetds,mssql-host,1433,test_database,test_table = { "version": 0, "big_endian": false, "last_read_id": "2" }

run_id = { "value": "01 00 00 00" }

In the example above, the value is "2". The parameter in the configuration should look like start-uid("2").

Differences in configuration

This section describes the sql() source-related differences in configuration between syslog-ng PE 6 LTS and syslog-ng PE 7 LTS, including the features not ported from syslog-ng PE 6 LTS and the features that changed in usage since syslog-ng PE 6 LTS.

For more information about the mssql(), oracle(), or sql() sources, see mssql, oracle, sql: collecting messages from an SQL database.

Features not ported
  • Only MSSQL and Oracle are supported in syslog-ng PE 7 LTS. MySQL and PostGreSQL users cannot upgrade to syslog-ng PE 7 LTS.
  • archive-query(): It needs to be removed from configuration, because it is not supported in syslog-ng PE 7 LTS. Users need to find another way to cleanup old records.
  • read-old-records(): This option is not supported in 7 LTS, as its functionality has been replaced by the start-uid() option. If you follow To upgrade the sql() source between syslog-ng PE 6 LTS and syslog-ng PE 7 LTS, your configuration will have a start-uid() in the appropriate configuration level and no specific action will be necessary.
Changes in features

  • default-facility()

    The default value is user instead of local0.

    To achieve original behavior, you have to set default-facility(local0) explicitly in your configuration.

  • default-priority()

    The default value is notice instead of info.

    In syslog-ng PE 7 LTS, default-priority has been renamed to default-severity(). For compatibility reasons, the original default-priority() option can be still used. However, One Identity recommends replacing it with default-severity() instead.

    To achieve original behavior, you have to set default-severity(info) explicitly in your configuration.

  • follow-freq()

    The default value of follow-freq() has changed to 60 seconds (1 minute). The original default value was 10 seconds in syslog-ng PE 6 LTS.

    In syslog-ng PE 7 LTS, if time-reopen() is set, then syslog-ng PE will use the value you set. If neither time-reopen(), nor follow-freq() is set, the default value is 60 seconds.

    To achieve original behavior, you have to set follow-freq(10) explicitly in your configuration.

  • host-template()

    In syslog-ng PE 6 LTS, it was mandatory to set keep-hostname(yes) for host-template() to work. In syslog-ng PE 7 LTS, syslog-ng PE will automatically set keep-hostname(yes) even if you do not set it in your configuration. If you set host-template(), but also set keep-hostname(no), syslog-ng PE warns you about it, but otherwise ignores the keep-hostname() setting.

    In 6 LTS, syslog-ng PE allowed empty host-template() options in configuration. In syslog-ng PE 7 LTS, the template is mandatory in host-template(template). The syslog-ng PE application will not start with such configuration. As a result, you have to delete empty host-template() options from your configuration.

  • message-template()

    In 6 LTS, syslog-ng PE allowed empty message-template() options in configuration. In syslog-ng PE 7 LTS, the template is mandatory in message-template(template). The syslog-ng PE application will not start with such configuration. As a result, you have to delete empty message-template() options from your configuration.

  • program-template()

    In syslog-ng PE6 LTS, allowed empty program-template() options in configuration. In syslog-ng PE 7 LTS, the template is mandatory in program-template(template). The syslog-ng PE application will not start with such configuration. As a result, you have to delete empty program-template() options from your configuration.

  • prefix()

    You have to add a trailing dot to the prefix(). For example, if prefix(".sql") was provided in the configuration, the same option should look like prefix(".sql.") in syslog-ng PE 7 LTS.

    If the prefix does not end with a trailing dot, syslog-ng PE 7 LTS will add it automatically, and emit a warning about it.

    The default value has been changed to .sql. to reflect this change.

  • read-old-records()

    • read-old-records(yes) works the same way as in syslog-ng PE 6 LTS.

    • read-old-records(no): In syslog-ng PE 6 LTS, whenever syslog-ng PE starts, the application will only read those entries that are created after start. This means between restarts or reload, message loss is possible. In 7 LTS, syslog-ng PE will store the id of the message that was fetched last time in the persist file, even in case of read-old-records(no). When syslog-ng PE starts, it will continue from that id. This means that read-old-records() only takes effect when syslog-ng PE first starts. After restarts or reloads, read-old-records() is ignored.

Uninstalling syslog-ng PE

If you need to uninstall syslog-ng PE for some reason, you have the following options:

  • If you have installed syslog-ng PE from a .deb package: Execute the dpkg -r syslog-ng-premium-edition command to remove syslog-ng, or the dpkg -P syslog-ng-premium-edition command to remove syslog-ng PE and the configuration files as well. Note that removing syslog-ng PE does not restore the syslog daemon used before syslog-ng.

  • If you have installed syslog-ng PE from an .rpm package: Execute the rpm -e syslog-ng-premium-edition command to remove syslog-ng PE. Note that removing syslog-ng PE does not restore the syslog daemon used before syslog-ng PE.

  • If you have installed syslog-ng PE from a .pkg package: Execute the pkgrm BBsyslng command to remove syslog-ng PE. Note that removing syslog-ng PE does not restore the syslog daemon used before syslog-ng.

    For automatic uninstall (answering y to all questions): Execute the yes | pkgrm BBsyslng command.

    The following files have to be deleted manually:

    • <syslog-ng path>/etc/syslog-ng.conf

    • <syslog-ng path>/var/syslog-ng.persist

    • <syslog-ng path>/var/syslog-ng-00000.qf

    • anything else under the <syslog-ng path>/var directory

  • If you have installed syslog-ng PE using the .run installer: Execute the uninstall.sh script located at /opt/syslog-ng/bin/uninstall.sh. The uninstall script will automatically restore the syslog daemon used before installing syslog-ng. To completely remove syslog-ng PE, including the configuration files, use the uninstall.sh --purge command.

Configuring Microsoft SQL Server to accept logs from syslog-ng

The following describes how to configure your Microsoft SQL Server to enable remote logins and accept log messages from syslog-ng.

To configure your Microsoft SQL Server to enable remote logins and accept log messages from syslog-ng

  1. Start the SQL Server Management Studio application. Select Start > Programs > Microsoft SQL Server 2005 > SQL Server Management Studio.

  2. Create a new database.

    1. Figure 20: Creating a new MSSQL database 1.

      In the Object Explorer, right-click on the Databases entry and select New Database.

    2. Figure 21: Creating a new MSSQL database 2.

      Enter the name of the new database (for example, syslogng) into the Database name field and click OK.

  3. Create a new database user and associate it with the new database.

    1. Figure 22: Creating a new MSSQL user 1.

      In the Object Explorer, select Security, right-click on the Logins entry, then select New Login.

    2. Figure 23: Creating a new MSSQL user 2.

      Enter a name (for example, syslog-ng) for the user into the Login name field.

    3. Select the SQL Server Authentication option and enter a password for the user.

    4. In the Default database field, select the database created in Step 2 (for example, syslogng).

    5. In the Default language field, select the language of log messages that you want to store in the database, then click OK.

      		 Caution:

      Incorrect language settings may result in the database converting the messages to a different character-encoding format. That way the log messages may become unreadable, causing information loss.

    6. In the Object Explorer, select Security > Logins, then right-click on the new login created in the previous step, and select Properties.

    7. Figure 24: Associating database with the new user

      Select User Mapping. In the Users mapped to this login option, check the line corresponding to the new login (for example, syslogng). In the Database role membership field, check the db_owner and public options.

  4. Figure 25: Associating database with the new user

    Enable remote logins for SQL users.

    In the Object Explorer right-click on your database server, and select Properties > Security, and set the Server Authentication option to SQL Server and Windows Authentication mode.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating