syslog-ng Premium Edition 7.0.19 - Administration Guide

The monitoring() source

The monitoring() source allows you to select which statistics of syslog-ng PE you want to monitor. In addition, the statistics are available as structured name-value pairs, so you can format the output similarly to other log messages. That way, you can easily convert the statistics and metrics, for example, into JSON or WELF format. That way, you can send the statistics of your log messages into a monitoring solution.

The monitoring() source queries the statistics (counters) that syslog-ng PE collects, formats them, and optionally resets the counters. The monitoring() source emits only these messages, making it easy to route them to their appropriate destination. The stats-level() global option determines exactly which statistics syslog-ng PE collects.

Declaration

source s_monitor{
    monitoring(
        query("*")
    );};

@version: 7.0
options{
    stats-level(3);
    keep-hostname(no);
};
source s_monitor{ monitoring(
    query("*")
    freq(10)
    message-template('$(format-json --scope nv_pairs)')
    );};
destination d_file {
    file("/var/log/syslog-ng-statistics.log");
};
log {
    source(s_monitor);
    destination(d_file);
};

[2017-04-03T14:00:31.786133] Outgoing message; message='Apr  3 14:00:31 example-hostname syslog-ng[12281]: {"src":{"severity":{"7":{"processed":"0"},"6":{"processed":"0"},"5":{"processed":"0"},"4":{"processed":"0"},"3":{"processed":"0"},"2":{"processed":"0"},"1":{"processed":"0"},"0":{"processed":"0"}},"monitoring":{"s_monitor#0":{"stamp":"0","processed":"0"}},"facility":{"other":{"processed":"0"},"9":{"processed":"0"},"8":{"processed":"0"},"7":{"processed":"0"},"6":{"processed":"0"},"5":{"processed":"0"},"4":{"processed":"0"},"3":{"processed":"0"},"23":{"processed":"0"},"22":{"processed":"0"},"21":{"processed":"0"},"20":{"processed":"0"},"2":{"processed":"0"},"19":{"processed":"0"},"18":{"processed":"0"},"17":{"processed":"0"},"16":{"processed":"0"},"15":{"processed":"0"},"14":{"processed":"0"},"13":{"processed":"0"},"12":{"processed":"0"},"11":{"processed":"0"},"10":{"processed":"0"},"1":{"processed":"0"},"0":{"processed":"0"}}},"source":{"s_monitor":{"processed":"0"}},"global":{"sdata_updates":{"processed":"0"},"payload_reallocs":{"processed":"2"},"msg_clones":{"processed":"0"}},"destination":{"d_file":{"processed":"0"}},"center":{"received":{"processed":"0"},"queued":{"processed":"0"}},"PROGRAM":"syslog-ng","PID":"12281"}\x0a'

{
  "center" : {
      "queued" : { "processed" : "0" },
      "received" : { "processed" : "0" }
    },
  "destination" : { "d_file" : { "processed" : "0" } },
  "global" : {
      "msg_clones" : { "processed" : "0" },
      "payload_reallocs" : { "processed" : "2" },
      "sdata_updates" : { "processed" : "0" }
    },
  "PID" : "12281",
  "PROGRAM" : "syslog-ng",
  "source" : { "s_monitor" : { "processed" : "0" } },
  "src" : {
      "facility" : {
          "0" : { "processed" : "0" },
          "1" : { "processed" : "0" },
          "2" : { "processed" : "0" },
          "3" : { "processed" : "0" },
          "4" : { "processed" : "0" },
          "5" : { "processed" : "0" },
          "6" : { "processed" : "0" },
          "7" : { "processed" : "0" },
          "8" : { "processed" : "0" },
          "9" : { "processed" : "0" },
          "10" : { "processed" : "0" },
          "11" : { "processed" : "0" },
          "12" : { "processed" : "0" },
          "13" : { "processed" : "0" },
          "14" : { "processed" : "0" },
          "15" : { "processed" : "0" },
          "16" : { "processed" : "0" },
          "17" : { "processed" : "0" },
          "18" : { "processed" : "0" },
          "19" : { "processed" : "0" },
          "20" : { "processed" : "0" },
          "21" : { "processed" : "0" },
          "22" : { "processed" : "0" },
          "23" : { "processed" : "0" },
          "other" : { "processed" : "0" }
        },
      "monitoring" : { "s_monitor#0" : {
              "processed" : "0",
              "stamp" : "0"
            } },
      "severity" : {
          "0" : { "processed" : "0" },
          "1" : { "processed" : "0" },
          "2" : { "processed" : "0" },
          "3" : { "processed" : "0" },
          "4" : { "processed" : "0" },
          "5" : { "processed" : "0" },
          "6" : { "processed" : "0" },
          "7" : { "processed" : "0" }
        }
    }
}

monitoring() source options

The monitoring() driver has the following options. Only the query() option is required, other options are optional.

clear-on-read()

Description: Reset the counters after reading. Note that if a destination is not available, syslog-ng PE will not reset its counter even if clear-on-read() is set to yes.

If you use multiple monitoring source, and you use the clear-on-read() parameter, make sure to adjust the queries appropriately. Overlapping queries that read and reset the same counters result in incorrect statistics.

freq()

Description: Specifies how often does syslog-ng PE execute the query and send a statistics message.

message-template()

Description: Specifies how the message containing the queried statistics is formatted. You can use macros and template functions in the format string. For example, you can format the message as a JSON object:

source s_monitor{ monitoring(
    query("*")
    freq(10)
    message-template('$(format-json --scope nv_pairs)')
    );};

Note that here you can only format the payload of the message (that is the, ${MESSAGE} part). You can format the headers or other parts of the outgoing message in the destination driver.

query()

Description: Specifies which statistical counters will be included in the messages. Note that the list of available counters depends on your syslog-ng PE configuration (mainly the configured sources and destinations) and on the stats-level() global option. The * string includes every available counters. The syntax of the query option is identical to the syslog-ng-ctl query get <query> command.

source s_monitor{ monitoring(
    query("*")
    );};

For example, the "destination*" query lists the configured destinations, and the metrics related to each destination. An example output:

destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.dropped: 0
destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.processed: 0
destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.stored: 0
destination.d_elastic.stats.processed: 0

The monitoring-welf() source

This source is actually preconfigured monitoring() source that generates statistics messages in WELF format. Starting with syslog-ng PE version 7.0.2, syslog-ng PE uses this driver for new installations to generate statistics. By default, a message is sent every 10 minutes (600 seconds).

@version: 7.0
@include 'scl.conf'
options {
    stats_level(3);
};
source s_monitoring_welf {
    monitoring-welf(freq(10) query('*'));
};
destination d_file {
    file("/tmp/output.txt");
};
log {
    source(s_monitoring_welf);
    destination(d_file);
};

The output is similar to the following:

Apr 3 14:03:26 example-host syslog-ng[12363]: PID=12363 PROGRAM=syslog-ng center.queued.processed=0 center.received.processed=0 destination.d_file.processed=0 global.msg_clones.processed=0 global.payload_reallocs.processed=2 global.sdata_updates.processed=0 source.s_monitoring_welf.processed=0 src.facility.0.processed=0 src.facility.1.processed=0 src.facility.10.processed=0 src.facility.11.processed=0 src.facility.12.processed=0 src.facility.13.processed=0 src.facility.14.processed=0 src.facility.15.processed=0 src.facility.16.processed=0 src.facility.17.processed=0 src.facility.18.processed=0 src.facility.19.processed=0 src.facility.2.processed=0 src.facility.20.processed=0 src.facility.21.processed=0 src.facility.22.processed=0 src.facility.23.processed=0 src.facility.3.processed=0 src.facility.4.processed=0 src.facility.5.processed=0 src.facility.6.processed=0 src.facility.7.processed=0 src.facility.8.processed=0 src.facility.9.processed=0 src.facility.other.processed=0 src.monitoring.s_monitoring_welf#0.processed=0 src.monitoring.s_monitoring_welf#0.stamp=0 src.severity.0.processed=0 src.severity.1.processed=0 src.severity.2.processed=0 src.severity.3.processed=0 src.severity.4.processed=0 src.severity.5.processed=0 src.severity.6.processed=0 src.severity.7.processed=0\x0a'

Multithreading and scaling in syslog-ng PE

Starting with version 4 F1, syslog-ng PE can process sources and destinations in multithreaded mode to scale to multiple CPUs or cores for increased performance. Starting with version 5 F4, this multithreaded mode is the default.