syslog-ng Open Source Edition 3.33 - Administration Guide

Table of Contents

Summary of contents Target audience and prerequisites Summary of changes Acknowledgments

Introduction to syslog-ng

What syslog-ng is What syslog-ng is not Why is syslog-ng needed? What is new in syslog-ng Open Source Edition version 3.33? Who uses syslog-ng? Supported platforms

The concepts of syslog-ng

The philosophy of syslog-ng Logging with syslog-ng

The route of a log message in syslog-ng

Modes of operation

Client mode Relay mode Server mode

Global objects Timezones and daylight saving

How syslog-ng OSE assigns timezone to the message A note on timezones and timestamps

Product licensing High availability support The structure of a log message

BSD-syslog or legacy-syslog messages

The PRI message part The HEADER message part The MSG message part

IETF-syslog messages The Enterprise-wide message model

Message representation in syslog-ng OSE Structuring macros, metadata, and other value-pairs

Specifying data types in value-pairs value-pairs()

Things to consider when forwarding messages between syslog-ng OSE hosts Commercial version of syslog-ng

Installing syslog-ng

Compiling syslog-ng from source Compiling options of syslog-ng OSE Uninstalling syslog-ng OSE Configuring Microsoft SQL Server to accept logs from syslog-ng

The syslog-ng OSE quick-start guide

Configuring syslog-ng on client hosts Configuring syslog-ng on server hosts Configuring syslog-ng relays

Configuring syslog-ng on relay hosts How relaying log messages works

Managing and checking syslog-ng OSE service on Linux

The syslog-ng OSE configuration file

Location of the syslog-ng configuration file The configuration syntax in detail Notes about the configuration syntax Defining configuration objects inline Using channels in configuration objects Global and environmental variables Modules in syslog-ng Open Source Edition (syslog-ng OSE)

Loading modules Listing configuration options Visualize the configuration

Managing complex syslog-ng configurations

Including configuration files Reusing configuration blocks Generating configuration blocks from a script

Python code in external files Logging from your Python code

source: Read, receive, and collect log messages

How sources work default-network-drivers: Receive and parse common syslog messages

default-network-drivers() source options

internal: Collecting internal messages

internal() source options

file: Collecting messages from text files

Notes on reading kernel messages file() source options

wildcard-file: Collecting messages from multiple text files

wildcard-file() source options

linux-audit: Collecting messages from Linux audit logs

linux-audit() source options

network: Collecting messages using the RFC3164 protocol (network() driver)

network() source options

nodejs: Receiving JSON messages from nodejs applications

nodejs() source options

mbox: Converting local email messages to log messages

mbox() source options

osquery: Collect and parse osquery result logs

osquery() source options

pipe: Collecting messages from named pipes

pipe() source options

pacct: Collecting process accounting logs on Linux

pacct() options

program: Receiving messages from external applications

program() source options

python: writing server-style Python sources

Python LogMessage API python() and python-fetcher() source options

python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps

snmptrap() source options

sun-streams: Collecting messages on Sun Solaris

sun-streams() source options

syslog: Collecting messages using the IETF syslog protocol (syslog() driver)

syslog() source options

system: Collecting the system-specific log messages of a platform

system() source options

systemd-journal: Collecting messages from the systemd-journal system log storage

systemd-journal() source options

systemd-syslog: Collecting systemd messages using a socket

systemd

tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE

tcp(), tcp6(), udp() and udp6() source options: OBSOLETE

Change an old source driver to the network() driver

unix-stream, unix-dgram: Collecting messages from UNIX domain sockets

UNIX credentials and other metadata unix-stream() and unix-dgram() source options

stdin: Collecting messages from the standard input stream

stdin() source options

destination: Forward, send, and store log messages

amqp: Publishing messages using AMQP

amqp() destination options

collectd: sending metrics to collectd

collectd() destination options

elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED)

Prerequisites How syslog-ng OSE interacts with Elasticsearch Client modes Search Guard and syslog-ng OSE Elasticsearch2 destination options (DEPRECATED) Example use cases of sending logs to Elasticsearch using syslog-ng

elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API

Batch mode and load balancing elasticsearch-http() destination options

file: Storing messages in plain-text files

file() destination options

graphite: Sending metrics to Graphite

graphite() destination options

Sending logs to Graylog

graylog() destination options

hdfs: Storing messages on the Hadoop Distributed File System (HDFS)

Prerequisites How syslog-ng OSE interacts with HDFS Storing messages with MapR-FS Kerberos authentication with syslog-ng hdfs() destination HDFS destination options

Posting messages over HTTP

HTTP destination options

http: Posting messages over HTTP without Java

Batch mode and load balancing HTTP destination options The Azure auth header plugin The Python HTTP header plugin

kafka: Publishing messages to Apache Kafka (Java implementation)

Prerequisites How syslog-ng OSE interacts with Apache Kafka Kafka destination options

kafka(): Publishing messages to Apache Kafka (C implementation, using the librdkafka client)

Shifting from Java implementation to C implementation Before you begin Flow control in syslog-ng PE and the Kafka client Options of the kafka() destination's C implementation

loggly: Using Loggly

loggly() destination options

logmatic: Using Logmatic.io

logmatic() destination options

mongodb: Storing messages in a MongoDB database

How syslog-ng OSE connects the MongoDB server mongodb() destination options

mqtt() destination: sending messages from a local network to an MQTT broker

Prerequisites to using the mqtt() destination Limitations to using the mqtt() destination Options of the mqtt() destination Possible error messages you may encounter while using the mqtt() destination

network: Sending messages to a remote log server using the RFC3164 protocol (network() driver)

network() destination options

osquery: Sending log messages to osquery's syslog table

osquery() destination options

pipe: Sending messages to named pipes

pipe() destination options

program: Sending messages to external applications

program() destination options

pseudofile() destination options

python: writing custom Python destinations

python() destination options

redis: Storing name-value pairs in Redis

redis() destination options

riemann: Monitoring your data with Riemann

riemann() destination options

slack: Sending alerts and notifications to a Slack channel

Slack destination options

smtp: Generating SMTP messages (email) from logs

smtp() destination options

snmp: Sending SNMP traps

snmp() destination options

Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database

Using the sql() driver with an Oracle database Using the sql() driver with a Microsoft SQL database The way syslog-ng interacts with the database

MySQL-specific interaction methods MsSQL-specific interaction methods

sql() destination options

stomp: Publishing messages using STOMP

stomp() destination options

Sumo Logic destinations: sumologic-http() and sumologic-syslog()

sumologic-http() sumologic-syslog() sumologic-http() and sumologic-syslog() destination options

sumologic-http() destination options sumologic-syslog() destination options

syslog: Sending messages to a remote logserver using the IETF-syslog protocol

syslog() destination options

syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)

tcp(), tcp6(), udp(), and udp6() destination options

Change an old destination driver to the network() driver

Telegram: Sending messages to Telegram

telegram() destination options

unix-stream, unix-dgram: Sending messages to UNIX domain sockets

unix-stream() and unix-dgram() destination options

usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover

log: Filter and route log messages using log paths, flags, and filters

Embedded log statements

Using embedded log statements

if-else-elif: Conditional expressions Junctions and channels Log path flags

Managing incoming and outgoing messages with flow-control

Flow-control and multiple destinations Configuring flow-control

Using disk-based and memory buffering

Enabling reliable disk-based buffering Enabling normal disk-based buffering Enabling memory buffering About disk queue files

Using filters Combining filters with boolean operators Comparing macro values in filters Using wildcards, special characters, and regular expressions in filters Tagging messages Filter functions

Dropping messages

Global options of syslog-ng OSE

Configuring global syslog-ng options Global options

TLS-encrypted message transfer

Secure logging using TLS Encrypting log messages with TLS

Configuring TLS on the syslog-ng clients Configuring TLS on the syslog-ng server

Mutual authentication using TLS

Configuring TLS on the syslog-ng clients Configuring TLS on the syslog-ng server

Password-protected keys TLS options

template and rewrite: Format, modify, and manipulate log messages

Customize message format using macros and templates

Formatting messages, filenames, directories, and tablenames Templates and macros Date-related macros Hard versus soft macros Macros of syslog-ng OSE

Example use case: using the $DESTIP, the $DESTPORT, and the $PROTO macros

Using template functions Template functions of syslog-ng OSE Modifying the on-the-wire message format

Modifying messages using rewrite rules

Replacing message parts Setting message fields to specific values Setting severity with the set-severity() rewrite function Setting the facility field with the set-facility() rewrite function Setting the priority of a message with the set-pri() rewrite function Unsetting message fields Creating custom SDATA fields Setting multiple message fields to specific values map-value-pairs: Rename value-pairs to normalize logs Conditional rewrites Adding and deleting tags Rewrite the timezone of a message Anonymizing credit card numbers

Regular expressions

Options of regular expressions

The type() options of regular expressions The flags() options of regular expressions

Perl Compatible Regular Expressions (PCRE) Literal string searches Glob patterns without regular expression support

Optimizing regular expressions

parser: Parse and segment structured messages

Parsing syslog messages

Options of syslog-parser() parsers

Parsing messages with comma-separated and similar values

Options of CSV parsers

Parsing key=value pairs

Options of key=value parsers

Options of JSON parsers

Limitations of the XML parsers Options of the XML parsers

Parsing dates and timestamps

Options of date-parser() parsers

Python parser Parsing tags Apache access log parser

Options of apache-accesslog-parser() parsers

Linux audit parser

Options of linux-audit-parser() parsers

Cisco parser Parsing enterprise-wide message model (EWMM) messages iptables parser Netskope parser panos-parser(): parsing PAN-OS log messages

Message format parsed by panos-parser() PAN-OS parser options

Sudo parser Websense parser Check Point Log Exporter parser

db-parser: Process message content with a pattern database (patterndb)

Classifying log messages

The structure of the pattern database How pattern matching works Artificial ignorance

Using pattern databases

Using parser results in filters and templates Downloading sample pattern databases Correlating log messages using pattern databases

Referencing earlier messages of the context

Triggering actions for identified messages

Conditional actions External actions Actions and message correlation

Creating pattern databases

Using pattern parsers

Pattern parsers of syslog-ng OSE

What's new in the syslog-ng pattern database format V5 The syslog-ng pattern database format

Element: patterndb Element: ruleset Element: patterns Element: rules Element: rule Element: patterns Element: urls Element: values Element: examples Element: example Element: actions Element: action Element: create-context Element: tags

Correlating log messages

Correlating messages using the grouping-by() parser

Referencing earlier messages of the context Options of grouping-by parsers

Enriching log messages with external data

Adding metadata from an external file

Using filters as selector Shell-style globbing in the selector Options add-contextual-data()

Looking up GeoIP data from IP addresses (DEPRECATED)

Options of geoip parsers

Looking up GeoIP2 data from IP addresses

Referring to parts of the message as a macro Using the GeoIP2 parser Transferring your logs to Elasticsearch using GeoIP2 Options of geoip2 parsers

Statistics of syslog-ng

Metrics and counters of syslog-ng OSE Log statistics from the internal() source

Multithreading and scaling in syslog-ng OSE

Multithreading concepts of syslog-ng OSE Configuring multithreading Optimizing multithreaded performance

Troubleshooting syslog-ng

Possible causes of losing log messages Creating syslog-ng core files Collecting debugging information with strace, truss, or tusc Running a failure script Stopping syslog-ng Reporting bugs and finding help Recover data from orphaned diskbuffer files No local logs after specifying an unusual storage directory No logs after specifying an unusual port number Error messages SELinux prevents syslog-ng OSE from using the execmem access on a process

Best practices and examples

General recommendations Handling large message load Using name resolution in syslog-ng

Resolving hostnames locally

Collecting logs from chroot Configuring log rotation Load balancing logs between multiple destinations

Load balancing with a round robin load balancing method based on the R_MSEC macro of syslog-ng OSE Configuration generator for the load balancing method based on MSEC hashing

The syslog-ng manual pages

The syslog-ng manual pages

dqtool.1 loggen.1 pdbtool.1 persist-tool.1 syslog-ng-ctl.1 syslog-ng-debun.1 syslog-ng.8 syslog-ng.conf.5

Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License Glossary

ose-3.29-panos

New parser: panos-parser()

From version 3.29, the syslog-ng OSE application supports the panos-parser() parser as SCL.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating

© ALL RIGHTS RESERVED. Terms of Use Privacy Cookie Preference Center