Chatta subito con l'assistenza

Ottieni assistenza in tempo reale
Completa la registrazione

Accedi

Richiedi prezzo

Contatta il supporto vendite

syslog-ng Open Source Edition 3.19 - Administration Guide

Table of Contents

Summary of contents Target audience and prerequisites Products covered in this guide Summary of changes Acknowledgments

Introduction to syslog-ng

What syslog-ng is What syslog-ng is not Why is syslog-ng needed? What is new in syslog-ng Open Source Edition 3.19? Who uses syslog-ng? Supported platforms

The concepts of syslog-ng

The philosophy of syslog-ng Logging with syslog-ng

The route of a log message in syslog-ng

Modes of operation

Client mode Relay mode Server mode

Global objects Timezones and daylight saving

How syslog-ng OSE assigns timezone to the message A note on timezones and timestamps

Product licensing High availability support The structure of a log message

BSD-syslog or legacy-syslog messages IETF-syslog messages The Enterprise-wide message model

Message representation in syslog-ng OSE Structuring macros, metadata, and other value-pairs

Specifying data types in value-pairs value-pairs()

Things to consider when forwarding messages between syslog-ng OSE hosts Commercial version of syslog-ng

Installing syslog-ng

Compiling syslog-ng from source Compiling options of syslog-ng OSE Uninstalling syslog-ng OSE Configuring Microsoft SQL Server to accept logs from syslog-ng

The syslog-ng OSE quick-start guide

Configuring syslog-ng on client hosts Configuring syslog-ng on server hosts Configuring syslog-ng relays

Configuring syslog-ng on relay hosts How relaying log messages works

The syslog-ng OSE configuration file

Location of the syslog-ng configuration file The configuration syntax in detail Notes about the configuration syntax Defining configuration objects inline Using channels in configuration objects Global and environmental variables Modules in syslog-ng OSE

Loading modules

Managing complex syslog-ng configurations

Including configuration files Reusing configuration blocks Generating configuration blocks from a script

Python code in external files

source: Read, receive, and collect log messages

How sources work default-network-drivers: Receive and parse common syslog messages

default-network-drivers() source options

internal: Collecting internal messages

internal() source options

file: Collecting messages from text files

Notes on reading kernel messages file() source options

wildcard-file: Collecting messages from multiple text files

wildcard-file() source options

linux-audit: Collecting messages from Linux audit logs

linux-audit() source options

network: Collecting messages using the RFC3164 protocol (network() driver)

network() source options

nodejs: Receiving JSON messages from nodejs applications

nodejs() source options

mbox: Converting local e-mail messages to log messages

mbox() source options

osquery: Collect and parse osquery result logs

osquery() source options

pipe: Collecting messages from named pipes

pipe() source options

pacct: Collecting process accounting logs on Linux

pacct() options

program: Receiving messages from external applications

program() source options

python: writing server-style Python sources

Python LogMessage API python() and python-fetcher() source options

python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps

snmptrap() source options

sun-streams: Collecting messages on Sun Solaris

sun-streams() source options

syslog: Collecting messages using the IETF syslog protocol (syslog() driver)

syslog() source options

system: Collecting the system-specific log messages of a platform

system() source options

systemd-journal: Collecting messages from the systemd-journal system log storage

systemd-journal() source options

systemd-syslog: Collecting systemd messages using a socket

systemd

tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE

tcp(), tcp6(), udp() and udp6() source options: OBSOLETE

Change an old source driver to the network() driver

unix-stream, unix-dgram: Collecting messages from UNIX domain sockets

UNIX credentials and other metadata unix-stream() and unix-dgram() source options

stdin: Collecting messages from the standard input stream

stdin() source options

destination: Forward, send, and store log messages

amqp: Publishing messages using AMQP

amqp() destination options

elasticsearch: Sending messages directly to Elasticsearch version 1.x (DEPRECATED)

Prerequisites How syslog-ng OSE interacts with Elasticsearch Client modes Elasticsearch destination options

elasticsearch2: Sending logs directly to Elasticsearch and Kibana 2.0 or higher

Prerequisites How syslog-ng OSE interacts with Elasticsearch Client modes Search Guard and syslog-ng OSE Elasticsearch2 destination options Example use cases of sending logs to Elasticsearch using syslog-ng

file: Storing messages in plain-text files

file() destination options

graphite: Sending metrics to Graphite

graphite() destination options

Sending logs to Graylog

graylog() destination options

hdfs: Storing messages on the Hadoop Distributed File System (HDFS)

Prerequisites How syslog-ng OSE interacts with HDFS Storing messages with MapR-FS Kerberos authentication with syslog-ng hdfs() destination HDFS destination options

Posting messages over HTTP

HTTP destination options

http: Posting messages over HTTP without Java

Batch mode and load balancing HTTP destination options

kafka: Publishing messages to Apache Kafka

Prerequisites How syslog-ng OSE interacts with Apache Kafka Kafka destination options

loggly: Using Loggly

loggly() destination options

logmatic: Using Logmatic.io

logmatic() destination options

mongodb: Storing messages in a MongoDB database

How syslog-ng OSE connects the MongoDB server mongodb() destination options

network: Sending messages to a remote log server using the RFC3164 protocol (network() driver)

network() destination options

osquery: Sending log messages to osquery's syslog table

osquery() destination options

pipe: Sending messages to named pipes

pipe() destination options

program: Sending messages to external applications

program() destination options

pseudofile() destination options

python: writing custom Python destinations

python() destination options

redis: Storing name-value pairs in Redis

redis() destination options

riemann: Monitoring your data with Riemann

riemann() destination options

slack: Sending alerts and notifications to a Slack channel

Slack destination options

smtp: Generating SMTP messages (e-mail) from logs

smtp() destination options

Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database

Using the sql() driver with an Oracle database Using the sql() driver with a Microsoft SQL database The way syslog-ng interacts with the database

MySQL-specific interaction methods MsSQL-specific interaction methods

sql() destination options

stomp: Publishing messages using STOMP

stomp() destination options

syslog: Sending messages to a remote logserver using the IETF-syslog protocol

syslog() destination options

syslog-ng: Forwarding messages and tags to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)

tcp(), tcp6(), udp(), and udp6() destination options

Change an old destination driver to the network() driver

Telegram: Sending messages to Telegram

telegram() destination options

unix-stream, unix-dgram: Sending messages to UNIX domain sockets

unix-stream() and unix-dgram() destination options

usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover

log: Filter and route log messages using log paths, flags, and filters

Embedded log statements

Using embedded log statements

if-else-elif: Conditional expressions Junctions and channels Log path flags

Managing incoming and outgoing messages with flow-control

Flow-control and multiple destinations Configuring flow-control

Using disk-based and memory buffering

Enabling reliable disk-based buffering Enabling normal disk-based buffering Enabling memory buffering About disk queue files

Using filters Combining filters with boolean operators Comparing macro values in filters Using wildcards, special characters, and regular expressions in filters Tagging messages Filter functions

Dropping messages

Global options of syslog-ng OSE

Configuring global syslog-ng options Global options

TLS-encrypted message transfer

Secure logging using TLS Encrypting log messages with TLS

Configuring TLS on the syslog-ng clients Configuring TLS on the syslog-ng server

Mutual authentication using TLS

Configuring TLS on the syslog-ng clients Configuring TLS on the syslog-ng server

Password-protected keys TLS options

template and rewrite: Format, modify, and manipulate log messages

Customize message format using macros and templates

Formatting messages, filenames, directories, and tablenames Templates and macros Date-related macros Hard vs. soft macros Macros of syslog-ng OSE Using template functions Template functions of syslog-ng OSE Modifying the on-the-wire message format

Modifying messages using rewrite rules

Replacing message parts Setting message fields to specific values Unsetting message fields Creating custom SDATA fields Setting multiple message fields to specific values map-value-pairs: Rename value-pairs to normalize logs Conditional rewrites Adding and deleting tags Anonymizing credit card numbers

Regular expressions

Types and options of regular expressions Optimizing regular expressions

parser: Parse and segment structured messages

Parsing syslog messages

Options of syslog-parser parsers

Parsing messages with comma-separated and similar values

Options of CSV parsers

Parsing key=value pairs

Options of key=value parsers

The JSON parser

Options of JSON parsers

Options of XML parsers

Parsing dates and timestamps

Options of date-parser() parsers

The Apache Access Log Parser

Options of apache-accesslog-parser() parsers

The Cisco Parser The Linux Audit Parser

Options of linux-audit-parser() parsers

The Python Parser Parsing EWMM messages The sudo parser The iptables parser

db-parser: Process message content with a pattern database (patterndb)

Classifying log messages

The structure of the pattern database How pattern matching works Artificial ignorance

Using pattern databases

Using parser results in filters and templates Downloading sample pattern databases Correlating log messages using pattern databases

Referencing earlier messages of the context

Triggering actions for identified messages

Conditional actions External actions Actions and message correlation

Creating pattern databases

Using pattern parsers

Pattern parsers of syslog-ng OSE

What's new in the syslog-ng pattern database format V5 The syslog-ng pattern database format

Element: patterndb Element: ruleset Element: patterns Element: rules Element: rule Element: patterns Element: urls Element: values Element: examples Element: example Element: actions Element: action Element: create-context Element: tags

Correlating log messages

Correlating messages using the grouping-by() parser

Referencing earlier messages of the context Options of grouping-by parsers

Enriching log messages with external data

Adding metadata from an external file

Using filters as selector Options add-contextual-data()

Looking up GeoIP data from IP addresses (DEPRECATED)

Options of geoip parsers

Looking up GeoIP2 data from IP addresses

Referring to parts of the message as a macro Using the GeoIP2 parser Transferring your logs to Elasticsearch using GeoIP2 Options of geoip2 parsers

Statistics of syslog-ng

Metrics and counters of syslog-ng OSE Log statistics from the internal() source

Multithreading and scaling in syslog-ng OSE

Multithreading concepts of syslog-ng OSE Configuring multithreading Optimizing multithreaded performance

Troubleshooting syslog-ng

Possible causes of losing log messages Creating syslog-ng core files Collecting debugging information with strace, truss, or tusc Running a failure script Stopping syslog-ng Reporting bugs and finding help Recover data from orphaned diskbuffer files No local logs after specifying an unusual storage directory No logs after specifying an unusual port number Error messages

Best practices and examples

General recommendations Handling large message load Using name resolution in syslog-ng

Resolving hostnames locally

Collecting logs from chroot Configuring log rotation

The syslog-ng manual pages

dqtool.1 loggen.1 pdbtool.1 syslog-ng-ctl.1 syslog-ng-debun.1 syslog-ng.8 syslog-ng.conf.5

Third-party contributions

GNU General Public License

Preamble TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

Section 0 Section 1 Section 2 Section 3 Section 4 Section 5 Section 6 Section 7 Section 8 Section 9 Section 10 NO WARRANTY Section 11 Section 12

How to Apply These Terms to Your New Programs

GNU Lesser General Public License License attributions

Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License About us

destination-load-balancing-url

Starting with version 3.19, you can specify multiple URLs (separated with a whitespace). In this case, syslog-ng OSE sends log messages to the specified URLs in a load-balance fashion. This means that syslog-ng OSE sends each message to only one URL. For example, you can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. Note that the order of the messages as they arrive on the servers can differ from the order syslog-ng OSE has received them, so use load-balancing only if your server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the order of the messages will be incorrect.

Caution:

If you set multiple URLs in the url() option, set the persist-name() option as well to avoid data loss.

Related Documents

The document was helpful.

Seleziona valutazione

I easily found the information I needed.

Seleziona valutazione

© ALL RIGHTS RESERVED. Termini di utilizzo Privacy Cookie Preference Center