立即与支持人员聊天
与支持团队交流

syslog-ng Premium Edition 7.0.34 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng PE The syslog-ng PE quick-start guide The syslog-ng PE configuration file Collecting log messages — sources and source drivers
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files google-pubsub: collecting messages from the Google Pub/Sub messaging service wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs mssql, oracle, sql: collecting messages from an SQL database network: Collecting messages using the RFC3164 protocol (network() driver) office365: Fetching logs from Office 365 osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6,udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol udp-balancer: Receiving UDP messages at very high rate unix-stream, unix-dgram: Collecting messages from UNIX domain sockets windowsevent: Collecting Windows event logs
Sending and storing log messages — destinations and destination drivers
elasticsearch2>: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector file: Storing messages in plain-text files google_bigquery(): Sending logs to a Google BigQuery table google_bigquery_managedaccount(): Sending logs to a Google BigQuery table authenticated by Google Cloud managed service account google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service google_pubsub-managedaccount(): Sending logs to the Google Cloud Pub/Sub messaging service authenticated by Google Cloud managed service account hdfs: Storing messages on the Hadoop Distributed File System (HDFS) http: Posting messages over HTTP kafka(): Publishing messages to Apache Kafka (Java implementation) (DEPRECATED) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) logstore: Storing messages in encrypted files mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) pipe: Sending messages to named pipes program: Sending messages to external applications python: writing custom Python destinations sentinel(): Sending logs to the Microsoft Azure Sentinel cloud snmp: Sending SNMP traps smtp: Generating SMTP messages (email) from logs splunk-hec: Sending messages to Splunk HTTP Event Collector sql(): Storing messages in an SQL database stackdriver: Sending logs to the Google Stackdriver cloud syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal — usertty() destination Client-side failover
Routing messages: log paths, flags, and filters Global options of syslog-ng PE TLS-encrypted message transfer Advanced Log Transport Protocol Reliability and minimizing the loss of log messages Manipulating messages parser: Parse and segment structured messages Processing message content with a pattern database Correlating log messages Enriching log messages with external data Monitoring statistics and metrics of syslog-ng Multithreading and scaling in syslog-ng PE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Glossary

ALTP options

The following options are specific to the ALTP protocol. Note that when using ALTP in a source or a destination, the options of the syslog() or the network() driver can be used as well.

 Caution:

Note that in earlier versions of syslog-ng Premium Edition, Advanced Log Transport Protocol is called Reliable Log Transfer Protocol (RLTP).

allow-plain-compress()
Accepted values: yes | no
Default: no

Description: Enable on-the-wire compression in the ALTP communication. Note that this option must be enabled both on the server and the client side to have any effect. Enabling compression can significantly reduce the bandwidth required to transport the messages, but can slightly decrease the performance of syslog-ng PE, reducing the number of transferred messages. The allow-plain-compress() option can be used in source and destination drivers as well. Available in syslog-ng PE 7.0.12 and later. Note that in earlier versions, the name of this option was allow-compress().

batch-size()
Accepted values: number
Default: 1000

Description: Specifies the number of lines that are sent to a destination at a time. The syslog-ng PE application waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network, but also increases message latency. Available in syslog-ng PE 7.0.10 and later.

message-acknowledgment-timeout()
Type: number (seconds)
Default: 900

Description: When the receiver (syslog-ng PE server) receives and successfully processes a message, it sends an acknowledgment to the sender (the syslog-ng PE client). If the receiver does not acknowledge receiving the messages within this period, the sender terminates the connection with the receiver. Use this option only in destination drivers.

response-timeout()
Type: number (seconds)
Default: 60

Description: If syslog-ng PE does not receive any protocol-related message in the given timeframe (except for message acknowledgment, which is governed by the message-acknowledgment-timeout() option), syslog-ng PE terminates the connection with the peer, and the "Connection broken" message appears in the logs of the sender (the syslog-ng PE client). This is normal, and happens when the sender does not send any new message to the receiver.

Under normal circumstances, you should not change the value of this option. The response-timeout() option can be used in source and destination drivers as well.

tls-required()
Type: yes, optional, no
Default: optional

Description: Determines whether STARTTLS is to be used during communication. If the option is set to yes, you must also configure the tls() option to specify other parameters of the TLS connection (for example, the authentication of the server and the client).

The tls-required() option can be used in source and destination drivers as well.

For example, if you configure tls-required(yes) on the server side and tls-required(no) on the client side, the connection is dropped. If one of them is set to optional, the configuration of the other side will decide if TLS is used or not. If both sides are set to optional, and the tls() option is properly configured, TLS encryption will be used. The following table summarizes the possible options and their results.

Note that the various parameters of the tls() option are considered in the connection only if the tls-required() settings of the peers result in TLS-encryption in the following table. In other words: the tls-required() option of ALTP determines if TLS should be used at all, while the peer-verify() option of the tls()setting determines if the TLS connection can be actually established.

tls-required() setting on the server
yes no optional
tls-required() setting on the client yes TLS-encryption rejected connection TLS-encryption
no rejected connection unencrypted connection unencrypted connection
optional TLS-encryption unencrypted connection TLS-encryption if the tls() option is set, unencrypted connection otherwise

Setting tls-required(optional) on your server allows you to receive both encrypted and unencrypted connections on the same port.

Examples for using ALTP

 Caution:

Note that in earlier versions of syslog-ng Premium Edition, Advanced Log Transport Protocol is called Reliable Log Transfer Protocol (RLTP).

Example: Simple ALTP connection

The sender and the receiver use ALTP over the network() protocol. Since the tls() option is not configured neither on the sender nor on the receiver, the communication will be unencrypted.

Receiver configuration (syslog-ng PE server):

source s_network_altp {
        network(
            ip("127.0.0.1")
            port("5555")
            transport(altp)
            ip-protocol(4)
        );
};

Sender configuration (syslog-ng PE client):

destination d_network_altp {
        network(
            "127.0.0.1"
            port("5555")
            transport(altp)
            ip-protocol(4)
        );
};
Example: ALTP with TLS encryption

The following example configures a sender and a receiver to communicate using ALTP. Since the tls-required() option is set to optional on the receiver and yes on the sender, and the tls() option is configured, the communication will be TLS-encrypted. For the sender (syslog-ng PE client), reliable disk-buffering is enabled to prevent data loss.

Receiver configuration (syslog-ng PE server):

source s_syslog_altp {
        syslog(
            ip("127.0.0.1")
            port("4444")
            transport(altp(tls-required(optional)))
            ip-protocol(4)
            tls(
                peer-verify(required-trusted)
                ca-dir("/var/tmp/client/")
                key-file("/var/tmp/server/server_priv.key")
                cert-file("/var/tmp/server/server.crt")
            )
        );
};

Sender configuration (syslog-ng PE client):

destination d_syslog_altp {
        syslog(
            "127.0.0.1"
            port("4444")
            transport(altp(tls-required(yes)))
            ip-protocol(4)
            disk-buffer( mem-buf-size(200000) disk-buf-size(2000000) reliable(yes) )
            tls(
                peer-verify(required-trusted)
                ca-dir("/var/tmp/server/")
                key-file("/var/tmp/client/client_priv.key")
                cert-file("/var/tmp/client/client.crt")
            )
        );
};

Reliability and minimizing the loss of log messages

Introduction

Advanced Log Transport Protocol (ALTP) interacts with flow control and the disk-buffer option to ensure that the loss of log messages is minimized or is prevented completely. This section explains how each loss prevention method contributes to reliability and minimizing log message loss. Flow control, the disk-buffer option, and ALTP are explained in detail elsewhere in the document. In this section, we present a high-level overview of all of these mechanisms and highlight considerations such as:

  • What best practices exist in various scenarios, how to set key parameters

  • When is a log message considered "delivered"

  • Under what circumstances can log loss occur

Each of the following sections discusses a different scenario and uses figures to aid comprehension.

NOTE: Each figure depicts a scenario in which the volume of incoming messages makes it necessary to use all buffers and control windows at maximum capacity.

Important information

Any of the mechanisms that syslog-ng PE uses to prevent or minimize the loss of log messages only works if the hardware and operating system work normally. When there is an issue with the hardware or operating system that the application and syslog-ng PE run on, log loss may occur. Issues include operating system crash (for example, kernel panic), memory errors, disk errors, power outage, and so on.

相关文档

The document was helpful.

选择评级

I easily found the information I needed.

选择评级