Preface
Summary of contents
Target audience and prerequisites
Products covered in this guide
Summary of changes
Acknowledgments
Introduction to syslog-ng
What syslog-ng is
What syslog-ng is not
Why is syslog-ng needed?
What is new in syslog-ng Open Source Edition 3.25?
Who uses syslog-ng?
Supported platforms
The concepts of syslog-ng
The philosophy of syslog-ng
Logging with syslog-ng
Modes of operation
Global objects
Timezones and daylight saving
Product licensing
High availability support
The structure of a log message
Message representation in syslog-ng OSE
Structuring macros, metadata, and other value-pairs
Things to consider when forwarding messages between syslog-ng OSE hosts
Commercial version of syslog-ng
Installing syslog-ng
Compiling syslog-ng from source
Compiling options of syslog-ng OSE
Uninstalling syslog-ng OSE
Configuring Microsoft SQL Server to accept logs from syslog-ng
The syslog-ng OSE quick-start guide
Configuring syslog-ng on client hosts
Configuring syslog-ng on server hosts
Configuring syslog-ng relays
Managing and checking syslog-ng OSE service on Linux
The syslog-ng OSE configuration file
Location of the syslog-ng configuration file
The configuration syntax in detail
Notes about the configuration syntax
Defining configuration objects inline
Using channels in configuration objects
Global and environmental variables
Modules in syslog-ng OSE
Managing complex syslog-ng configurations
source: Read, receive, and collect log messages
Including configuration files
Reusing configuration blocks
Generating configuration blocks from a script
Python code in external files
Logging from your Python code
How sources work
default-network-drivers: Receive and parse common syslog messages
internal: Collecting internal messages
file: Collecting messages from text files
wildcard-file: Collecting messages from multiple text files
linux-audit: Collecting messages from Linux audit logs
network: Collecting messages using the RFC3164 protocol (network() driver)
nodejs: Receiving JSON messages from nodejs applications
mbox: Converting local email messages to log messages
osquery: Collect and parse osquery result logs
pipe: Collecting messages from named pipes
pacct: Collecting process accounting logs on Linux
program: Receiving messages from external applications
python: writing server-style Python sources
python-fetcher: writing fetcher-style Python sources
snmptrap: Read Net-SNMP traps
sun-streams: Collecting messages on Sun Solaris
syslog: Collecting messages using the IETF syslog protocol (syslog() driver)
system: Collecting the system-specific log messages of a platform
systemd-journal: Collecting messages from the systemd-journal system log storage
systemd-syslog: Collecting systemd messages using a socket
tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE
unix-stream, unix-dgram: Collecting messages from UNIX domain sockets
stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP
collectd: sending metrics to collectd
elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED)
log: Filter and route log messages using log paths, flags, and filters
Prerequisites
How syslog-ng OSE interacts with Elasticsearch
Client modes
Search Guard and syslog-ng OSE
Elasticsearch2 destination options (DEPRECATED)
Example use cases of sending logs to Elasticsearch using syslog-ng
elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API
file: Storing messages in plain-text files
graphite: Sending metrics to Graphite
Sending logs to Graylog
hdfs: Storing messages on the Hadoop Distributed File System (HDFS)
Prerequisites
How syslog-ng OSE interacts with HDFS
Storing messages with MapR-FS
Kerberos authentication with syslog-ng hdfs() destination
HDFS destination options
Posting messages over HTTP
http: Posting messages over HTTP without Java
kafka: Publishing messages to Apache Kafka (Java implementation)
kafka: Publishing messages to Apache Kafka (C implementation, using the librdkafka client)
Shifting from Java implementation to C implementation
Before you begin
Flow control in syslog-ng OSE and the Kafka client
Options of the kafka destination's C implementation
loggly: Using Loggly
logmatic: Using Logmatic.io
mongodb: Storing messages in a MongoDB database
network: Sending messages to a remote log server using the RFC3164 protocol (network() driver)
osquery: Sending log messages to osquery's syslog table
pipe: Sending messages to named pipes
program: Sending messages to external applications
pseudofile()
python: writing custom Python destinations
redis: Storing name-value pairs in Redis
riemann: Monitoring your data with Riemann
slack: Sending alerts and notifications to a Slack channel
smtp: Generating SMTP messages (email) from logs
snmp: Sending SNMP traps
Splunk: Sending log messages to Splunk
sql: Storing messages in an SQL database
Using the sql() driver with an Oracle database
Using the sql() driver with a Microsoft SQL database
The way syslog-ng interacts with the database
sql() destination options
stomp: Publishing messages using STOMP
syslog: Sending messages to a remote logserver using the IETF-syslog protocol
syslog-ng(): Forward logs to another syslog-ng node
tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)
Telegram: Sending messages to Telegram
unix-stream, unix-dgram: Sending messages to UNIX domain sockets
usertty: Sending messages to a user terminal: usertty() destination
Write your own custom destination in Java or Python
Client-side failover
Log paths
Managing incoming and outgoing messages with flow-control
Using disk-based and memory buffering
Global options of syslog-ng OSE
TLS-encrypted message transfer
Enabling reliable disk-based buffering
Enabling normal disk-based buffering
Enabling memory buffering
About disk queue files
Filters
Using filters
Combining filters with boolean operators
Comparing macro values in filters
Using wildcards, special characters, and regular expressions in filters
Tagging messages
Filter functions
Dropping messages
Secure logging using TLS
Encrypting log messages with TLS
Mutual authentication using TLS
Password-protected keys
TLS options
template and rewrite: Format, modify, and manipulate log messages
Customize message format using macros and templates
parser: Parse and segment structured messages
Formatting messages, filenames, directories, and tablenames
Templates and macros
Date-related macros
Hard versus soft macros
Macros of syslog-ng OSE
Using template functions
Template functions of syslog-ng OSE
Modifying the on-the-wire message format
Modifying messages using rewrite rules
Replacing message parts
Setting message fields to specific values
Unsetting message fields
Creating custom SDATA fields
Setting multiple message fields to specific values
map-value-pairs: Rename value-pairs to normalize logs
Conditional rewrites
Adding and deleting tags
Rewrite the timezone of a message
Anonymizing credit card numbers
Regular expressions
Parsing syslog messages
Parsing messages with comma-separated and similar values
Parsing key=value pairs
JSON parser
XML parser
Parsing dates and timestamps
Python parser
Parsing tags
Apache access log parser
Linux audit parser
Cisco parser
Parsing enterprise-wide message model (EWMM) messages
iptables parser
Netskope parser
Sudo parser
Websense parser
Check Point Log Exporter parser
db-parser: Process message content with a pattern database (patterndb)
Classifying log messages
Using pattern databases
Correlating log messages
Enriching log messages with external data
Using parser results in filters and templates
Downloading sample pattern databases
Correlating log messages using pattern databases
Triggering actions for identified messages
Creating pattern databases
Adding metadata from an external file
Looking up GeoIP data from IP addresses (DEPRECATED)
Looking up GeoIP2 data from IP addresses
Statistics of syslog-ng
Multithreading and scaling in syslog-ng OSE
Multithreading concepts of syslog-ng OSE
Configuring multithreading
Optimizing multithreaded performance
Troubleshooting syslog-ng
Possible causes of losing log messages
Creating syslog-ng core files
Collecting debugging information with strace, truss, or tusc
Running a failure script
Stopping syslog-ng
Reporting bugs and finding help
Recover data from orphaned diskbuffer files
No local logs after specifying an unusual storage directory
No logs after specifying an unusual port number
Error messages
SELinux prevents syslog-ng OSE from using the execmem access on a process
Best practices and examples
General recommendations
Handling large message load
Using name resolution in syslog-ng
Collecting logs from chroot
Configuring log rotation
The syslog-ng manual pages
Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License
Glossary
hdfs-append-enabled()
| Type: | true | false |
| Default: | false |
Description: When hdfs-append-enabled is set to true, syslog-ng OSE will append new data to the end of an already existing HDFS file. Note that in this case, archiving is automatically disabled, and syslog-ng OSE will ignore the hdfs-archive-dir option.
When hdfs-append-enabled is set to false, the syslog-ng OSE application always creates a new file if the previous has been closed. In that case, appending data to existing files is not supported.
When you choose to write data into an existing file, syslog-ng OSE does not extend the filename with a UUID suffix because there is no need to open a new file (a new unique ID would mean opening a new file and writing data into that).
|
|
Caution:
Before enabling the hdfs-append-enabled option, ensure that your HDFS server supports the append operation and that it is enabled. Otherwise syslog-ng OSE will not be able to append data into an existing file, resulting in an error log. |
