syslog-ng Premium Edition 7.0.29 - Performance Guideline for syslog-ng Premium Edition

Log messages can be collected and processed at a faster rate in the latest version of syslog-ng Premium Edition compared to version 6 LTS and earlier versions but several configuration aspects will affect the rate at which log messages are collected and stored. The following tables show the results of tests performed with syslog-ng PE version 7.0. Taking the following factors into consideration will optimize syslog-ng Premium Edition performance:

Number of network connections:

In a multithreaded environment, an increase in connections will have no significant impact on the rate at which syslog-ng PE processes log messages.

Table 1: Number of network connections
Number of Connections	Messages Per Second	Average Data Rate (MB/sec)
10	640,000	240
50	550,000	205
100	530,000	200
200	545,000	205
Configuration: path – TCP, destination – multiple files (using macros), message size: 400 bytes

Encrypted log transfer:

The syslog-ng PE application uses the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows the mutual authentication of the host and the server using X.509 certificates.

Table 2: Encrypted log transfer — 10 connections
	Legacy syslog		IETF syslog
	Not Encrypted	TLS Encryption	Not Encrypted	TLS Encryption
Messages per second	640,000	620,000	65,000	65,000
Average data rate (MB/sec)	240	230	35	35
Configuration: path – TCP, multithreaded, 10 connections, destination – multiple files (using macros), message size: 400 bytes

Table 3: Encrypted log transfer — 100 connections
	Legacy syslog		IETF syslog
	Not Encrypted	TLS Encryption	Not Encrypted	TLS Encryption
Messages per second	565,000	565,000	60,000	60,000
Average data rate (MB/sec)	210	210	30	30
Configuration: path – TCP, multithreaded, 100 connections, destination – multiple files (using macros), message size: 400 bytes

Type of storage:

The syslog-ng PE application can:

Send syslog messages through the network to a syslog-ng PE server.
Store log messages in a database.
Store log messages in the Hadoop Distributed File System (HDFS).
Send log messages to Elasticsearch.

Table 4: Type of storage — 10 connections
Type of Storage	Messages Per Second	Average Data Rate (MB/sec)
Plain text file	270,000	100
Multiple plain text files (using macros, with log messages divided by hostname)	640,000	240
Network destination — legacy syslog	250,000	95
Database destination — MongoDB	In the case of MongoDB destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on MongoDB-related performance measurements, contact One Identity. If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer.
Database destination — SQL	In the case of SQL destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on SQL-related performance measurements, contact One Identity. If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer.
Configuration: path – TCP, multithreaded, 10 connections, message size: 400 bytes

Table 5: Type of storage — 100 connections
Type of Storage	Messages Per Second	Average Data Rate (MB/sec)
Plain text file	410,000	155
Multiple plain text files (using macros, with log messages divided by hostname)	505,000	190
Network destination — legacy syslog	245,000	90
Database destination — MongoDB	In the case of MongoDB destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on MongoDB-related performance measurements, contact One Identity. If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer.
Database destination — SQL	In the case of SQL destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on SQL-related performance measurements, contact One Identity. If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer.
HDFS	110,000	40
	Note: Processing speed is heavily influenced by the number of HDFS data nodes in use. When syslog-ng writes multiple files to HDFS, and Hadoop places these on different data nodes, then processing speed might increase in proportion to the number of data nodes used (not necessarily in a linear fashion). The data provided here shows performance in the case of a single data node.
Elasticsearch	1,260 (with flush_limit(1)) 9,700 (with flush_limit(5000))	1 (with flush_limit(1)) 5 (with flush_limit(5000))
Configuration: path – TCP, multithreaded, 100 connections, message size: 400 bytes

Number of files and directories when reading log messages from multiple plain text files:

When reading log messages from a set of files, the number of directories and the number of files per directory used have no significant impact on performance.

Table 6: Number of files and directories — using the inotify monitor method
Number of Directories	Number of Files Per Directory	Messages Per Second	Average Data Rate (MB/sec)
1	1	110,000	45
	10	175,000	70
	100	150,000	60
10	1	180,000	70
	10	150,000	60
	100	130,000	50
100	1	150,000	60
	10	130,000	50
	100	130,000	50
Configuration: path – TCP, multithreaded, monitor-method(inotify), File source message size: 400 bytes

Table 7: Number of files and directories — using the poll monitor method
Number of Directories	Number of Files Per Directory	Messages Per Second	Average Data Rate (MB/sec)
1	1	110,000	45
	10	165,000	65
	100	150,000	60
10	1	175,000	70
	10	150,000	60
	100	130,000	50
100	1	150,000	60
	10	130,000	50
	100	125,000	50
Configuration: path – TCP, multithreaded, monitor-method(poll), File source message size: 400 bytes

Disk buffer:

The syslog-ng Premium Edition stores messages on the local hard disk if the central log server or the network connection to the server becomes unavailable.

Table 8: Disk buffer
	Without Disk Buffer	Reliable	Normal
Messages per second	345,000	40,000	60,000
Average data rate (MB/sec)	130	15	20
Configuration: path – TCP, multithreaded, disk buffer: 1000 MB, 100 connections, message size: 400 bytes

Log pre-processing:

Depending on the type of pre-processing, the rate at which syslog-ng PE collects messages can vary. Rewriting, using parsers, as well as pattern recognition processing through PatternDB have a significant impact on the message processing rate. Regular expressions have only a light impact, while facility filtering and tag filtering have virtually no impact at all.

Note that in a multithreaded environment, PatternDB has a particularly large impact on performance.

When combining multiple types of pre-processing, processing rate will drop below the processing rate of the slowest pre-processing method used.

Table 9: Log pre-processing — 10 connections
	Messages Per Second	Average Data Rate (MB/sec)
No pre-processing	680,000	255
Simple regexp (for example, matching a single string)	570,000	210
Facility filter	670,000	250
Tag filter	650,000	245
PatternDB (10 % of messages matched)	40,000	15
Simple rewrite (for example, rewrite hostname)	245,000	90
Python parser	35,000	15
JSON parser	40,000	25
kv parser	190,000	70
XML parser	15,000	20
Configuration: path – TCP, multithreaded, 10 connections, message size: 400 bytes

Table 10: Log pre-processing — 100 connections
	Messages Per Second	Average Data Rate (MB/sec)
No pre-processing	515,000	195
Simple regexp (for example, matching a single string)	510,000	190
Facility filter	500,000	185
Tag filter	530,000	200
PatternDB (10 % of messages matched)	35,000	15
Simple rewrite (for example, rewrite hostname)	360,000	135
Python parser	35,000	15
JSON parser	35,000	25
kv parser	140,000	50
XML parser	15,000	15
Configuration: path – TCP, multithreaded, 100 connections, message size: 400 bytes

The test environment consisted of a single client and a server hardware, connected via a Gigabit switch. Note that in certain test runs, the client opened several separate connections to the servers to simulate real-life logging environments. The syslog-ng Premium Edition application was installed from the .run package.

Hardware parameters:

The client hardware had the following main parameters:

2x Intel® Xeon® Processor E5-2620 v3 (15M Cache, 2.40 GHz, 8 GT/s Intel® QPI, 6 cores)
Hyperthreading disabled, turbo boost disabled
16 GB RAM
10 Gbps Ethernet
HDD 500 GB
Operating system: ubuntu-xenial amd64

The server hardware had the following main parameters:

2x Intel® Xeon® Processor E5-2620 v3 (15M Cache, 2.40 GHz, 8 GT/s Intel® QPI, 6 cores)
Hyperthreading disabled, turbo boost disabled
16 GB RAM
10 Gbps Ethernet
SSD 500 GB
Operating system: ubuntu-xenial amd64

Performance improvement:

The following settings were used for performance improvement:

Improving performance with lots of connections:

max_connections = active_connections log_iw_size = number of active_connections * 1000 log_fetch_limit = 1000 flush_lines = 1000 log_fifo_size = log_iw_size * 2 use_dns = no keep_hostname = yes keep_timestamp = no
Improving performance with a few connections but high amount of traffic:

Source side:

log_iw_size = number of active_connections * 100,000 log_fetch_limit = number of active_connections * 100,000
Destination side:
log_fifo_size = max_connections * (log_iw_size/number of active_connections) flush_lines = 10,000 or greater

Resource usage:

The performance tests were carried out in multithreaded mode:

threaded(yes)

One way to optimize the resource usage of syslog-ng PE is to limit the number of worker threads that syslog-ng uses. This helps prevent syslog-ng PE from using all available CPUs. You can limit the number of worker threads using the --worker-threads command-line option that sets the maximum total number of threads syslog-ng PE can use, including the main syslog-ng PE thread.

Note, however, that SQL sources and destinations, as well as Java destinations, such as Elasticsearch, HDFS, and Apache Kafka, always run in their own, separate threads. This means that the --worker-threads command-line option has no impact on them.

Please select your product:

To serve you better, please complete the Purpose of your Chat:

Recommended Solutions for Your Problem