Log messages can be collected and processed at a faster rate in the latest version of syslog-ng Premium Edition compared to version 6 LTS and earlier versions but several configuration aspects will affect the rate at which log messages are collected and stored. The following tables show the results of tests performed with syslog-ng PE version 7.0. Taking the following factors into consideration will optimize syslog-ng Premium Edition performance:
Number of network connections:
In a multithreaded environment, an increase in connections will have no significant impact on the rate at which syslog-ng PE processes log messages.
| Number of Connections | Messages Per Second | Average Data Rate (MB/sec) |
|---|---|---|
|
10 |
640,000 |
240 |
|
50 |
550,000 |
205 |
|
100 |
530,000 |
200 |
|
200 |
545,000 |
205 |
|
Configuration: path – TCP, destination – multiple files (using macros), message size: 400 bytes | ||
Encrypted log transfer:
The syslog-ng PE application uses the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows the mutual authentication of the host and the server using X.509 certificates.
|
|
Legacy syslog | IETF syslog | ||
|---|---|---|---|---|
| Not Encrypted | TLS Encryption | Not Encrypted | TLS Encryption | |
|
Messages per second |
640,000 |
620,000 |
65,000 |
65,000 |
|
Average data rate (MB/sec) |
240 |
230 |
35 |
35 |
|
Configuration: path – TCP, multithreaded, 10 connections, destination – multiple files (using macros), message size: 400 bytes | ||||
|
|
Legacy syslog | IETF syslog | ||
|---|---|---|---|---|
| Not Encrypted | TLS Encryption | Not Encrypted | TLS Encryption | |
|
Messages per second |
565,000 |
565,000 |
60,000 |
60,000 |
|
Average data rate (MB/sec) |
210 |
210 |
30 |
30 |
|
Configuration: path – TCP, multithreaded, 100 connections, destination – multiple files (using macros), message size: 400 bytes | ||||
Type of storage:
The syslog-ng PE application can:
-
Send syslog messages through the network to a syslog-ng PE server.
-
Store log messages in a database.
-
Store log messages in the Hadoop Distributed File System (HDFS).
-
Send log messages to Elasticsearch.
| Type of Storage | Messages Per Second | Average Data Rate (MB/sec) |
|---|---|---|
|
Plain text file |
270,000 |
100 |
|
Multiple plain text files (using macros, with log messages divided by hostname) |
640,000 |
240 |
|
Network destination — legacy syslog |
250,000 |
95 |
|
Database destination — MongoDB |
In the case of MongoDB destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on MongoDB-related performance measurements, contact One Identity. If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer. | |
|
Database destination — SQL |
In the case of SQL destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on SQL-related performance measurements, contact One Identity. If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer. | |
|
Configuration: path – TCP, multithreaded, 10 connections, message size: 400 bytes | ||
| Type of Storage | Messages Per Second | Average Data Rate (MB/sec) |
|---|---|---|
|
Plain text file |
410,000 |
155 |
|
Multiple plain text files (using macros, with log messages divided by hostname) |
505,000 |
190 |
|
Network destination — legacy syslog |
245,000 |
90 |
|
Database destination — MongoDB |
In the case of MongoDB destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on MongoDB-related performance measurements, contact One Identity. If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer. | |
|
Database destination — SQL |
In the case of SQL destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on SQL-related performance measurements, contact One Identity. If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer. | |
|
HDFS |
110,000 |
40 |
|
|
Note: Processing speed is heavily influenced by the number of HDFS data nodes in use. When syslog-ng writes multiple files to HDFS, and Hadoop places these on different data nodes, then processing speed might increase in proportion to the number of data nodes used (not necessarily in a linear fashion). The data provided here shows performance in the case of a single data node. | |
|
Elasticsearch |
1,260 (with flush_limit(1)) 9,700 (with flush_limit(5000)) |
1 (with flush_limit(1)) 5 (with flush_limit(5000)) |
|
Configuration: path – TCP, multithreaded, 100 connections, message size: 400 bytes | ||
Number of files and directories when reading log messages from multiple plain text files:
When reading log messages from a set of files, the number of directories and the number of files per directory used have no significant impact on performance.
| Number of Directories | Number of Files Per Directory | Messages Per Second | Average Data Rate (MB/sec) |
|---|---|---|---|
|
1 |
1 |
110,000 |
45 |
|
10 |
175,000 |
70 | |
|
100 |
150,000 |
60 | |
|
10 |
1 |
180,000 |
70 |
|
10 |
150,000 |
60 | |
|
100 |
130,000 |
50 | |
|
100 |
1 |
150,000 |
60 |
|
10 |
130,000 |
50 | |
|
100 |
130,000 |
50 | |
|
Configuration: path – TCP, multithreaded, monitor-method(inotify), File source message size: 400 bytes | |||
| Number of Directories | Number of Files Per Directory | Messages Per Second | Average Data Rate (MB/sec) |
|---|---|---|---|
|
1 |
110,000 |
45 | |
|
10 |
165,000 |
65 | |
|
100 |
150,000 |
60 | |
|
10 |
1 |
175,000 |
70 |
|
10 |
150,000 |
60 | |
|
100 |
130,000 |
50 | |
|
100 |
1 |
150,000 |
60 |
|
10 |
130,000 |
50 | |
|
100 |
125,000 |
50 | |
|
Configuration: path – TCP, multithreaded, monitor-method(poll), File source message size: 400 bytes | |||
Disk buffer:
The syslog-ng Premium Edition stores messages on the local hard disk if the central log server or the network connection to the server becomes unavailable.
| Without Disk Buffer | Reliable | Normal | |
|---|---|---|---|
|
Messages per second |
345,000 |
40,000 |
60,000 |
|
Average data rate (MB/sec) |
130 |
15 |
20 |
|
Configuration: path – TCP, multithreaded, disk buffer: 1000 MB, 100 connections, message size: 400 bytes | |||
Log pre-processing:
Depending on the type of pre-processing, the rate at which syslog-ng PE collects messages can vary. Rewriting, using parsers, as well as pattern recognition processing through PatternDB have a significant impact on the message processing rate. Regular expressions have only a light impact, while facility filtering and tag filtering have virtually no impact at all.
Note that in a multithreaded environment, PatternDB has a particularly large impact on performance.
When combining multiple types of pre-processing, processing rate will drop below the processing rate of the slowest pre-processing method used.
| Messages Per Second | Average Data Rate (MB/sec) | |
|---|---|---|
|
No pre-processing |
680,000 |
255 |
|
Simple regexp (for example, matching a single string) |
570,000 |
210 |
|
Facility filter |
670,000 |
250 |
|
Tag filter |
650,000 |
245 |
|
PatternDB (10 % of messages matched) |
40,000 |
15 |
|
Simple rewrite (for example, rewrite hostname) |
245,000 |
90 |
|
Python parser |
35,000 |
15 |
|
JSON parser |
40,000 |
25 |
|
kv parser |
190,000 |
70 |
|
XML parser |
15,000 |
20 |
|
Configuration: path – TCP, multithreaded, 10 connections, message size: 400 bytes | ||
| Messages Per Second | Average Data Rate (MB/sec) | |
|---|---|---|
|
No pre-processing |
515,000 |
195 |
|
Simple regexp (for example, matching a single string) |
510,000 |
190 |
|
Facility filter |
500,000 |
185 |
|
Tag filter |
530,000 |
200 |
|
PatternDB (10 % of messages matched) |
35,000 |
15 |
|
Simple rewrite (for example, rewrite hostname) |
360,000 |
135 |
|
Python parser |
35,000 |
15 |
|
JSON parser |
35,000 |
25 |
|
kv parser |
140,000 |
50 |
|
XML parser |
15,000 |
15 |
|
Configuration: path – TCP, multithreaded, 100 connections, message size: 400 bytes | ||