Chat now with support
Chat mit Support

syslog-ng Premium Edition 7.0.34 - Performance Guideline for syslog-ng Premium Edition

Configuration guidelines

Log messages can be collected and processed at a faster rate in the latest version of syslog-ng Premium Edition compared to version 6 LTS and earlier versions but several configuration aspects will affect the rate at which log messages are collected and stored. The following tables show the results of tests performed with syslog-ng PE version 7.0. Taking the following factors into consideration will optimize syslog-ng Premium Edition performance:

Number of network connections:

In a multithreaded environment, an increase in connections will have no significant impact on the rate at which syslog-ng PE processes log messages.

Table 1: Number of network connections
Number of Connections Messages Per Second Average Data Rate (MB/sec)

10

640,000

240

50

550,000

205

100

530,000

200

200

545,000

205

Configuration: path – TCP, destination – multiple files (using macros), message size: 400 bytes

Encrypted log transfer:

The syslog-ng PE application uses the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows the mutual authentication of the host and the server using X.509 certificates.

Table 2: Encrypted log transfer — 10 connections

 

Legacy syslog IETF syslog
Not Encrypted TLS Encryption Not Encrypted TLS Encryption

Messages per second

640,000

620,000

65,000

65,000

Average data rate (MB/sec)

240

230

35

35

Configuration: path – TCP, multithreaded, 10 connections, destination – multiple files (using macros), message size: 400 bytes

Table 3: Encrypted log transfer — 100 connections

 

Legacy syslog IETF syslog
Not Encrypted TLS Encryption Not Encrypted TLS Encryption

Messages per second

565,000

565,000

60,000

60,000

Average data rate (MB/sec)

210

210

30

30

Configuration: path – TCP, multithreaded, 100 connections, destination – multiple files (using macros), message size: 400 bytes

Type of storage:

The syslog-ng PE application can:

Table 4: Type of storage — 10 connections
Type of Storage Messages Per Second Average Data Rate (MB/sec)

Plain text file

270,000

100

Multiple plain text files (using macros, with log messages divided by hostname)

640,000

240

Network destination — legacy syslog

250,000

95

Database destination — MongoDB

In the case of MongoDB destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on MongoDB-related performance measurements, contact One Identity.

If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer.

Database destination — SQL

In the case of SQL destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on SQL-related performance measurements, contact One Identity.

If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer.

Configuration: path – TCP, multithreaded, 10 connections, message size: 400 bytes

Table 5: Type of storage — 100 connections
Type of Storage Messages Per Second Average Data Rate (MB/sec)

Plain text file

410,000

155

Multiple plain text files (using macros, with log messages divided by hostname)

505,000

190

Network destination — legacy syslog

245,000

90

Database destination — MongoDB

In the case of MongoDB destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on MongoDB-related performance measurements, contact One Identity.

If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer.

Database destination — SQL

In the case of SQL destinations, performance is influenced by a number of criteria unrelated to syslog-ng. If you need information on SQL-related performance measurements, contact One Identity.

If you are an existing customer, contact our Support Team. Otherwise, contact your Pre-Sales Engineer.

HDFS

110,000

40

Note: Processing speed is heavily influenced by the number of HDFS data nodes in use. When syslog-ng writes multiple files to HDFS, and Hadoop places these on different data nodes, then processing speed might increase in proportion to the number of data nodes used (not necessarily in a linear fashion). The data provided here shows performance in the case of a single data node.

Elasticsearch

1,260 (with flush_limit(1))

9,700 (with flush_limit(5000))

1 (with flush_limit(1))

5 (with flush_limit(5000))

Configuration: path – TCP, multithreaded, 100 connections, message size: 400 bytes

Number of files and directories when reading log messages from multiple plain text files:

When reading log messages from a set of files, the number of directories and the number of files per directory used have no significant impact on performance.

Table 6: Number of files and directories — using the inotify monitor method
Number of Directories Number of Files Per Directory Messages Per Second Average Data Rate (MB/sec)

1

1

110,000

45

10

175,000

70

100

150,000

60

10

1

180,000

70

10

150,000

60

100

130,000

50

100

1

150,000

60

10

130,000

50

100

130,000

50

Configuration: path – TCP, multithreaded, monitor-method(inotify), File source message size: 400 bytes

Table 7: Number of files and directories — using the poll monitor method
Number of Directories Number of Files Per Directory Messages Per Second Average Data Rate (MB/sec)

1

110,000

45

10

165,000

65

100

150,000

60

10

1

175,000

70

10

150,000

60

100

130,000

50

100

1

150,000

60

10

130,000

50

100

125,000

50

Configuration: path – TCP, multithreaded, monitor-method(poll), File source message size: 400 bytes

Disk buffer:

The syslog-ng Premium Edition stores messages on the local hard disk if the central log server or the network connection to the server becomes unavailable.

Table 8: Disk buffer
Without Disk Buffer Reliable Normal

Messages per second

345,000

40,000

60,000

Average data rate (MB/sec)

130

15

20

Configuration: path – TCP, multithreaded, disk buffer: 1000 MB, 100 connections, message size: 400 bytes

Log pre-processing:

Depending on the type of pre-processing, the rate at which syslog-ng PE collects messages can vary. Rewriting, using parsers, as well as pattern recognition processing through PatternDB have a significant impact on the message processing rate. Regular expressions have only a light impact, while facility filtering and tag filtering have virtually no impact at all.

Note that in a multithreaded environment, PatternDB has a particularly large impact on performance.

When combining multiple types of pre-processing, processing rate will drop below the processing rate of the slowest pre-processing method used.

Table 9: Log pre-processing — 10 connections
Messages Per Second Average Data Rate (MB/sec)

No pre-processing

680,000

255

Simple regexp (for example, matching a single string)

570,000

210

Facility filter

670,000

250

Tag filter

650,000

245

PatternDB (10 % of messages matched)

40,000

15

Simple rewrite (for example, rewrite hostname)

245,000

90

Python parser

35,000

15

JSON parser

40,000

25

kv parser

190,000

70

XML parser

15,000

20

Configuration: path – TCP, multithreaded, 10 connections, message size: 400 bytes

Table 10: Log pre-processing — 100 connections
Messages Per Second Average Data Rate (MB/sec)

No pre-processing

515,000

195

Simple regexp (for example, matching a single string)

510,000

190

Facility filter

500,000

185

Tag filter

530,000

200

PatternDB (10 % of messages matched)

35,000

15

Simple rewrite (for example, rewrite hostname)

360,000

135

Python parser

35,000

15

JSON parser

35,000

25

kv parser

140,000

50

XML parser

15,000

15

Configuration: path – TCP, multithreaded, 100 connections, message size: 400 bytes

The test environment

The test environment consisted of a single client and a server hardware, connected via a Gigabit switch. Note that in certain test runs, the client opened several separate connections to the servers to simulate real-life logging environments. The syslog-ng Premium Edition application was installed from the .run package.

Hardware parameters:

The client hardware had the following main parameters:

  • 2x Intel® Xeon® Processor E5-2620 v3 (15M Cache, 2.40 GHz, 8 GT/s Intel® QPI, 6 cores)

  • Hyperthreading disabled, turbo boost disabled

  • 16 GB RAM

  • 10 Gbps Ethernet

  • HDD 500 GB

  • Operating system: ubuntu-xenial amd64

The server hardware had the following main parameters:

  • 2x Intel® Xeon® Processor E5-2620 v3 (15M Cache, 2.40 GHz, 8 GT/s Intel® QPI, 6 cores)

  • Hyperthreading disabled, turbo boost disabled

  • 16 GB RAM

  • 10 Gbps Ethernet

  • SSD 500 GB

  • Operating system: ubuntu-xenial amd64

Performance improvement:

The following settings were used for performance improvement:

  • Improving performance with lots of connections:

    max_connections = active_connections log_iw_size = number of active_connections * 1000 log_fetch_limit = 1000 flush_lines = 1000 log_fifo_size = log_iw_size * 2 use_dns = no keep_hostname = yes keep_timestamp = no

  • Improving performance with a few connections but high amount of traffic:

    Source side:

    log_iw_size = number of active_connections * 100,000 log_fetch_limit = number of active_connections * 100,000

    Destination side: log_fifo_size = max_connections * (log_iw_size/number of active_connections) flush_lines = 10,000 or greater

Resource usage:

The performance tests were carried out in multithreaded mode:

threaded(yes)

One way to optimize the resource usage of syslog-ng PE is to limit the number of worker threads that syslog-ng uses. This helps prevent syslog-ng PE from using all available CPUs. You can limit the number of worker threads using the --worker-threads command-line option that sets the maximum total number of threads syslog-ng PE can use, including the main syslog-ng PE thread.

Note, however, that SQL sources and destinations, as well as Java destinations, such as Elasticsearch, HDFS, and Apache Kafka, always run in their own, separate threads. This means that the --worker-threads command-line option has no impact on them.

Verwandte Dokumente

The document was helpful.

Bewertung auswählen

I easily found the information I needed.

Bewertung auswählen