Log collection and search
Log collection and search
Measurement setup:
An indexed logspace was set up with all fields indexed. 25,000,000 log messages were sent via TLS source into the logspace every hour for 24 hours to represent "historical" data. Then we kept sending 25,000,000 log messages every hour, while executing an increasing number of parallel search queries to simulate 1 to 29 concurrent busy users.
User simulation was achieved by executing the same RPC API queries that a front end (i.e. internet browser) would send. These are: splitting the time interval in question into 30 equal parts and calculating the number of results per interval (these are the bars on the search interface) and also fetching the first 200 results (part of this is show at the bottom of the search interface).
Measurement quantities:
-
Rate in 1000x message/second, that is, how many log messages we can send into SSB while running searches at the same time.
-
Number of searches finished in 3 minutes of the test.
Case 1
The search queries are very simple exact token searches. Memory limit for the logspace is 1024Mb. Search was for X<n> OR Y<m> with variable n,m numbers to avoid effects of the query cache of SSB.
Figure 1: Case 1: T1
Figure 2: Case 1: T4
Figure 3: Case 1: T10
Case 2
Addition of *Z to the start or the query to force much more disk IO usage. Results were the same regarding throughput and obviously lower for number of searches.
Figure 4: Case 2: T1
Performance of SSB versions
Performance of SSB versions
The following charts show how the search performance of SSB has improved, using SSB 4 LTS (4.0) as a baseline. We used relatively longer queries, where the response time was at least 2 seconds, so that the results are easier to measure and compare. Otherwise, various other factors could have distorted the measurements. Note the following point:
-
The structure of index files was optimized in SSB 4 F2, greatly increasing the search performance of SSB. If searching is slow in your SSB and you are using SSB 4 LTS, consider upgrading to a newer version. (Note that this will affect only the index files created after the upgrade.)
-
The search algorithms were optimized in SSB 4 F2, decreasing the memory usage of search by an average of 80%. If search causes high memory consumption in your SSB and you are using SSB 4 LTS, consider upgrading to a newer version.
Figure 5: Average response time
Figure 6: Relative response time
Summary
The test measurements show that the processing capabilities and search performance of syslog-ng Store Box have increased significantly since version 4 LTS, and that SSB is capable of receiving and processing high-volume log traffic. The largest SSB appliance is capable of scaling up to 100,000 event per second (100k EPS).
If the search performance of SSB is not adequate in your environment (search is slow, or greatly increases the memory consumption), check the version of your SSB. If you are using SSB 4 LTS, consider upgrading to a newer version.
If you have questions about the performance of SSB, or need help in optimizing the configuration of your SSB appliance, contact our Professional Services Team.
