Voltar
-
Título
Balancing logs using RCPTID and MOD -
Descrição
Sending logs to different destinations, or to the same destination using multiple connections, to "load-balance" the number of logs coming in is desired. -
Causa
Cause 1.)The endpoint which is receiving the forwarded logs from the Syslog-ng PE server and/or relay receives too many logs too quickly.
Cause 2.)Logs need to be sent to 2 or more locations to "load-balance" the number of logs. -
Resolução
(An example configuration for balancing logs across 4 destination servers can be found at the end of this knowledge base article for reference)
Change the configuration of the Syslog-ng PE server and/or relay to "load-balance" the logs to multiple destinations, or a single destination using multiple connections, based upon the Receipt ID (RCPTID) of the log processed by the Syslog-ng PE server/relay.
Each log received has a unique ID applied to the log starting at the number 1 and growing by 1 whole integer with each log that is received.
This RCPTID is used to balance the logs using a Modulo function (MOD) and by changing the MOD number the logs can be balanced between as many as destinations as desired.
To implement this change follow the steps below editting the syslog-ng.conf file, with any text edittor, located at /opt/syslog-ng/etc/syslog-ng.conf
Step 1 - Add the use-uniqid(yes) option to the global options, an example of this is below:@version: 7.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # https://www.balabit.com/support/documentation # @include "scl.conf" options { stats_freq(0); use-uniqid(yes); };
Step 2 - Create the filter statements in which the logs will be balanced. An example of this is below:filter f_rcptid_1_4 { "$(% ${RCPTID} 4)" == "0" }; filter f_rcptid_2_4 { "$(% ${RCPTID} 4)" == "1" }; filter f_rcptid_3_4 { "$(% ${RCPTID} 4)" == "2" }; filter f_rcptid_4_4 { "$(% ${RCPTID} 4)" == "3" };The number 4 in this line: "$(% ${RCPTID} 4)" is where the Modulo function is defined and in this example, 4 filters are created to balance logs across 4 destinations.
If the number of destinations to balance across was 3 for example, the Modulo function would be changed to 3 and only 3 filters would be used (see below example):filter f_rcptid_1_3 { "$(% ${RCPTID} 3)" == "0" }; filter f_rcptid_2_3 { "$(% ${RCPTID} 3)" == "1" }; filter f_rcptid_3_3 { "$(% ${RCPTID} 3)" == "2" };The number 2 in this line: == "2" is the defining variable which is the number called after the Modulo function has completed.
Step 3 - Define as many log paths as there are filter statements, for example, if balancing between 7 destinations, 7 log path statements would need to be created. In addition, the filter needs to be defined in the log path as seen in the example below (replacing the filter name with the corresponding filter name created):log { source(s_testlog); filter(f_example_filter); destination(d_example_destination); };
Step 4 - Save the syslog-ng.conf file and restart the syslog-ng service. Logs should now be balanced between the number of destinations desired.
An example configuration for balancing logs across 4 destination servers can be found below:@version: 7.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # https://www.balabit.com/support/documentation # @include "scl.conf" options { stats_freq(0); use-uniqid(yes); }; ############## # sources # ############## source s_testlog { syslog( ip("0.0.0.0") port("601") transport("tcp") flags(no-parse) ); }; ################### # Destinations # ################### destination d_lb_endpoint_0 { syslog( "10.10.10.11" port("1011") transport("tcp") ); }; destination d_lb_endpoint_1 { syslog( "10.10.10.12" port("1012") transport("tcp") ); }; destination d_lb_endpoint_0 { syslog( "10.10.10.13" port("1013") transport("tcp") ); }; destination d_lb_endpoint_1 { syslog( "10.10.10.14" port("1014") transport("tcp") ); }; ############## # Filters # ############## filter f_rcptid_1_4 { "$(% ${RCPTID} 4)" == "0" }; filter f_rcptid_2_4 { "$(% ${RCPTID} 4)" == "1" }; filter f_rcptid_3_4 { "$(% ${RCPTID} 4)" == "2" }; filter f_rcptid_4_4 { "$(% ${RCPTID} 4)" == "3" }; ################ # Log Paths # ################ log { source(s_testlog); filter(f_rcptid_1_4); destination(d_lb_endpoint_1); flags(final); }; log { source(s_testlog); filter(f_rcptid_2_4); destination(d_lb_endpoint_2); flags(final); }; log { source(s_testlog); filter(f_rcptid_3_4); destination(d_lb_endpoint_3); flags(final); }; log { source(s_testlog); filter(f_rcptid_4_4); destination(d_lb_endpoint_4); flags(final); };