In Windows Event Forwarders, the subscription's status at a given time is recorded in a file (or a Redis key-value database, depending on your use case) that is commonly called a state. Accordingly, while discussing Windows Event Collector (WEC) clustering , these sections mention two types of states: file-based states and Redis states.
NOTE: File-based states are local states and are not synchronized between multiple WEC instances (as opposed to shared states) File-based states can only be used locally, by a single WEC instance at a time.
The three data types within file-based states
States that are recorded in a file (that is, file-based states) consist of three data types:
-
An event source (also called a host), where the events are received.
-
A subscription ID.
-
A bookmark (also called a position).
In case of file-based states, this information is stored within files (in the stateDir directory). Each file stored in the stateDir directory contains subscription IDs and positions to the given host, one file each.
Example: data structure within the stateDir directory
stateDir\win1: subscription_1 - position_1 subscription_2 - position_2 ... subscription_n - position_n
NOTE: Real state files store these data in a different format. This format is only true for the information structure within the stateDir directory.
Mapping the file-based state structure to Redis states
If the user wants to use Redis states instead of file-based states while WEC clustering, the information has to be mapped to HSET data structures, a data structure native to Redis. Event sources are capable of identifying HSET data structures in file-based states' data structure contents, and the same state structure of subscription IDs and positions to the given host is mapped to Redis states as follows:
-
In Redis, the Redis connections are mapped to the stateDir directory of the file-based state.
-
In Redis, the HSETs are mapped to the file name of the event source (or host) of the file-based state.
-
In Redis, the HKEYs within the HSETs are mapped to subscription IDs of the file-based state.
-
In Redis, the HVALUEs within the HSETs are mapped to bookmarks (positions) of the file-based state.
What happens if the WEC instance is disconnected from a load balancer?
When a WEC instance is disconnected from a load balancer, the subscriptions connected to this instance are redirected to a different available WEC instance. When the new WEC instance receives the subscription, the WEC instance looks the subscription up in the Redis state, then populates the position from Redis. After this, the WEC instance continues processing requests from the last position loaded from the Redis backend. For each event (batch), the WEC instance writes bookmarks to Redis.
NOTE: Redis manages concurrent read or write operations.
What happens if Redis becomes unavailable for a WEC?
If for some reason, Redis becomes unavailable for a WEC istance, the WEC instance performs the following, in this order:
-
The WEC instance stops the listener.
-
The WEC instance stops all TCP connections established between the load balancer and the WEC instance.
-
The WEC instance stops sending messages to syslog-ng Premium Edition (syslog-ng PE).
-
The WEC instance starts to ping Redis.
NOTE: The load balancer is connected to the listener, so when the WEC instance is disconnected, the load balancer redirects the Windows event source to a different WEC instance. When Redis becomes available again, the load balancer detects it, and starts forwarding the events to the original WEC instance.
This section describes the process of converting file-based states to Redis states, and the related potential limitations and issues.
For more information about file-based states and Redis states, see States in WEC clustering (file-based states and Redis states).
Conversion steps and limitations
During the conversion process, you move your existing file-based states into Redis states. The WEC binary supports moving the file-based states with a dedicated command line option (-i):
root@ubuntu1:/opt/syslog-ng/var/wecstate# /opt/syslog-ng/libexec/wec -h Usage of /opt/syslog-ng/libexec/wec:
-c string
Configuration file name (default "/opt/syslog-ng/etc/wec.yaml")
-i Initializing shared state from stateDir
-s string
Persistent state directory (default "/opt/syslog-ng/var/wecstate")
-v WEC version
To convert file-based states to Redis states
-
Load your existing file-based states from your stateDir directory.
-
Connect to Redis server.
NOTE: If your Redis server is not available, the WEC will not start at all, and the conversion process stops.
-
Save your previously loaded file-based states into a Redis HSET.
-
Move the files containing your WEC states to your stateDir/.converted folder.
|
|
CAUTION: If moving the WEC states to your stateDir/.converted folder is unsuccessful, your file-based state will contain out-of-date data. If the final conversion step is unsuccessful, and you have your -i option enabled, the out-of-date data in your file-based states overwrite the data in your Redis states after your next WEC restart (which results in message duplication). To avoid this, remove the -i command line option before you restart your WEC following your state conversion. |
NOTE: If for some reason, your WEC cluster configuration does not work as expected, your previous file-base states will not be available if you delete them. Instead, One Identity recommends moving the files to the folder, where they will be available for recovery if needed.
NOTE: State conversion has two possible results:
If any of the state conversion steps is unsuccessful, the WEC instance stops with an error. As a result, even if the file-based state is successfully saved to Redis, but moving the files is unsuccessful, the WEC instance stops and prints an error message:
Failed to move state files, remove it manually
This is an expected behavior, developed to avoid unwanted, huge message duplication and related issues. Instead of message duplication, if the file-based state is converted, but moving the state files is unsuccessful for some reason, you can re-initialize Redis with the state files that contain old bookmarks.
This section describes an example use case for Windows Event Collector (WEC) clustering .
For more detailed information about the working mechanism of WEC clustering, see The working mechanism of Windows Event Collector (WEC) clustering.
Limitations
|
|
Caution:
Hazard of data loss!
In syslog-ng Premium Edition (syslog-ng PE) version 7.0.23, Redis Cluster is not supported. If you attempt to set up your configuration to enable the Redis Cluster feature, your WEC cluster will not function properly. |
Configuration
In the example use case for WEC clustering, the following configuration is used:
bookmarks:
backend: redis
uri: 192.168.0.14:6379
password: pwd #optional
NOTE: In the example use case, multiple WEC instances are running behind the load balancer. Load balancing in this configuration will only work with multiple WEC instances if you have the same Redis backend configured on all WEC instances, with the same subscription used by all of the WEC instances. Otherwise, you will encounter message duplication.
Required components for a functional WEC cluster configuration
For a functional WEC cluster configuration, the required components are the following:
-
WEC instances, with the following considerations:
-
The WEC instances must have a shared state (on the Redis backend).
-
The WEC instances must be configured to use the same subscription.
The subscriptions part of the configuration should be the same in every WEC instance, for example:
subscriptions:
- batchsizelimit: 1
batchtimeoutlimit: 1.0
computers:
- '*'
connectionretry: 1.0
contentformat: RenderedText
heartbeats: 3.0
name: wec-1-subscription
queries: "<QueryList>\n <Query Id=\"0\">\n <Select Path=\"Application\">*</Select>\n\
\ </Query>\n</QueryList>\n"readexistingevents: false
* wec is connected to LoadBalancer (server and port is set to Load Balancer)
-
Windows instances that forward requests to load balancers.
-
An appropriately installed, set up, maintained, and monitored TCP level load balancer.
The customer has the following responsibilities:
-
Installing the TCP level load balancer.
-
Setting up the TCP level load balancer.
-
Maintaining the TCP level load balancer.
-
Monitoring the TCP level load balancer.
-
Configuring the SSL certificates appropriately.
NOTE: When configuring your SSL certificates while WEC clustering, consider that the load balancer functions as a proxy in your configuration.
For more information about configuring SSL certificates for Windows Event Collectors, see Generating SSL certificates for Windows Event Collector.
-
An appropriately set up, configured, maintained, and monitored Redis server.
The customer has the following responsibilities:
-
Setting up the Redis server.
-
Configuring the Redis server.
-
Maintaining the Redis server.
-
Monitoring the Redis server.
This section provides troubleshooting information and solutions in connection with Windows Event Collector (WEC) clustering .
