Chat now with support
Chat mit Support

syslog-ng Premium Edition 7.0.30 - Windows Event Collector Administration Guide

Converting file-based states to Redis states

This section describes the process of converting file-based states to Redis states, and the related potential limitations and issues.

For more information about file-based states and Redis states, see States in WEC clustering (file-based states and Redis states).

Conversion steps and limitations

During the conversion process, you move your existing file-based states into Redis states. The WEC binary supports moving the file-based states with a dedicated command line option (-i): 

root@ubuntu1:/opt/syslog-ng/var/wecstate# /opt/syslog-ng/libexec/wec -h Usage of /opt/syslog-ng/libexec/wec:
-c string
Configuration file name (default "/opt/syslog-ng/etc/wec.yaml")
-i Initializing shared state from stateDir
-s string
Persistent state directory (default "/opt/syslog-ng/var/wecstate")
-v WEC version

To convert file-based states to Redis states

  1. Load your existing file-based states from your stateDir directory.

  2. Connect to Redis server.

    NOTE: If your Redis server is not available, the WEC will not start at all, and the conversion process stops.

  3. Save your previously loaded file-based states into a Redis HSET.

  4. Move the files containing your WEC states to your stateDir/.converted folder.

    CAUTION: If moving the WEC states to your stateDir/.converted folder is unsuccessful, your file-based state will contain out-of-date data. If the final conversion step is unsuccessful, and you have your -i option enabled, the out-of-date data in your file-based states overwrite the data in your Redis states after your next WEC restart (which results in message duplication). To avoid this, remove the -i command line option before you restart your WEC following your state conversion.

    NOTE: If for some reason, your WEC cluster configuration does not work as expected, your previous file-base states will not be available if you delete them. Instead, One Identity recommends moving the files to the folder, where they will be available for recovery if needed.

    NOTE: State conversion has two possible results:

    • successful

    • unsuccessful

    If any of the state conversion steps is unsuccessful, the WEC instance stops with an error. As a result, even if the file-based state is successfully saved to Redis, but moving the files is unsuccessful, the WEC instance stops and prints an error message:

    Failed to move state files, remove it manually

    This is an expected behavior, developed to avoid unwanted, huge message duplication and related issues. Instead of message duplication, if the file-based state is converted, but moving the state files is unsuccessful for some reason, you can re-initialize Redis with the state files that contain old bookmarks.

An example use case for WEC clustering

This section describes an example use case for Windows Event Collector (WEC) clustering .

For more detailed information about the working mechanism of WEC clustering, see The working mechanism of Windows Event Collector (WEC) clustering.

Limitations

Caution:

Hazard of data loss!

In syslog-ng Premium Edition (syslog-ng PE) version 7.0.23, Redis Cluster is not supported. If you attempt to set up your configuration to enable the Redis Cluster feature, your WEC cluster will not function properly.

Configuration

In the example use case for WEC clustering, the following configuration is used: 

bookmarks:
backend: redis 
uri: 192.168.0.14:6379 
password: pwd #optional

NOTE: In the example use case, multiple WEC instances are running behind the load balancer. Load balancing in this configuration will only work with multiple WEC instances if you have the same Redis backend configured on all WEC instances, with the same subscription used by all of the WEC instances. Otherwise, you will encounter message duplication.

Required components for a functional WEC cluster configuration

For a functional WEC cluster configuration, the required components are the following:

  • WEC instances, with the following considerations:

    • The WEC instances must have a shared state (on the Redis backend).

    • The WEC instances must be configured to use the same subscription.

      The subscriptions part of the configuration should be the same in every WEC instance, for example:

      subscriptions:
- batchsizelimit: 1
batchtimeoutlimit: 1.0
computers:
- '*'
connectionretry: 1.0
contentformat: RenderedText
heartbeats: 3.0
name: wec-1-subscription
queries: "<QueryList>\n  <Query Id=\"0\">\n    <Select Path=\"Application\">*</Select>\n\
\  </Query>\n</QueryList>\n"readexistingevents: false
* wec is connected to LoadBalancer (server and port is set to Load Balancer)

  • Windows instances that forward requests to load balancers.

  • An appropriately installed, set up, maintained, and monitored TCP level load balancer.

    The customer has the following responsibilities:

    • Installing the TCP level load balancer.

    • Setting up the TCP level load balancer.

    • Maintaining the TCP level load balancer.

    • Monitoring the TCP level load balancer.

    • Configuring the SSL certificates appropriately.

      NOTE: When configuring your SSL certificates while WEC clustering, consider that the load balancer functions as a proxy in your configuration.

      For more information about configuring SSL certificates for Windows Event Collectors, see Generating SSL certificates for Windows Event Collector.

  • An appropriately set up, configured, maintained, and monitored Redis server.

    The customer has the following responsibilities:

    • Setting up the Redis server.

    • Configuring the Redis server.

    • Maintaining the Redis server.

    • Monitoring the Redis server.

Troubleshooting for WEC clustering

This section provides troubleshooting information and solutions in connection with Windows Event Collector (WEC) clustering .

Log messages and why the WEC sends them

This section describes the possible log messages you may get while using Windows Event Collector (WEC) clustering with syslog-ng Premium Edition (syslog-ng PE), and why the WEC sends them.

  • If Redis is not available during startup, the WEC instance cannot start. In this case, you will get a similar log message:

    2020-11-16T21:24:03.843Z        FATAL   state/redisstate.go:17  RedisConn: Error connecting to Redis    {"error": "RedisConn: connection failure: dial tcp 192.168.0.14:6379: connect: connection refused"}

  • If Redis is disconnected, you will get a similar log message: 

    2020-11-16T21:11:12.818Z        ERROR   state/redisconn.go:55   RedisConn: dial failed  {"error": "dial tcp 192.168.0.14:6379: connect: connection refused"}

  • If you are trying to ping Redis periodically (in this case, the ping period is 1 second), you will get a similar log message:

    2020-11-16T21:11:12.818Z        DEBUG   state/redisconn.go:115  RedisConn is still disconnected
2020-11-16T21:11:13.819Z        ERROR   state/redisconn.go:55   RedisConn: dial failed  {"error": "dial tcp 192.168.0.14:6379: connect: connection refused"}

  • I Redis eventually becomes available, you will get a similar log message:

    2020-11-16T21:13:59.829Z        DEBUG   state/redisconn.go:136  RedisConn is connected
2020-11-16T21:13:59.829Z        INFO    wec/main.go:120 Redis connection restored, starting server...
2020-11-16T21:13:59.830Z        INFO    eventstorage/datagrameventstorage.go:34 Trying to connect to unix datagram socket       {"unix-datagram": "/home/vagrant/wec_unix_dgram"}
2020-11-16T21:13:59.830Z        INFO    eventstorage/datagrameventstorage.go:44 Connected to unix datagram socket       {"unix-datagram": "/home/vagrant/wec_unix_dgram"}
Verwandte Dokumente

The document was helpful.

Bewertung auswählen

I easily found the information I needed.

Bewertung auswählen