Chat now with support
Chat with Support

syslog-ng Store Box 6.1.0 - Administration Guide

Preface Introduction The concepts of SSB The Welcome Wizard and the first login Basic settings User management and access control Managing SSB Configuring message sources Storing messages on SSB Forwarding messages from SSB Log paths: routing and processing messages Configuring syslog-ng options Searching log messages Searching the internal messages of SSB Classifying messages with pattern databases The SSB RPC API Monitoring SSB Troubleshooting SSB Security checklist for configuring SSB

Preventing disk space fill up

The following describes how to prevent disk space from filling up.

To prevent disk space from filling up

  1. Navigate to Basic Settings > Management > Disk space fill up prevention.

  2. Set the limit of maximum disk utilization in percents in the respective field. When disk space is used above the set limit, SSB disconnects all clients. The default value is 90, and you can set values between 1-99.

  3. Optional step: Enable the Automatically start archiving option to automatically start all configured archiving/cleanup jobs when disk usage goes over the limit.

    NOTE:

    If there is no archiving policy set, enabling this option will not trigger automatic archiving.

  4. Navigate to Basic Settings > Alerting & Monitoring > System related traps and enable alert Disk usage is above the defined ratio.

  5. Click Commit.

Configuring message rate alerting

With message rate alerting, you can detect the following abnormalities in SSB:

  • The syslog-ng inside SSB has stopped working.

  • One of the clients/sites sending logs is not detectable.

  • One of the clients/sites is sending too many logs, probably unnecessarily.

Message rate alerting can be set for sources, spaces and destinations (remote or local).

To configure message rate alerting

  1. Navigate to Log and select Sources, Spaces or Destinations.

  2. Enable Message rate alerting.

  3. In case of Sources, select the counter to be measured:

    • Messages: Number of messages

    • Messages/sender: Number of messages per sender (the last hop)

    • Messages/hostname: Number of messages per host (based on the hostname in the message)

    In case of Spaces or Destinations, the counter is the number of messages.

  4. Select the time period (between 5 minutes and 24 hours) during which the range is to be measured.

  5. Enter the range that is considered normal in the Minimum and Maximum fields.

  6. Select the alerting frequency in the Alert field. Once sends only one alert (and after the problem is fixed, a "Fixed" message), Always sends an alert each time the result of the measurement falls outside the preset range.

    Example: Creating an early time alert

    In case you want an early time alert, can create a normal (non master) alert with a very low minimum number of messages and a low check interval.

    Figure 33: Log > Sources > Message rate alerting — Create an early time alert

  7. If you have set more than one message rate alerts, you can set a master alert where applicable. To set an alert to be a master alert, select the Master alert checkbox next to it.

    When a master alert is triggered (and while it remains triggered), all other alerts for the given source/destination/space are suppressed. A master alert only blocks the other alerts that would be triggered at the given timeslot. A 24-hour alert does not block alerts that would be triggered at, for example 00:05.

    Suggestions for setting the master alert:

    • set the master alert to low check interval (5 minutes, if possible)

    • set the master alert to a lower check interval than the alerts it suppresses

    • set the master alert to have more lax limits than the alerts it suppresses

    The following examples demonstrate a few common use cases of a Master alert.

    Example: Using the master alert to indicate unexpected events

    The user has 2 relays (sender) and 10 hosts per each relay (=20 hosts). Each host sends approximately 5-10 messages in 5 minutes. Two message rate alerts are set, and one master alert to signal extreme unexpected events. Such event can be that either a host is undetectable and probably has stopped working, or that it sends too many logs, probably due to an error. The following configuration helps detecting these errors without having to receive hundreds of alerts unnecessarily.

    Figure 34: Log > Sources > Message rate alerting — Use a master alert to indicate unexpected events

  8. Optional step: Global alerts count the number of all messages received by syslog-ng on all sources, including internal messages.

    1. Navigate to Log > Options > Message rate alerting statistics. To add a global alert, click at Global alerts.

    2. Select the time period (between 5 minutes and 24 hours) during which the range is to be measured.

    3. Enter the range that is considered normal in the Minimum and Maximum fields.

    4. Select the alerting frequency in the Alert field. Once sends only one alert (and after the problem is fixed, a "Fixed" message), Always sends an alert each time the result of the measurement falls outside the preset range.

    5. To set the alert as a system-wide master alert, select Global master alert. It will suppress all other log rate alerts on SSB when it is triggered.

      NOTE:

      In the following cases, a so-called "always"-type super-master alert is triggered automatically.

      If all or some of the statistics from syslog-ng cannot be fetched, an alert is sent out and all other errors are suppressed until the error is fixed.

      If, for some reason, syslog-ng sends an unprocessable amount of statistics (for example because of some invalid input data), a similar super-master alert is triggered and stops processing the input.

  9. Optional step: Navigate to Log > Options > Message rate alerting statistics. Set the maximum number of alerts you want to receive in Limit of alerts sent out in a batch to prevent alert flooding. SSB will send alerts up to the predefined value and then one single alert stating that too many message alerts were generated and the excess amount have not been sent.

    Caution:

    Hazard of data loss The alerts over the predefined limit will be unreachable.

System related traps

Table 5: System related traps
Name SNMP alert ID Description
Login failed xcbLoginFailure Failed login attempts from SSB web interface.
Successful login xcbLogin Successful login attempts into SSB web interface.
Logout from the management interface xcbLogout Logouts from SSB web interface.
Configuration changed xcbConfigChange Any modification of SSB's configuration.
General alert xcbAlert

General alerts and error messages occurring on SSB.

Note, that alerts on general alerts and errors are sent whenever there is an alert or error level message in the SSB system log. These messages are very verbose and mainly useful only for debugging purposes.

Enabling these alerts may result in multiple e-mails or SNMP traps sent about the same event.

General error xcbError
Data and configuration backup failed xcbBackupFailed Alerts if the backup procedure is unsuccessful.
Data archiving failed xcbArchiveFailed Alerts if the archiving procedure is unsuccessful.
Database error occurred xcbDBError An error occurred in the database where SSB stores alerts and accounting information. Contact our support team (see About us for contact information).
License limit reached xcbLimitReached Maximum number of clients has been reached.
HA node state changed xcbHaNodeChanged A node of the SSB cluster changed its state, for example, a takeover occurred.
Timestamping error occured xcbTimestampError An error occurred during the timestamping process, for example the timestamping server did not respond.
Time sync lost xcbTimeSyncLost The system time became out of sync.
Raid status changed xcbRaidStatus The status of the node's RAID device changed its state.
Hardware error occured xcbHWError SSB detected a hardware error.
Firmware is tainted xcbFirmwareTainted A user has locally modified a file from the console.
Disk usage is above the defined ratio xcbDiskFull Disk space is used above the limit set in Disk space fill up prevention.

Alerts related to syslog-ng

Table 6: Alerts related to syslog-ng
Name SNMP alert ID Description
syslog-ng failure syslogngFailureTrap The syslog-ng application did not start properly, shut down unexpectedly, or encountered another problem. Depending on the error, SSB may not accept incoming messages or send them to the destinations.
Remote syslog-ng peer configuration changed peerConfigChangeTrap The configuration of the syslog-ng application running on a remote host that sents its logs to SSB has been changed. Note that such changes are detected only if the remote peer uses at least version 3.0 of syslog-ng or version 3.0 of the syslog-ng Agent, and if messages from the internal source are sent to SSB.
Logspace exceeded warning size spaceSizeLimit The size of a logspace has exceeded the size set as warning limit.
Message rate was outside the specified limits ssbAbsoluteMessageRateAlert The message rate has exceeded the minimum or maximum value.
Too many message rate alerts were generated ssbRateLimitTooManyAlerts SSB is generating too many message rate alerts, probably due to unusual traffic that may need investigation and further user actions.
Error during syslog-ng traffic statistics processing ssbStatisticsError There was an error during querying and processing statistics of incoming, forwarded, stored, and dropped messages.
Maximum number of connections has already been reached syslogngConcurrentConnectionsReached There was an attempt to establish a new connection but this would have meant exceeding the log source's maximum number of allowed connections (set in Log > Sources > Maximum connections). The new connection was refused by syslog-ng.
A destination path contains an invalid fragment syslogngInvalidPathError syslog-ng was unable to open a specific logspace destination, because its path contains a prohibited fragment (such as a reference to a parent directory).
Maximum number of dynamic clusters has been reached syslogngDynamicClustersMaximumReached SSB collects various statistics about log messages received, processed, and dropped for objects (every source, destination, and individual application or program). To avoid performance issues, the maximal number of objects that SSB collects statistics for is 100000. This alert means that SSB has reached this limit.
Related Documents