Chat now with support
Chat with Support

syslog-ng Store Box 6.1.0 - Administration Guide

Preface Introduction The concepts of SSB The Welcome Wizard and the first login Basic settings User management and access control Managing SSB Configuring message sources Storing messages on SSB Forwarding messages from SSB Log paths: routing and processing messages Configuring syslog-ng options Searching log messages Searching the internal messages of SSB Classifying messages with pattern databases The SSB RPC API Monitoring SSB Troubleshooting SSB Security checklist for configuring SSB

Accessing shared files

This section describes how to access log files that are shared using a share policy. For details on sharing log files, see Accessing log files across the network.

Every shared logspace is available as a separate shared folder, even if they all use a single share policy. The name of the shared folder is the name of the logspace.

Within the shared folder, the log files are organized into the following directory structure: YEAR/MM-DD/. The files are named according to the filename template set for the logspace. The extension of logstore files is .store, while the extension of text files is .log. Note that the root directory of the share may also contain various files related to the logspace, like index files for logstores. All files are read-only.

When using NFS for sharing the logspace, the name of the shared folder will be the following: /exports/{logspace_id}/....

Mount a shared logspace

The following examples show how to mount a shared logspace.

On Linux NFS:
mount -t nfs {ssb_ip}:/exports/{logspace_id} {where_to_mount}
On Linux SMB:

From SSB version 5.2.0, SSB only supports SMB 2.1 and later. If you are using a Linux version that uses SMB protocol version earlier than 2.1, add the option -o vers=2.1 to ensure that SSB uses SMB 2.1. For example:

mount -t cifs //{ssb_ip}/{logspace_id} /path/to/mount/shared/logspace/ -o username={username},password={password},vers=2.1
On Windows NFS:
  1. Make sure that you have the "Services for NFS" Windows component installed. If not, you can install the NFS client from the Windows interface.

  2. Open regedit, and navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ClientForNFS\CurrentVersion\Default

  3. Create two new DWORD keys called AnonymousGID and AnonymousUID. Set their values to 0.

  4. Restart the NFS client service from an elevated privilege command prompt. Use the following commands: nfsadmin client stop, then nfsadmin client start

  5. Mount the share from the command prompt. (Alternatively, you can also use the 'Map network drive...' function of the File Explorer.)

    mount {ssb_ip}://exports/{logspace_id} {drive_letter}:

    For example, the following command mounts the local logspace as drive G:

    mount 192.168.1.1://exports/local G:

    After mounting the shared logspace, it is visible in the file explorer. If it is not visible in the file explorer, you have probably used a different user to mount the share. To avoid this problem, you can mount the share again with the same user. Otherwise, you can access it from the command prompt using the {drive_letter}: command, even if it is not visible in the file explorer.

On Windows SMB:

Map the share from the command prompt. (Alternatively, you can also use the 'Map network drive...' function of the file explorer.)

net use {drive_letter}: \\{ssb_ip}\{logspace_name} /user:{user_name} "{password}"

For example, the following command maps the local logspace as drive G:

net use G: \\192.168.1.1:\local /user:myuser "mypassword"

After mapping the shared logspace, it is visible in the File Explorer. If it is not visible in the file explorer, you have probably used a different user to mount the share. To avoid this problem, you can mount the share again with the same user. Otherwise, you can access it from the command prompt using the {drive_letter}: command, even if it is not visible in the file explorer.

NOTE:

NOTE: In case of accessing shared files in domain mode, also include the domain name in the command: net use {drive_letter}: \\{ssb_ip}\{logspace_name} /user:{domain_name}\{user_name} "{password}"

For example, the following command maps the local logspace as drive G:

net use G: \\192.168.1.1:\local /user:mydomain\myuser "mypassword"

For information on viewing encrypted logspace files, see Viewing encrypted logs with logcat.

Forwarding messages from SSB

SSB can forward log messages to remote destinations. The remote destination can be an SQL database running on a remote server, a syslog or log analyzing application running on a remote server, or a Hadoop Distributed File System (HDFS) destination.

Forwarding log messages to SQL databases

This section describes how to forward log messages from SSB to a remote SQL database server.

Tested SQL destinations:

SSB 6.1 was tested with the following database servers:

  • MS SQL (with "select @@version")

    Microsoft SQL Server 2005 - 9.00.5057.00 (Intel X86)   Mar 25 2011 13:50:04   Copyright (c) 1988-2005 Microsoft Corporation  Standard Edition on Windows NT 5.2 (Build 3790: Service Pack 2)
  • PostgreSQL (with "select version()")

    PostgreSQL 8.3.15 on i486-pc-linux-gnu, compiled by GCC cc (GCC) 4.2.4 (Ubuntu 4.2.4-1ubuntu4)
  • MySQL (with "select version()")

    5.0.51a-3ubuntu5.8-log
  • Oracle (with "SELECT * FROM V$VERSION;")

    Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
    PL/SQL Release 11.2.0.4.0 - Production
    "CORE	11.2.0.4.0	Production"
    TNS for Linux: Version 11.2.0.4.0 - Production
    NLSRTL Version 11.2.0.4.0 - Production
    Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
    PL/SQL Release 12.1.0.2.0 - Production
    "CORE	12.1.0.2.0	Production"
    TNS for Linux: Version 12.1.0.2.0 - Production
    NLSRTL Version 12.1.0.2.0 - Production

To forward log messages from SSB to a remote SQL database server

  1. To create a new remote destination, navigate to Log > Destinations and select .

  2. Enter a name for the destination.

    NOTE:

    This name will be used in the name of the database tables created by SSB. For compatibility reasons, it can contain only numbers, lowercase characters, and the underscore (_) character, for example example_database_destination.

  3. Select Database Server.

    Figure 110: Log > Destinations — Creating database destinations

  4. Select the type of the remote database from the Database type field.

  5. Enter the IP address or hostname of the database server into the Address field. If the database is running on a non-standard port, adjust the Port setting.

  6. Enter the name and password of the database user account used to access the database into the Username and Password fields, respectively. This user needs to have the appropriate privileges for creating new tables.

    NOTE:

    SSB accepts passwords that are not longer than 150 characters. The following special characters can be used: !"#$%&'()*+,-./:;<=>?@[]^-`{|}

  7. Enter the name of the database that will store the log messages into the Database name field.

  8. Optional step: Enter the number of log message lines into the Flush lines field that SSB should wait before sending them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network. However, it also increases message latency.

    NOTE:

    Flush lines is in connection with the Output memory buffer value. (To set the Output memory buffer value, navigate to Log > Destinations). The value of Output memory buffer has to be greater than or equal to the value of Flush lines.

  9. SSB will automatically start a new table for every day or every month. Optionally, you can also create custom tables. Select the table naming template from the Table rotation field.

  10. Select which columns should SSB insert into the database. You can use one of the predefined templates, or select Custom columns to create a custom template. The available templates are described in SQL templates in SSB.

  11. SSB can automatically delete older messages and tables from the database. By default, messages are deleted after one month. Adjust the Retention time as needed for your environment.

  12. The logs stored in the database can be accessed using the search interface of SSB. Enter the name of the usergroup who can access the logs into the Access control > Group field. To add more groups (if needed), click .

  13. The timestamps of most log messages is accurate only to the second. SSB can include more accurate timestamps: set how many digits should be included in the Timestamp fractions of a second field. This option corresponds to the frac_digits() parameter of syslog-ng.

  14. If the server and SSB are located in a different timezone and you use the Legacy message template (which does not include timezone information), select the timezone of the server from the Timezone field.

  15. Set the size of the disk buffer (in Megabytes) in the Output disk buffer field. If the remote server becomes unavailable, SSB will buffer messages to the hard disk, and continue sending the messages when the remote server becomes available. This option corresponds to the log_disk_fifo_size() parameter of syslog-ng.

    Note that SSB does not pre-allocate the hard disk required for the disk buffer, so make sure that the required disk space is available on SSB. For details on creating archiving policies and adjusting the disk-fillup prevention, see Archiving and cleanup and Preventing disk space fill up.

    Example: Calculating disk buffer size

    The size of the disk buffer you need depends on the rate of the incoming messages, the size of the messages, and the length of the network outage that you want to cover. For example:

    • SSB is receiving 15000 messages per second

    • On the average, one message is 250 bytes long

    • You estimate that the longest time the destination will be unavailable is 4 hours

    In this case, you need a disk buffer for 250 [bytes] * 15000 [messages per second] * 4*60*60 [seconds] = 54000000000 [bytes], which is 54000 Megabytes (in other words, a bit over 50 GB).

  16. Click Commit.

  17. To start sending messages to the destination, include the new destination in a logpath. For details, see Log paths: routing and processing messages.

  18. To test if the database is accessible, select Test connection.

SQL templates in SSB

The following sections describe the SQL templates available in SSB:

Related Documents