syslog-ng Open Source Edition 3.18 - Administration Guide

mongodb: Storing messages in a MongoDB database

The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database. For the list of available optional parameters, see mongodb() destination options.

mongodb(parameters);

The mongodb() driver does not support creating indexes, as that can be a very complex operation in MongoDB. If needed, the administrator of the MongoDB database must ensure that indexes are created on the collections.

The mongodb() driver does not add the _id field to the message: the MongoDB server will do that automatically, if none is present. If you want to override this field from syslog-ng OSE, use the key() parameter of the value-pairs() option.

The syslog-ng OSE mongodb() driver is compatible with MongoDB server version 1.4 and newer.

destination d_mongodb {
    mongodb();
};

destination d_mongodb {
    mongodb(
        uri("mongodb://localhost:27017/syslog")
        collection("messages")
        value-pairs(
            scope("selected-macros" "nv-pairs" "sdata")
        )
    );
};

destination d_mongodb {
    mongodb(
        servers("localhost:27017")
        database("syslog")
        collection("messages")
        value-pairs(
            scope("selected-macros" "nv-pairs" "sdata")
        )
    );
};

How syslog-ng OSE connects the MongoDB server

When syslog-ng OSE connects the MongoDB server during startup, it completes the following steps.

1. The syslog-ng OSE application connects the first address listed in the servers() option.

2. • If the server is accessible and it is a master MongoDB server, syslog-ng OSE authenticates on the server (if needed), then starts sending the log messages to the server.

   • If the server is not accessible, or it is not a master server in a MongoDB replicaset and it does not send the address of the master server, syslog-ng OSE connects the next address listed in the servers() option.

   • If the server is not a master server in a MongoDB replicaset, but it sends the address of the master server, syslog-ng OSE connects the received address.

3. When syslog-ng OSE connects the master MongoDB server, it retrieves the list of replicas (from the replSet option of the server), and appends this list to the servers() option.

   Caution: This means that syslog-ng OSE can send log messages to addresses that are not listed in its configuration. Make sure to include the address of your master server in your syslog-ng OSE configuration file, otherwise you risk losing log messages if all the addresses listed in the syslog-ng OSE configuration are offline. Addresses retrieved from the MongoDB servers are not stored, and can be lost when syslog-ng OSE is restarted. The retrieved addresses are not lost if the server() option of the destination was not changed in the configuration file since the last restart. The failover mechanism used in the mongodb() driver is different from the client-side failover used in other drivers.

4. The syslog-ng OSE application attempts to connect another server if the servers() list contains at least two addresses, and one of the following events happens:

   • The safe-mode() option is set to no, and the MongoDB server becomes unreachable.

   • The safe-mode() option is set to yes, and syslog-ng OSE cannot insert a log message into the database because of an error.

   In such case, syslog-ng OSE starts to connect the addresses in from the servers() list (starting from the first address) to find the new master server, authenticates on the new server (if needed), then continues to send the log messages to the new master server.

   During this failover step, one message can be lost if the safe-mode() option is disabled.

5. If the original master becomes accessible again, syslog-ng OSE will automatically connect to the original master.

mongodb() destination options

The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database.

The mongodb() destination has the following options:

disk-buffer()

Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:

Options reliable() and disk-buf-size() are required options.

destination d_demo {
    network(
        "127.0.0.1"
        port(3333)
        disk-buffer(
            mem-buf-size(10000)
            disk-buf-size(2000000)
            reliable(yes)
            dir("/tmp/disk-buffer")
        )
    );
};

destination d_demo {
    network(
        "127.0.0.1"
        port(3333)
           disk-buffer(
            mem-buf-length(10000)
            disk-buf-size(2000000)
            reliable(no)
            dir("/tmp/disk-buffer")
        )
    );
};

frac-digits()

Description: The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using the hook-commands() when syslog-ng OSE starts or stops

To execute an external program when syslog-ng OSE starts or stops, use the following options:

Using the hook-commands() when syslog-ng OSE reloads

To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:

source {
   network(transport(udp)
	hook-commands(
          startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
          shutdown("iptables -D LOGCHAIN 1")
        )
     );
};

local-time-zone()

Description: Sets the timezone used when expanding filename and tablename templates.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

log-fifo-size()

Description: The number of messages that the output queue can store.

on-error()

Description: Controls what happens when type-casting fails and syslog-ng OSE cannot convert some data to the specified type. By default, syslog-ng OSE drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().

• drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of syslog-ng OSE.

• drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.

• fallback-to-string: Convert the property to string and log an error message to the internal() source.

• silently-drop-message: Drop the entire message silently, without logging the error.

• silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.

• silently-fallback-to-string: Convert the property to string silently, without logging the error.

throttle()

Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

value-pairs()

scope("selected-macros" "nv-pairs")

Description: The value-pairs() option creates structured name-value pairs from the data and metadata of the log message. For details on using value-pairs(), see Structuring macros, metadata, and other value-pairs.

network: Sending messages to a remote log server using the RFC3164 protocol (network() driver)

The network() destination driver can send syslog messages conforming to RFC3164 from the network using the TCP, TLS, and UDP networking protocols.

• UDP is a simple datagram oriented protocol, which provides "best effort service" to transfer messages between hosts. It may lose messages, and no attempt is made to retransmit lost messages. The BSD-syslog protocol traditionally uses UDP.

  Use UDP only if you have no other choice.

• TCP provides connection-oriented service: the client and the server establish a connection, each message is acknowledged, and lost packets are resent. TCP can detect lost connections, and messages are lost, only if the TCP connection breaks. When a TCP connection is broken, messages that the client has sent but were not yet received on the server are lost.

• The syslog-ng application supports TLS (Transport Layer Security, also known as SSL) over TCP. For details, see Encrypting log messages with TLS.

Declaration:

network("<destination-address>" [options]);

The network() destination has a single required parameter that specifies the destination host address where messages should be sent. If name resolution is configured, you can use the hostname of the target server. By default, syslog-ng OSE sends messages using the TCP protocol to port 601.

destination d_tcp { network("10.1.2.3" port(1999)); };

destination d_tcp { network("target_host" port(1999)); };

destination d_tcp6 {
    network(
        "::1"
        port(2222)
        transport(tcp)
        ip-protocol(6)
        );
};

destination d_tcp { network("10.1.2.3" port(1999) flags(syslog-protocol) ); };