The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database. For the list of available optional parameters, see mongodb() destination options.
mongodb(parameters);
The mongodb() driver does not support creating indexes, as that can be a very complex operation in MongoDB. If needed, the administrator of the MongoDB database must ensure that indexes are created on the collections.
The mongodb() driver does not add the _id field to the message: the MongoDB server will do that automatically, if none is present. If you want to override this field from syslog-ng OSE, use the key() parameter of the value-pairs() option.
The syslog-ng OSE mongodb() driver is compatible with MongoDB server version 1.4 and newer.
|
NOTE:
By default, syslog-ng OSE handles every message field as a string. For details on how to send selected fields as other types of data (for example, handle the PID as a number), see Specifying data types in value-pairs. |
The following example creates a mongodb() destination using only default values.
destination d_mongodb { mongodb(); };
The following example displays the default values.
destination d_mongodb { mongodb( uri("mongodb://localhost:27017/syslog") collection("messages") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
The following example shows the same setup using the deprecated libmongo-client syntax (as used in syslog-ng OSE version
destination d_mongodb { mongodb( servers("localhost:27017") database("syslog") collection("messages") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
When syslog-ng OSE connects the MongoDB server during startup, it completes the following steps.
The syslog-ng OSE application connects the first address listed in the servers() option.
If the server is accessible and it is a master MongoDB server, syslog-ng OSE authenticates on the server (if needed), then starts sending the log messages to the server.
If the server is not accessible, or it is not a master server in a MongoDB replicaset and it does not send the address of the master server, syslog-ng OSE connects the next address listed in the servers() option.
If the server is not a master server in a MongoDB replicaset, but it sends the address of the master server, syslog-ng OSE connects the received address.
When syslog-ng OSE connects the master MongoDB server, it retrieves the list of replicas (from the replSet option of the server), and appends this list to the servers() option.
|
Caution:
|
The syslog-ng OSE application attempts to connect another server if the servers() list contains at least two addresses, and one of the following events happens:
The safe-mode() option is set to no, and the MongoDB server becomes unreachable.
The safe-mode() option is set to yes, and syslog-ng OSE cannot insert a log message into the database because of an error.
In such case, syslog-ng OSE starts to connect the addresses in from the servers() list (starting from the first address) to find the new master server, authenticates on the new server (if needed), then continues to send the log messages to the new master server.
During this failover step, one message can be lost if the safe-mode() option is disabled.
If the original master becomes accessible again, syslog-ng OSE will automatically connect to the original master.
The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database.
The mongodb() destination has the following options:
Type: | string |
Default: | messages |
Description: The name of the MongoDB collection where the log messages are stored (collections are similar to SQL tables). Note that the name of the collection must not start with a dollar sign ($), and that it may contain dot (.) characters.
|
Caution:
Hazard of data loss! The syslog-ng OSE application does not verify that the specified collection name does not contain invalid characters. If you specify a collection with an invalid name, the log messages sent to the MongoDB database will be irrevocably lost without any warning. |
Type: | string |
Default: | syslog |
This option is deprecated and will be removed from syslog-ng OSE. Use the uri() option instead.
Description: The name of the MongoDB database where the log messages are stored. Note that the name of the database must not start with a dollar sign ($) and it cannot contain dot (.) characters.
|
Caution:
Hazard of data loss! The syslog-ng OSE application does not verify that the specified database name does not contain invalid characters. If you specify a database with an invalid name, the log messages sent to the MongoDB database will be irrevocably lost without any warning. |
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
reliable() | |||
Type: | yes|no | ||
Default: | no | ||
Description: If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
|
disk-buf-size() | |
Type: | number (bytes) |
Default: | |
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option. |
mem-buf-length() | |
Type: | number (messages) |
Default: | 10000 |
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes. |
mem-buf-size() | |
Type: | number (bytes) |
Default: | 163840000 |
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no. |
qout-size() | |
Type: | number (messages) |
Default: | 64 |
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty. |
Options reliable() and disk-buf-size() are required options.
In the following case reliable disk-buffer() is used.
destination d_demo { network( "127.0.0.1" port(3333) disk-buffer( mem-buf-size(10000) disk-buf-size(2000000) reliable(yes) dir("/tmp/disk-buffer") ) ); };
In the following case normal disk-buffer() is used.
destination d_demo { network( "127.0.0.1" port(3333) disk-buffer( mem-buf-length(10000) disk-buf-size(2000000) reliable(no) dir("/tmp/disk-buffer") ) ); };
Type: | number |
Default: | 0 |
Description: The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
|
NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable syslog-ng OSE to execute external applications. |
To execute an external program when syslog-ng OSE starts or stops, use the following options:
startup() | |
Type: | string |
Default: | N/A |
Description: Defines the external program that is executed as syslog-ng OSE starts. |
shutdown() | |
Type: | string |
Default: | N/A |
Description: Defines the external program that is executed as syslog-ng OSE stops. |
To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:
setup() | |
Type: | string |
Default: | N/A |
Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload. |
teardown() | |
Type: | string |
Default: | N/A |
Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload. |
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as syslog-ng OSE is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed.
source { network(transport(udp) hook-commands( startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT") shutdown("iptables -D LOGCHAIN 1") ) ); };
Type: | name of the timezone, or the timezone offset |
Default: | The local timezone. |
Description: Sets the timezone used when expanding filename and tablename templates.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
Type: | number |
Default: | Use global setting. |
Description: The number of messages that the output queue can store.
Accepted values: |
drop-message|drop-property|fallback-to-string| silently-drop-message|silently-drop-property|silently-fallback-to-string |
Default: | Use the global setting (which defaults to drop-message) |
Description: Controls what happens when type-casting fails and syslog-ng OSE cannot convert some data to the specified type. By default, syslog-ng OSE drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().
drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of syslog-ng OSE.
drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
fallback-to-string: Convert the property to string and log an error message to the internal() source.
silently-drop-message: Drop the entire message silently, without logging the error.
silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
silently-fallback-to-string: Convert the property to string silently, without logging the error.
Type: | string |
Default: | n/a |
This option is deprecated and will be removed from syslog-ng OSE. Use the uri() option instead.
Description: Password of the database user.
Type: | string |
Default: | empty |
This option is deprecated and will be removed from syslog-ng OSE. Use the uri() option instead.
Description: If the path() option is set, syslog-ng OSE will connect to the database using the specified UNIX domain socket. Note that you cannot set the path() and the servers() options at the same time.
Type: | number (of attempts) |
Default: | 3 |
Description: The number of times syslog-ng OSE attempts to send a message to this destination. If syslog-ng OSE could not send a message, it will try again until the number of attempts reaches retries, then drops the message.
For MongoDB operations, syslog-ng OSE uses a one-minute timeout: if an operation times out, syslog-ng OSE assumes the operation has failed.
Type: | yes or no |
Default: | yes |
This option is deprecated and will be removed from syslog-ng OSE. Use the uri() option instead.
Description: If safe-mode() is enabled, syslog-ng OSE performs an extra check after each insert to verify that the insert succeeded. The insert is successful only if this second check is successful. Note that enabling this option reduces the performance of the driver.
Type: | list of hostname:port pairs |
Default: | 127.0.0.1:27017 |
This option is deprecated and will be removed from syslog-ng OSE. Use the uri() option instead.
Description: Specifies the hostname or IP address and the port number of the database server. When specifying an IP address, IPv4 (for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well.
To send the messages to a MongoDB replicaset, specify the addresses of the database servers as a comma-separated list, for example: servers(192.168.1.1:27017,192.168.3.3:27017)
For details on how syslog-ng OSE connects the MongoDB server, see How syslog-ng OSE connects the MongoDB server.
To connect to the server using a UNIX domain socket, use path option. Note that you cannot set the path() and the servers() options at the same time.
Type: | number |
Default: | 0 |
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
Type: | string |
Default: | mongodb://127.0.0.1:27017/syslog?wtimeoutMS=60000&socketTimeoutMS=60000&connectTimeoutMS=60000 |
Description: Available in syslog-ng OSE 3.8 and later. Please refer to the MongoDB URI format documentation for detailed syntax.
Type: | string |
Default: | n/a |
This option is deprecated and will be removed from syslog-ng OSE. Use the uri() option instead.
Description: Name of the database user. Note that the mongodb() driver currently does not support TLS-encrypted authentication.
Type: | parameter list of the value-pairs() option |
Default: | scope("selected-macros" "nv-pairs") |
Description: The value-pairs() option creates structured name-value pairs from the data and metadata of the log message. For details on using value-pairs(), see Structuring macros, metadata, and other value-pairs.
|
NOTE:
Empty keys are not logged. |
|
NOTE:
By default, syslog-ng OSE handles every message field as a string. For details on how to send selected fields as other types of data (for example, handle the PID as a number), see Specifying data types in value-pairs. |
The network() destination driver can send syslog messages conforming to RFC3164 from the network using the TCP, TLS, and UDP networking protocols.
UDP is a simple datagram oriented protocol, which provides "best effort service" to transfer messages between hosts. It may lose messages, and no attempt is made to retransmit lost messages. The BSD-syslog protocol traditionally uses UDP.
Use UDP only if you have no other choice.
TCP provides connection-oriented service: the client and the server establish a connection, each message is acknowledged, and lost packets are resent. TCP can detect lost connections, and messages are lost, only if the TCP connection breaks. When a TCP connection is broken, messages that the client has sent but were not yet received on the server are lost.
The syslog-ng application supports TLS (Transport Layer Security, also known as SSL) over TCP. For details, see Encrypting log messages with TLS.
network("<destination-address>" [options]);
The network() destination has a single required parameter that specifies the destination host address where messages should be sent. If name resolution is configured, you can use the hostname of the target server. By default, syslog-ng OSE sends messages using the TCP protocol to port 601.
TCP destination that sends messages to 10.1.2.3, port 1999:
destination d_tcp { network("10.1.2.3" port(1999)); };
If name resolution is configured, you can use the hostname of the target server as well.
destination d_tcp { network("target_host" port(1999)); };
TCP destination that sends messages to the ::1 IPv6 address, port 2222.
destination d_tcp6 { network( "::1" port(2222) transport(tcp) ip-protocol(6) ); };
To send messages using the IETF-syslog message format without using the IETF-syslog protocol, enable the syslog-protocol flag. (For details on how to use the IETF-syslog protocol, see syslog() destination options.)
destination d_tcp { network("10.1.2.3" port(1999) flags(syslog-protocol) ); };
© ALL RIGHTS RESERVED. Terms of Use Privacy Cookie Preference Center