The stomp() driver sends messages to servers (message brokers) using the Simple (or Streaming) Text Oriented Message Protocol (STOMP), formerly known as TTMP. syslog-ng OSE supports version 1.0 of the STOMP protocol. The syslog-ng OSE stomp() driver supports persistence.
The name-value pairs selected with the value-pairs() option will be sent as STOMP headers, while the body of the STOMP message is empty by default (but you can add custom content using the body() option). Publishing the name-value pairs as headers makes it possible to use the Headers exchange-type and subscribe only to interesting log streams.
For the list of available parameters, see stomp() destination options.
Declaration:
stomp( host("<stomp-server-address>") );
Example: Using the stomp() driver
The following example shows the default values of the available options.
destination d_stomp {
stomp(
host("localhost")
port(61613)
destination("/topic/syslog")
body("") # optional, empty by default
persistent(yes)
ack(no)
username("user") # optional, empty by default
password("password") # optional, empty by default
value-pairs(scope(selected-macros, nv-pairs, sdata))
);
};
The stomp() driver publishes messages using the Simple (or Streaming) Text Oriented Message Protocol (STOMP).
The stomp() destination has the following options:
ack()
Description: Request the STOMP server to acknowledge the receipt of the messages. If you enable this option, then after sending a message, syslog-ng OSE waits until the server confirms that it has received the message. This delay can seriously limit the performance of syslog-ng OSE if the message rate is high, and the server cannot acknowledge the messages fast enough.
body()
| Type: |
string |
| Default: |
empty string |
Description: The body of the STOMP message. You can also use macros and templates.
destination()
| Type: |
string |
| Default: |
/topic/syslog |
Description: The name of the destination (message queue) on the STOMP server. It can include macros and templates.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
| reliable() |
| Type: |
yes|no |
| Default: |
no |
|
Description: If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
|
|
Caution:
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost. | |
| compaction() |
| Type: |
yes|no |
| Default: |
no |
|
Description: If set to yes, syslog-ng OSE prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times. |
NOTE: Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when syslog-ng is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop "unset" values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
| dir() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines the folder where the disk-buffer files are stored.
|
|
Caution:
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
syslog-ng OSE creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, syslog-ng OSE will look for or create disk-buffer files in their old location. To ensure that syslog-ng OSE uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to. |
NOTE: If the dir() path provided by the user does not exist, syslog-ng OSE creates the path with the same permission as the running instance. |
| disk-buf-size() |
| Type: |
number (bytes) |
| Default: |
|
| Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option. |
| mem-buf-length() |
| Type: |
number (messages) |
| Default: |
10000 |
| Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes. |
| mem-buf-size() |
| Type: |
number (bytes) |
| Default: |
163840000 |
| Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no. |
| qout-size() |
| Type: |
number (messages) |
| Default: |
64 |
| Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty. |
Options reliable() and disk-buf-size() are required options.
Example: Examples for using disk-buffer()
In the following case reliable disk-buffer() is used.
destination d_demo {
network(
"127.0.0.1"
port(3333)
disk-buffer(
mem-buf-size(10000)
disk-buf-size(2000000)
reliable(yes)
dir("/tmp/disk-buffer")
)
);
};
In the following case normal disk-buffer() is used.
destination d_demo {
network(
"127.0.0.1"
port(3333)
disk-buffer(
mem-buf-length(10000)
disk-buf-size(2000000)
reliable(no)
dir("/tmp/disk-buffer")
)
);
};
|
truncate-size-ratio() |
| Type: |
number (between 0 and 1) |
| Default: |
0.1 (10%) |
|
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space, so syslog-ng only truncates the file, if the possible disk gain is more than truncate-size-ratio() times disk-buf-size().
|
|
Caution:
One Identity does not recommend you to change truncate-size-ratio(). Only change its value if you know the performance implications of doing so. | |
batch-bytes()
| Accepted values: |
number [bytes] |
| Default: |
none |
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, syslog-ng OSE sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, syslog-ng OSE flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in syslog-ng OSE version 3.19 and later.
batch-lines()
Description: Specifies how many lines are flushed to a destination in one batch. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, syslog-ng OSE waits for 100 messages.
If the batch-timeout() option is disabled, the syslog-ng OSE application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload syslog-ng OSE or in case of network sources, the connection with the client is closed, syslog-ng OSE automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, syslog-ng OSE flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the syslog-ng OSE source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
| Type: |
time in milliseconds |
| Default: |
-1 (disabled) |
Description: Specifies the time syslog-ng OSE waits for lines to accumulate in the output buffer. The syslog-ng OSE application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, syslog-ng OSE sends messages to the destination at most once every batch-timeout() milliseconds.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable syslog-ng OSE to execute external applications.
Using the hook-commands() when syslog-ng OSE starts or stops
To execute an external program when syslog-ng OSE starts or stops, use the following options:
| startup() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines the external program that is executed as syslog-ng OSE starts. |
| shutdown() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines the external program that is executed as syslog-ng OSE stops. |
Using the hook-commands() when syslog-ng OSE reloads
To execute an external program when the syslog-ng OSE configuration is initiated or torn down, for example, on startup/shutdown or during a syslog-ng OSE reload, use the following options:
| setup() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload. |
| teardown() |
| Type: |
string |
| Default: |
N/A |
|
Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload. |
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as syslog-ng OSE is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed.
source {
network(transport(udp)
hook-commands(
startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
shutdown("iptables -D LOGCHAIN 1")
)
);
};
host()
| Type: |
hostname or IP address |
| Default: |
127.0.0.1 |
Description: The hostname or IP address of the STOMP server.
password()
| Type: |
string |
| Default: |
n/a |
Description: The password used to authenticate on the STOMP server.
persistent()
| Type: |
yes|no |
| Default: |
yes |
Description: If this option is enabled, the STOMP server or broker will store the messages on its hard disk. That way, the messages will be retained if the STOMP server is restarted, if the message queue is set to be durable on the STOMP server.
port()
| Type: |
number |
| Default: |
61613 |
Description: The port number of the STOMP server.
retries()
| Type: |
number (of attempts) |
| Default: |
3 |
Description: If syslog-ng OSE cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), syslog-ng OSE will wait for time-reopen() time, then tries sending the message again.
throttle()
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
| Accepted values: |
number [seconds] |
| Default: |
60 |
Description: The time to wait in seconds before a dead connection is reestablished.
username()
| Type: |
string |
| Default: |
empty string |
Description: The username used to authenticate on the STOMP server.
value-pairs()
| Type: |
parameter list of the value-pairs() option |
| Default: |
scope("selected-macros" "nv-pairs") |
Description: The value-pairs() option creates structured name-value pairs from the data and metadata of the log message. For details on using value-pairs(), see Structuring macros, metadata, and other value-pairs.
NOTE: Empty keys are not logged.
From version 3.27.1, the syslog-ng Open Source Edition (syslog-ng OSE) application can send log messages to Sumo Logic, a cloud-based log management and security analytics service, by using the sumologic-http() and sumologic-syslog() destinations.
Prerequisites
Currently, using the sumologic-http() and sumologic-syslog() destinations with syslog-ng OSE has the following prerequisites:
-
A Sumo Logic account.
If you do not yet have a Sumo Logic account, visit the official Sumo Logic website, and click Start free trial to create an account.
NOTE: A free trial version of the Sumo Logic account has limited functionalities and is only available for 90 days.
-
A Cloud Syslog Source configured with your Sumo Logic account.
For details, follow the configuration instructions under the Configure a Cloud Syslog Source section on the official Sumo Logic website.
NOTE: Transport-level security (TLS) 1.2 over TCP is required.
-
A Cloud Syslog Source Token (from the Cloud Syslog Source side).
-
TLS set up on your Sumo Logic account.
For detailed information about setting up TLS in your Sumo Logic account, see the description for setting up TLS on the Sumo Logic official website.
NOTE: After you download the DigiCert certificate, make sure you follow the certificate setup steps under the syslog-ng section.
-
Your Sumo Logic syslog client, configured to send data to the Sumo Logic cloud syslog service, by using syslog-ng OSE.
For detailed information, follow the instructions under the Send data to cloud syslog source with syslog-ng section on the official Sumo Logic website.
-
A verified connection and client configuration with the Sumo Logic service.
|
|
Caution:
To avoid potential data loss, One Identity strongly recommends that you verify your connection and client configuration with the Sumo Logic service before you start using the sumologic-http() or sumologic-syslog() destination with syslog-ng OSE in a production environment. |
-
(Optional) For using the sumologic-http() destination, you need a HTTP Hosted Collector configured in the Sumo Logic service.
To configure a Hosted Collector, follow the configuration instructions under the Configure a Hosted Collector section on the official Sumo Logic website.
-
(Optional) For using the sumologic-http() destination, you need the unique HTTP collector code you receive while configuring your Host Collector for HTTP requests.
Limitations
Currently, using the sumologic-syslog() and sumologic-http() destinations with syslog-ng OSE has the following limitations:
-
The minimum required version of syslog-ng OSE is version 3.27.1.
-
Message format must be in RFC 5424-compliant form. Messages over 64KB in length are truncated.
For more information about the message format limitations, see the Message format section on the official Sumo Logic website.
-
64 characters long Sumo Logic tokens must be passed in the message body.
NOTE: Although RFC 5424 limits the structured data field (SD-ID) to 32 characters, Sumo Logic tokens are 64 characters long. If your logging client enforces the 32 characters length limit, you must pass the token in the message body.
Declaration
Declaration for the sumologic-http() destination
destination d_sumo_http {
sumologic-http(
collector("ZaVnC4dhaV3_[...]UF2D8DRSnHiGKoq9Onvz-XT7RJG2FA6RuyE5z4A==")
deployment("eu")
tls(peer-verify(yes) ca-dir('/etc/syslog-ng/ca.d'))
);
};
Declaration for the sumologic-syslog() destination
destination d_sumo_syslog {
sumologic-syslog(
token("rqf/bdxYVaBLFMoU39[...]CCC5jwETm@41123")
deployment("eu")
tls(peer-verify(yes) ca-dir('/etc/syslog-ng/ca.d'))
);
};
To use the sumologic() driver, the scl.conf file must be included in your syslog-ng OSE configuration:
@include "scl.conf"
NOTE: The sumologic() driver is actually a reusable configuration snippet configured to send log messages using the network() and http() destination by using a template. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
The sumologic-http() and sumologic-syslog() destinations send log messages to Sumo Logic, a cloud-based log management and security analytics service.
Using the sumologic-http() destination, you can send data to the Sumo Logic service by utilizing a Hosted Collector hosted by Sumo Logic.
For more information about the sumologic-http() destination, see sumologic-syslog() .
Sending data using the sumologic-http() destination
Example: Using the sumologic-http() destination
The following example sends every log from the system() source to your Sumo Logic account.
log {
source { system(); };
destination {
sumologic-http(
collector("UNIQUE-HTTP-COLLECTOR-CODE-AS-PROVIDED-BY-sumologic")
deployment("ENDPOINT")
tls(peer-verify(yes) ca-dir('/etc/syslog-ng/ca.d'))
);
};
};
