syslog-ng Premium Edition 6.0.21 - Administration Guide

Description: Include the path to the directory where you copied the required libraries (see Procedure 7.2, “Prerequisites”), for example, client_lib_dir(/user/share/elasticsearch-2.2.0/lib).

Description: Specifies the client mode used to connect to the Elasticsearch server, for example, client_mode("node").

Description: Specifies the name or the Elasticsearch cluster, for example, cluster("my-elasticsearch-cluster"). Optionally, you can specify the name of the cluster in the Elasticsearch resource file. For details, see the section called “resource()”.

Description: The number of concurrent (simultaneous) requests that syslog-ng PE sends to the Elasticsearch server. Set this option to 1 or higher to increase performance. When using the concurrent_requests() option, make sure that the flush-limit() option is higher than one, otherwise it will not have any noticeable effect. For details, see the section called “flush_limit()”.

Description: Use this option to specify a custom ID for the records inserted into Elasticsearch. If this option is not set, the Elasticsearch server automatically generates and ID for the message. For example: custom_id(${UNIQID}) (Note that to use the ${UNIQID} macro, the use-uniqid() global option must be enabled. For details, see the section called “use-uniqid()”.)

Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:

Options reliable() and disk-buf-size() are required options.

destination d_demo {
    network(
            "127.0.0.1"
            port(3333)
            disk-buffer(
                mem-buf-size(10000)
                disk-buf-size(2000000)
                reliable(yes)
                dir("/tmp/disk-buffer")
            )
        );
};

destination d_demo {
    network(
            "127.0.0.1"
            port(3333)
            disk-buffer(
                mem-buf-length(10000)
                disk-buf-size(2000000)
                reliable(no)
                dir("/tmp/disk-buffer")
            )
        );
};

Description: The number of messages that syslog-ng PE sends to the Elasticsearch server in a single batch.

Description: The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

Description: Name of the Elasticsearch index to store the log messages. You can use macros and templates as well. For example, index("syslog-ng_${YEAR}.${MONTH}.${DAY}").

Description: The number of messages that the output queue can store.

Description: Controls what happens when type-casting fails and syslog-ng PE cannot convert some data to the specified type. By default, syslog-ng PE drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().

Description: The port number of the Elasticsearch server. This option is used only in transport mode: client_mode("transport")

Description: The number of times syslog-ng PE attempts to send a message to this destination. If syslog-ng PE could not send a message, it will try again until the number of attempts reaches retries, then drops the message.

Description: The list of Elasticsearch resources to load, separated by semicolons. For example, resource("/home/user/elasticsearch/elasticsearch.yml;/home/user/elasticsearch/elasticsearch2.yml").

Description: Specifies the hostname or IP address of the Elasticsearch server. When specifying an IP address, IPv4 (for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well. When specifying multiple addresses, use space to separate the addresses, for example, server("127.0.0.1 remote-server-hostname1 remote-server-hostname2")

This option is used only in transport mode: client_mode("transport")

Description: The message as sent to the Elasticsearch server. Typically, you will want to use the command-line notation of the format-json template function.

To add a @timestamp field to the message, for example, to use with Kibana, include the @timestamp=${ISODATE} expression in the template. For example: template($(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE}))

For details on formatting messages in JSON format, see the section called “format-json”.

Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see the section called “Date-related macros”.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see the section called “A note on timezones and timestamps”.

Description: The type of the index. For example, type("test").

Description: Enable creating non-existing directories.

Description: The group of the directories created by syslog-ng. To preserve the original properties of an existing directory, use the option without specifying an attribute: dir-group().

Description: The owner of the directories created by syslog-ng. To preserve the original properties of an existing directory, use the option without specifying an attribute: dir-owner().

Description: The permission mask of directories created by syslog-ng. Log directories are only created if a file after macro expansion refers to a non-existing directory, and directory creation is enabled (see also the create-dirs() option). For octal numbers prefix the number with 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing directory, use the option without specifying an attribute: dir-perm(). Note that when creating a new directory without specifying attributes for dir-perm(), the default permission of the directories is masked with the umask of the parent process (typically 0022).

Description: Flags influence the behavior of the destination driver.

Description: Specifies how many lines are sent to a destination at a time. The syslog-ng PE application waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the destination, but also increases message latency.

For optimal performance when sending messages to an syslog-ng PE server, make sure that the flush-lines() is smaller than the window size set using the log-iw-size() option in the source of your server.

Description: This is an obsolete option. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. For details, see the flush-lines() option.

Description: The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

Description: Forces an fsync() call on the destination fd after each write. Note: enabling this option may seriously degrade performance.

Description: Set the group of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: group().

Description: Sets the timezone used when expanding filename and tablename templates.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

Description: The number of messages that the output queue can store.

Description: An alias for the obsolete mark() option, retained for compatibility with syslog-ng version 1.6.x. The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() is periodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq() will be inherited from the global options. If the destination uses internal mark-mode(), then the global mark-freq() will be valid (does not matter what mark-freq() set in the destination side).

Description: The mark-mode() option can be set for the following destination drivers: file(), program(), unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.

Available in syslog-ng PE 4 LTS and later.

Description: If set to a value higher than 0, syslog-ng PE checks when the file was last modified before starting to write into the file. If the file is older than the specified amount of time (in seconds), then syslog-ng removes the existing file and opens a new file with the same name. In combination with for example the ${WEEKDAY} macro, this can be used for simple log rotation, in case not all history has to be kept. (Note that in this weekly log rotation example if its Monday 00:01, then the file from last Monday is not seven days old, because it was probably last modified shortly before 23:59 last Monday, so it is actually not even six days old. So in this case, set the overwrite-if-older() parameter to a-bit-less-than-six-days, for example, to 518000 seconds.

Description: Set the owner of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: owner().

Description: If set, syslog-ng PE will pad output messages to the specified size (in bytes). Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes).

Description: The permission mask of the file if it is created by syslog-ng. For octal numbers prefix the number with 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing file, use the option without specifying an attribute: perm().

Description: If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.

Description: Specifies a template defining the logformat to be used in the destination. Macros are described in the section called “Macros of syslog-ng PE”. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.

Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see the section called “Date-related macros”.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see the section called “A note on timezones and timestamps”.

Description: Set the cipher method used to encrypt the logstore. The following cipher methods are available: aes-128-cbc, aes-128-cfb, aes-128-cfb1, aes-128-cfb8, aes-128-ecb, aes-128-ofb , aes-192-cbc, aes-192-cfb, aes-192-cfb1, aes-192-cfb8, aes-192-ecb, aes-192-ofb , aes-256-cbc, aes-256-cfb, aes-256-cfb1, aes-256-cfb8, aes-256-ecb, aes-256-ofb , aes128 , aes192 , aes256, bf , bf-cbc , bf-cfb, bf-ecb , bf-ofb , blowfish, cast , cast-cbc , cast5-cbc , cast5-cfb, cast5-ecb, cast5-ofb , des, des-cbc, des-cfb , des-cfb1 , des-cfb8 , des-ecb , des-ede, des-ede-cbc, des-ede-cfb , des-ede-ofb, des-ede3 , des-ede3-cbc, des-ede3-cfb, des-ede3-ofb, des-ofb , des3 , desx , desx-cbc, rc2, rc2-40-cbc , rc2-64-cbc, rc2-cbc, rc2-cfb, rc2-ecb , rc2-ofb, rc4, and rc4-40. By default, syslog-ng PE uses the aes-128-cbc method.

Note that the size of the digest hash must be equal to or larger than the key size of the cipher method. For example, to use the aes-256-cbc cipher method, the digest method must be at least SHA-256. This option is available in syslog-ng PE 3.1 and later.

Description: This option is obsolete. Use the journal-block-size() option instead.

Size of a logstore chunk in kilobytes. Note that this size refers to the compressed size of the chunk. Also, the gzip library used for compressing the messages has a 32k long buffer, so messages may not appear in the actual logfile until this buffer is not filled. Logstore chunks are closed when they reach the specified size, or when the time limit set in chunk-time() expires.

Description: This option is obsolete.

Time limit in seconds: syslog-ng PE closes the chunk if no new messages arrive until the time limit expires. Logstore chunks are closed when the time limit expires, or when they reach the size specified in the chunk-size() parameter. If the time limit set in the time-reap() parameter expires, the entire file is closed.

Description: Compression level. 0 means uncompressed files, while 1-9 is the compression level used by gzip (9 means the highest but slowest compression, 3 is usually a good compromise).

Description: Enable creating non-existing directories.

Description: Set the digest method to use. The following digest methods are available: MD4, MD5, SHA0 (SHA), SHA1, RIPEMD160, SHA224, SHA256, SHA384, and SHA512. By default, syslog-ng PE uses the SHA1 method.

Note that the size of the digest hash must be equal to or larger than the key size of the cipher method. For example, to use the aes-256-cbc cipher method, the digest method must be at least SHA256. This option is available in syslog-ng PE 3.1 and later.

Description: The group of the directories created by syslog-ng. To preserve the original properties of an existing directory, use the option without specifying an attribute: dir-group().

Description: The owner of the directories created by syslog-ng. To preserve the original properties of an existing directory, use the option without specifying an attribute: dir-owner().

Description: The permission mask of directories created by syslog-ng. Log directories are only created if a file after macro expansion refers to a non-existing directory, and directory creation is enabled (see also the create-dirs() option). For octal numbers prefix the number with 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing directory, use the option without specifying an attribute: dir-perm(). Note that when creating a new directory without specifying attributes for dir-perm(), the default permission of the directories is masked with the umask of the parent process (typically 0022).

Description: Name of a file, that contains an X.509 certificate (and the public key) in PEM format. The syslog-ng PE application uses this certificate to encrypt the logstore files which can be decrypted using the private key of the certificate.

Description: Flags influence the behavior of the destination driver.

Description: The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

Description: Set the group of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: group().

Description: The number of messages that the output queue can store.

Description: The number of blocks in the journal file. If set to 0, syslog-ng will set it to the default value (4). The maximal value is 255. If journal-block-count() is set higher than 255, syslog-ng will use the maximum value.

destination d_logstore {
     logstore("/var/log/messages-logstore.lgs"
        encrypt-certificate ("/opt/syslog-ng/etc/syslog-ng/keys/public-server-certificate.pem")
        journal-block-size(524288)
        journal-block-count(5));
};

Description: The size of blocks (in bytes) in the journal file. The size of the block must be a multiple of the page size: if not, syslog-ng PE automatically increases it to the next multiple of the page size. The maximum size of a journal block is 32MB, the minimum size is 256KB. If the value specified as journal-block-size() is lower than minimum size or higher than the maximum size, syslog-ng PE will use the minimum or maximum size, respectively.

destination d_logstore {
     logstore("/var/log/messages-logstore.lgs"
        encrypt-certificate ("/opt/syslog-ng/etc/syslog-ng/keys/public-server-certificate.pem")
        journal-block-size(524288)
        journal-block-count(5));
};

Description: Set the owner of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: owner().

Description: The permission mask of the file if it is created by syslog-ng. For octal numbers prefix the number with 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing file, use the option without specifying an attribute: perm().

Description: Specifies a template defining the logformat to be used in the destination. Macros are described in the section called “Macros of syslog-ng PE”. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.

Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

Description: The minimum time (in seconds) that should expire between two timestamping requests. When syslog-ng closes a chunk, it checks how much time has expired since the last timestamping request: if it is higher than the value set in the timestamp-freq() parameter, it requests a new timestamp from the authority set in the timestamp-url() parameter.

By default, timestamping is disabled: the timestamp-freq() global option is set to 0. To enable timestamping, set it to a positive value.

Description: If the Timestamping Server has timestamping policies configured, specify the OID of the policy to use with this parameter. syslog-ng PE will include this ID in the timestamping requests sent to the TSA. This option is available in syslog-ng PE 3.1 and later.

Description: The URL of the Timestamping Authority used to request timestamps to sign logstore chunks. Note that syslog-ng PE currently supports only Timestamping Authorities that conform to RFC3161 Internet X.509 Public Key Infrastructure Time-Stamp Protocol, other protocols like Microsoft Authenticode Timestamping are not supported.

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see the section called “Date-related macros”.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see the section called “A note on timezones and timestamps”.