map-value-pairs: Rename value-pairs to normalize logs

The map-value-pairs() parser allows you to map existing name-value pairs to a different set of name-value pairs. You can rename them in bulk, making it easy to use for log normalization tasks (for example, when you parse information from different log messages, and want to convert them into a uniform naming scheme). You can use the normal value-pairs expressions, similarly to value-pairs based destinations.

Available in syslog-ng OSE version 3.10 and later.

parser parser_name {
    map-value-pairs(
        <list-of-value-pairs-options>
    );
};

parser p_remap_name_values {
    map-value-pairs(
        pair("username", "'($sha1 $.apache.username)")
        key('.apache.*' rekey(add-prefix("webserver")))
    );
};

Conditional rewrites

Starting with 3.2, it is possible to apply a rewrite rule to a message only if certain conditions are met. The condition() option effectively embeds a filter expression into the rewrite rule: the message is modified only if the message passes the filter. If the condition is not met, the message is passed to the next element of the log path (that is, the element following the rewrite rule in the log statement, for example, the destination). Any filter expression normally used in filters can be used as a rewrite condition. Existing filter statements can be referenced using the filter() function within the condition. For details on filters, see Filters.

How conditional rewriting works

Purpose:

The following procedure summarizes how conditional rewrite rules (rewrite rules that have the condition() parameter set) work. The following configuration snippet is used to illustrate the procedure:

rewrite r_rewrite_set{
    set(
        "myhost",
        value("HOST")
        condition(program("myapplication"))
    );
};
log {
    source(s1);
    rewrite(r_rewrite_set);
    destination(d1);
};

rewrite r_rewrite_set{set("myhost", value("HOST") condition(program("myapplication")));};

filter f_rewritefilter {program("myapplication");};
rewrite r_rewrite_set{set("myhost", value("HOST") condition(filter(f_rewritefilter)));};

Adding and deleting tags

To add or delete a tag, you can use rewrite rules. To add a tag, use the following syntax:

rewrite <name_of_the_rule> {
    set-tag("<tag-to-add>");
};

To delete a tag, use the following syntax:

rewrite <name_of_the_rule> {
    clear-tag("<tag-to-delete>");
};

You cannot use macros in the tags.

Anonymizing credit card numbers

Log messages of banking and e-commerce applications might include credit card numbers (Primary Account Number or PAN). According to privacy best practices and the requirements of the Payment Card Industry Data Security Standards (PCI-DSS), PAN must be rendered unreadable. The syslog-ng OSE application uses a regular expression to detect credit card numbers, and provides two ways to accomplish this: you can either mask the credit card numbers, or replace them with a hash. To mask the credit card numbers, use the credit-card-mask() or the credit-card-hash() rewrite rules in a log path.

@include "scl/rewrite/cc-mask.conf"

rewrite {
    credit-card-mask(value("<message-field-to-process>"));
};