Resend log messages from SSB to a remote destination (4287178)

Information needed for the procedure

• The name of the logspace where the logs to be resent reside
• Time period of the logs to be resent
• Destination host, port and message format
• Decide if you need to limit sending rate (eg. SIEM)
• Message filter if needed

In this guide we use the followings

- Logspace name: center
- Time period: 08:00 - 10:00 on 2nd of September
- Destination:  IP: 10.10.10.10  Port: 514  Format: BSD RFC3164
- Rate limit: 1000 messages per second
- Logs to be resent:  Syslog-ng's log (program filter)

Procedure

You have to change the bold part of the commands based on your needs.

1. Log in to the CORE SHELL.

cd /opt/ssb/var/logspace/name/center/2018/09-02/

3. Build the filter and check if you get the desired logs

If all logs of a whole day have to be resent no --filter option is needed.

/opt/syslog-ng/bin/lgstool cat messages.store --filter='program("syslog-ng") and ("${R_HOUR}" > "08" and "${R_HOUR}" < "10")'

4. Resend the logs with loggen

/opt/syslog-ng/bin/lgstool cat messages.store --filter='program("syslog-ng") and ("${R_HOUR}" >= "08" and "${R_HOUR}" < "10")' | \
/opt/syslog-ng/bin/loggen -r 1000 -R - 10.10.10.10 514

Notes:
SSB stores the logs in one logstore file per day at /opt/ssb/var/logspace/name/LOGSPACE_NAME/$YEAR/$MONTH-$DAY/messages.store. If the time period is greater than one day, the procedure has to be repeat on each day's logstore file.

In case of encrypted logspaces you have to copy the key file of the encryption certificate to SSB. Also, you have to add it to lgstool command using "-k" option.

For detailed usage of lgstool/loggen commands, filtering and macros see Syslog-ng Administration Guide.

Workaround

IETF format can transfer multiline messages but it is difficult to get the framing right.

An awk command below was written to do just that.  This one must use the no parse option so even if the output is not in the right format it will be sent.

---------------------------

/opt/syslog-ng/bin/lgstool cat -t '<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} -${PID} -${MSGID} -${SDATA} ${MSG}\n' messages.store | awk 'BEGIN{ len = 0 ; lin = "" } $1 ~ /^<[0-9]{1,3}>1$/ && NR!=1 { print len+1" "lin } { if ( $1 ~ /^<[0-9]{1,3}>1$/ ) { if (!( $5 ~ /^-$/)) { $5 = substr($5, 2) } ; if (!( $6 ~ /^-$/)) { $6 = substr($6, 2) } ; if (!( $7 ~ /^-$/)) { $7 = substr($7, 2) } ; lin = $0 ; len = length($0) } else { lin = lin"\n"$0 ; len = len + length($0) + 1 } }' | /opt/syslog-ng/bin/loggen -S -di -R - xxx.xxx.xxx.xxx 601

---------------------------

The receiver side log format must be set as syslog.

xxx.xxx.xxx.xxx is the destination IP.
loggen -S was used for TCP.

NOTE:
This is a workaround. If this worked for one logspace that does not mean it will work for another. There are many special characters, character encoding, etc. that may cause unintended behavior. Each logspace should be treated as a unique scenario.