The following describes how to configure the HDFS destination where you want to forward logs.
Prerequisites:
To configure the HDFS destination
- Navigate to Log > Destinations and select
to add a new destination.
- Enter a name for the destination.
- Select HDFS destination.
- In File path, specify the absolute path for the destination file on the HDFS server.
Figure 191: Log > Destinations — Configuring an HDFS destination
-
Select the syslog protocol to use from the Syslog protocol field.
-
To use the legacy BSD-syslog protocol described in RFC 3164, select Legacy and specify the message template to use. Select Legacy to use the message format described in the RFC, or ISO date to replace the original time stamp with an ISO8061 compliant time stamp that includes year and timezone information. To customize the format of the message contents using macros, select Custom message part only, or Custom on-wire message to completely reformat the message (including the headers). For details on using macros, see Hard versus soft macros in the syslog-ng PE Administration Guide and Templates and macros in the syslog-ng PE Administration Guide. If you have no special requirements, use the ISO date template.
-
To use the new IETF-syslog protocol, select Syslog. Note that most syslog applications and devices currently support only the legacy protocol. If you need, you can customize the contents of the message using macros. Note that for the IETF-syslog protocol, the header cannot be customized. For details on using macros, see Hard versus soft macros in the syslog-ng PE Administration Guide and Templates and macros in the syslog-ng PE Administration Guide.
- Configure message rate alerting. For detailed instructions, see Configuring message rate alerting.
-
The time stamps of most log messages is accurate only to the second. The syslog-ng Store Box(SSB) appliance can include more accurate time stamps: set how many digits should be included in the Timestamp fractions of a second field. This option corresponds to the frac_digits() parameter of syslog-ng.
-
If the server and SSB are located in a different timezone and you use the Legacy message template (which does not include timezone information), select the timezone of the server from the Timezone field.
-
Set the size of the disk buffer (in Megabytes) in the Output disk buffer field. If the remote server becomes unavailable, SSB will buffer messages to the hard disk, and continue sending the messages when the remote server becomes available. This option corresponds to the log_disk_fifo_size() parameter of syslog-ng.
Note that SSB does not pre-allocate the hard disk required for the disk buffer, so make sure that the required disk space is available on SSB. For details on creating archiving policies and adjusting the disk-fillup prevention, see Archiving and cleanup and Preventing disk space fill up.
Example: Calculating disk buffer size
The size of the disk buffer you need depends on the rate of the incoming messages, the size of the messages, and the length of the network outage that you want to cover. For example:
-
SSB is receiving 15000 messages per second
-
On the average, one message is 250 bytes long
-
You estimate that the longest time the destination will be unavailable is 4 hours
In this case, you need a disk buffer for 250 [bytes] * 15000 [messages per second] * 4*60*60 [seconds] = 54000000000 [bytes], which is 54000 Megabytes (in other words, a bit over 50 GB).
-
Click 
.
-
To start sending messages to the destination, include the new destination in a logpath. For details, see Log paths: routing and processing messages.
On the Log > Paths page, the HDFS destination will be displayed in the remote category.
Log paths: routing and processing messages
This section describes how to create and configure log paths in syslog-ng Store Box(SSB). Log paths and filters allow you to select and route messages to specific destinations. You can also parse and modify the log messages in log path using message parsers and rewriter rules. The log path processes the incoming messages as follows.
-
Parse the message as a syslog message (unless message parsing is explicitly disabled for the source).
-
Classify the message using a pattern database.
-
Modify the message using rewrite rules (before filtering).
-
Filter the messages, for example, based on sender hostname or message content. If the message does not match the configured filter, syslog-ng Store Box(SSB) will not send it to the destination.
-
Parse the text of the message (that is, the ${MESSAGE} part) using a key-value parser or the sudo parser.
-
Modify the message using rewrite rules (after filtering and other parsing).
-
SSB sends the message to the destinations set in the log path. The destinations are local, optionally encrypted files on SSB, or remote servers, such as a database server.
Two log paths are available by default in the syslog-ng Store Box(SSB) appliance (see Log > Paths):
Figure 192: Log > Paths — Default logpaths of SSB
-
The first log path collects the local messages of SSB. It sends every message of the web interface, the built-in syslog-ng server, and other internal components to the local logspace.
-
The second log path collects messages sent to SSB using the default syslog sources (for details, see Default message sources in SSB) or via SNMP (for details, see Receiving SNMP messages). These messages are stored in the center logspace.
NOTE: Note that both default log paths are marked as Final: if you create a new log path that collects logs from the default sources, make sure to adjust the order of the log paths, or disable the Final option for the default log path.
This section describes how to create a new log path.
To create a new log path
-
Navigate to Log > Paths and select . A new log path is added to the list of log paths.
-
Select a source for the log path from the Source field. Messages arriving to this source will be processed by this log path. To add more sources to the log path, select in the source field and repeat this step.
Figure 193: Log > Paths — Creating a new logpath
Remote sources receive messages from the network, while built-in sources are messages that originate on syslog-ng Store Box(SSB). However, note that the SNMP source (for details, see Receiving SNMP messages) is listed in the built-in section.
TIP: To process every message of every source, leave the source option on all. This is equivalent to using the catchall flag of syslog-ng.
-
Select a destination for the log path from the Destination field. Messages arriving to this source will be forwarded to this destination. To add more destinations to the log path, select in the destination field and repeat this step.
NOTE: Remote destinations forward the messages to external servers or databases and are configured on the Log > Destinations page (for details, see Forwarding messages from SSB).
Local destinations store the messages locally on SSB and are configured on the Log > Logspaces page (for details, see Storing messages on SSB).
If you do not want to store the messages arriving to this log path, leave the Destination field on none.
|
|
Caution:
The none destination discards messages — messages sent only to this destination will be lost irrevocably. |
-
If you do not want other log paths to process the messages sent to a destination by this log path, select the Final option.
The order of the log paths is important, especially if you use the Final option in one or more destinations, because SSB evaluates log paths in descending order. Use the 
, 
buttons to position the log path if needed.
-
To enable flow-control for this log path, select the flow-control option. For details on how flow-control works, see Managing incoming and outgoing messages with flow-control.
NOTE: As a result of toggling the flow-control status of the logpath, the output buffer size of the logpath's destination(s) will change. For the changes to take effect, navigate to Basic Settings > System > Service control and click Restart syslog-ng.
-
If you do not want to send every message from the sources to the destinations, use filters. Select the filter to use from the Filter field, click , and configure the filter as needed. To apply more filters, click and select a new filter. Note that SSB sends only those messages to the destinations that pass every listed filter of the log path. The available filters are described in Filtering messages.
Figure 194: Log > Paths — Filtering log messages
-
Click . After that, the new log path will start to collect log messages.
TIP: If you do not want to activate the log path immediately, deselect the Enable option.
