syslog-ng Store Box 7.4.0 - RPC API Quickstart Guide

OK, let's do some real searching now. The command for that is filter, and the arguments that have to be specified in sequence, separated by slashes are:

name of the logspace
from (as a UNIX timestamp)
to (timestamp, too)
your search expression (optional, defaults to none)
offset
limit (these work just as they do in SQL)

At first let's find when was syslog-ng started or restarted in the box:

$ wget -q --no-check-certificate -O - --header "Cookie: AUTHENTICATION_TOKEN=$SESSID" "https://$SSB_IP/api/1/search/logspace/filter/local?from=0&to=9999999999&search_expression=starting up&offset=0&limit=10" | jq '.result'

[
  {
    "tag": [],
    "dynamic": {
      ".SDATA.timeQuality.isSynced": "0"
    },
    "msgid": "",
    "stamp": 1384943027,
    "recvd": 1384943027,
    "pri": 5,
    "facility": 5,
    "host": "rpcapitest",
    "message": "logindexd starting up; version='4.2.4ssb3.2.23'",
    "program": "index-center",
    "pid": "4408"
  },
  {
    "tag": [],
    "dynamic": {
      ".SDATA.timeQuality.isSynced": "0"
    },
    "msgid": "",
    "stamp": 1384943027,
    "recvd": 1384943027,
    "pri": 5,
    "facility": 5,
    "host": "rpcapitest",
    "message": "logindexd starting up; version='4.2.4ssb3.2.23'",
    "program": "index-local",
    "pid": "4407"
  },
[..... and a screenful of other results .....]

As you can see I set the timestamps to 0 and some huge value to make sure time filtering does not apply. We will try that in a sec, too.

But first, this doesn't seem right, it's not only messages from syslog-ng, the indexer processes are talking as well. Let's only print the program messages from the result set with jq to see clearer:

$ wget -q --no-check-certificate -O - --header "Cookie: AUTHENTICATION_TOKEN=$SESSID" "https://$SSB_IP/api/1/search/logspace/filter/local?from=0&to=9999999999&search_expression=starting up&offset=0&limit=10" | jq '.result[].program'
```
"index-center"
"index-local"
"syslog-ng"
"index-local"
"index-center"
"index-local"
"index-local"
"index-local"
"index-local"
"index-local"
```
Let's alter our query to make sure we filter for the messages coming from syslog-ng only:

$ wget -q --no-check-certificate -O - --header "Cookie: AUTHENTICATION_TOKEN=$SESSID" "https://$SSB_IP/api/1/search/logspace/filter/local?from=0&to=9999999999&search_expression=starting up program:syslog-ng&offset=0&limit=10" | jq '.result'
```
[
  {
    "tag": [],
    "dynamic": [],
    "msgid": "",
    "stamp": 1384943029,
    "recvd": 1384943029,
    "pri": 5,
    "facility": 5,
    "host": "ssb1",
    "message": "syslog-ng starting up; version='4.2.4ssb3.2.23', cfg-fingerprint='54d093e5b748276c497a141390614e079fff6cc0', cfg-nonce-ndx='0', cfg-signature='d46966469bf5934c1c7280375f349dcc2d7906dc'",
    "program": "syslog-ng",
    "pid": "4433"
  }
```
It's much better now, as you can see it is a pretty fresh install and syslog-ng has only been started once. The only thing we did is that we added program:syslog-ng to our query. You can use everything there that you could use on the UI. Note that wget actually helps a lot here: it is automatically URL-encoding our query string, making sure that special characters such as the space and the colon are passed properly to the API. In other environments, you might have to do that yourself.
TIP:
To decrease the load on SSB when searching and receive your search results faster, note the following points.
- Use as small a time range as possible
- Prefer AND instead of OR
- Avoid unneeded wildcard characters, such as * and ?
- Use wildcard characters at the end of the tokens if possible

Filtering for time

In the previous examples, we filtered our results by expressions and made sure that no time limitation applied. Let's turn that the other way around and only filter for time.

Let's see the number of logs in the local logspace in a given five minute interval:

$ date +%s -d"Nov 20, 2013 12:55:00"
```
1384948500
```
$ date +%s -d"Nov 20, 2013 13:00:00"
```
1384948800
```
$ wget -q --no-check-certificate -O - --header "Cookie: AUTHENTICATION_TOKEN=$SESSID" "https://$SSB_IP/api/1/search/logspace/number_of_messages/local?from=1384948500&to=1384948800" | jq '.result'
```
38
```
You can of course combine that with regular filtering. Here we get the number of logs produced by indexer processes in that timerange.

$ wget -q --no-check-certificate -O - --header "Cookie: AUTHENTICATION_TOKEN=$SESSID" "https://$SSB_IP/api/1/search/logspace/number_of_messages/local?from=1384948500&to=1384948800&search_expression=program:index*" | jq '.result'
```
7
```
Note that I used a wildcard character in the query: that's possible, too. Pay attention of the URL-encoding and escaping in your environment if you do.

Fetching statistics

In our first example, we investigated which programs issued messages containing the words "starting" and "up". We could have created some statistics simply by using standard command line tools:

      2 index-center
     96 index-local
      1 syslog-ng

However, that's terribly ineffective: we had to pass all messages over the network and use the computing capabilities of our client machine to do the aggregation. This is feasible when we are talking about such numbers but not when there are billions of entries. In that case, it is a much better idea to use the built-in statistics engine of syslog-ng Store Box.

To have SSB generate you the same statistics, you can use the generate_statistics command. These are its arguments:
- name of the logspace
- column to generate statistics from
- from (as a UNIX timestamp)
- to
- your search expression (optional, defaults to none)
- offset (optional)
- limit (optional)
$ wget -q --no-check-certificate -O - --header "Cookie: AUTHENTICATION_TOKEN=$SESSID" "https://$SSB_IP/api/1/search/logspace/generate_statistics/local/program?from=0&to=9999999999&search_expression=starting up" | jq '.result'
```
[
  [
    "syslog-ng",
    1
  ],
  [
    "index-center",
    2
  ],
  [
    "index-local",
    97
  ]
    
```
As the number of entries here can also be huge, this command has the same offset/limit possibilities that the filter command has, and you can also fetch the number of entries (distinct values if you wish) with the number_of_statistics_entries command:

$ wget -q --no-check-certificate -O - --header "Cookie: AUTHENTICATION_TOKEN=$SESSID" "https://$SSB_IP/api/1/search/logspace/number_of_statistics_entries/local/program?from=0&to=9999999999&search_expression=starting up" | jq '.result'
```
3
```

Logging out

And of course your should finish your session by logging out:

$ wget -q --no-check-certificate -O - --header "Cookie: AUTHENTICATION_TOKE=$SESSID" "https://$SSB_IP/api/1/logout" | jq '.'
```
{
"warnings": [],
"error": {
"message": null,
"code": null
},
"result": true
}
```

Now the old session ID won't work any more and you'll get a 403 error when you try to perform a query:

$ wget --no-check-certificate -O - --header "Cookie: AUTHENTICATION_TOKEN=$SESSID" "https://$SSB_IP/api/1/search/logspace/number_of_statistics_entries/local/program?from=0&to=9999999999&search_expression=starting up"

--2014-01-07 14:25:21--  https://1.2.3.4/api/1/search/logspace/number_of_statistics_entries/local/program?from=0&to=9999999999&search_expression=starting%20up
Connecting to 1.2.3.4:443... connected.
WARNING: cannot verify 1.2.3.4's certificate, issued by `/C=AT/L=asdfasdf/O=rtyrty/OU=cbdfg/ST=ertertert/CN=rpcapitest.gamma.test.balabit root CA':
Self-signed certificate encountered.
HTTP request sent, awaiting response... 403 Forbidden
2014-01-07 14:25:21 ERROR 403: Forbidden.

Please select your product:

To serve you better, please complete the Purpose of your Chat:

Recommended Solutions for Your Problem

syslog-ng Store Box 7.4.0 - RPC API Quickstart Guide

Searching

Filtering for time

Fetching statistics

Logging out