Graylog API Query Results to CSV

I thought I would post this quick bit because it took me a while to figure out how to use the Graylog API to run some of the same searches that I’ve built dashboards on or use on a regular basis. There is still some refinement to do on the authentication piece, and maybe some optimization as well.
The need for this was to schedule the export of a search at regular intervals. The syntax in the example is different, but I wanted to get an export the raw logs of all the modifications to groups, adding users, deleting users, etc. in Active Directory for the past month, and then just ship that CSV off to another team.

For this example, I wanted to show several things. Primarily, how to get the data from the API based on a search, but also how to include some AND/OR operations with parenthesis. Basically from a shell on the Graylog server, I want to search for the string “Error” in logs from two servers, 10.10.10.101 and 10.10.10.102. The Graylog server is 10.10.20.10.

This is the query in the Graylog WebUI Search bar is:
Error AND (source:10.10.10.101 OR source:10.10.10.102)

The resulting url from the search in the WebUI is:
https://10.10.20.10/search?rangetype=relative&fields=message%2Csource&width=2560&highlightMessage=&relative=300&q=Error+AND+(source%3A10.10.10.101+OR+source%3A10.10.10.102)

The relative=300 refers to the last 300 seconds, aka 5 minutes.
The fields=message%2Csource refers to the columns to return, you can add other fields here as well.
Everything to the between q= and &range is the search query.

The resulting API query is:
curl -k --user admin -X GET "https://10.10.20.10:443/api/search/universal/relative/export?query=Error%20AND%20(source%3A10.10.10.101%20OR%20source%3A10.10.10.102)&range=300&fields=message%2Csource"
* Notice that in the URL, the spaces enclosing the “AND” and “OR” are represented by the “+” and that needs to change to “%20” in the API query.

That dumps the output in comma separated lines on the terminal, so just output to a file and there you have a CSV.
curl -k --user admin -X GET "https://10.10.20.10:443/api/search/universal/relative/export?query=Error%20AND%20(source%3A10.10.10.101%20OR%20source%3A10.10.10.102)&range=300&fields=message%2Csource" > 20181010_Errors_Last_5_Minutes_search.csv

This query will prompt for the password of admin, and this is the piece I still need to work on. It will work if put in admin:”supersecretpassword”, but I don’t want to leave that behind in my history, cron, or script. I’m nearly certain that there is a token or something that solves this problem, just haven’t had the time to dig in further.

Hope that helps!

Leave a reply