A fresh look at Logstash
Soon after the release of elasticsearch it became clear that elasticsearch was good at more than providing search. It turned out that it could be used to store logs very effectively. That is why logstash was using elasticsearch. It contained standard parsers for apache httpd logs. To obtain the logs it had file monitoring plugins. It had plugins to extend and filter the content, and it had plugins to send the content to elasticsearch. That is Logstash in a nutshell back in the days. Of course the logs had to be shown, therefore a tool called Kibana was created. Kibana was a nice tool to create highly interactive dashboards to show and analyse your data. Together they became the famous ELK suite. Nowadays we have a lot more options in all these tools. We have Ingest node in elastic to pre-process documents before they move into elasticsearch, we have beats to monitor files, databases, machines, etc. And we have very nice and new Kibana dashboards. Time to re-investigate what the combination of Logstash, Elasticsearch and Kibana can do. In this blog post I’ll focus on Logstash.
X-Pack
As the company elastic has to make some money as well, they have created a product called X-Pack. X-Pack has a lot of features that sometimes span multiple products. There is a security component, by using this you can make users login in when using kibana and secure your content. Other interesting parts of X-Pack are machine learning, graph and monitoring. Parts of X-Pack can be used free of charge, you do need a license though. For other parts you need a paid license. I personally like the monitoring part so I regularly install X-Pack. In this blogpost I’ll also investigate the X-Pack features for Logstash. I’ll focus on out-of-the-box functionality and mostly what all these nice new things like monitoring and pipeline viewing bring us.
Using the version 6 release candidate
As elastic has already given us a RC1 of their complete stack, I’ll use this one for the evaluation. Beware though, this is still a release candidate, so not production ready.
What does Logstash do
If you never really heard about Logstash, let me give you a very short introduction. Logstash can be used to obtain data from a multitude of different sources. Than filter, transform and enrich the data. Finally store the data to again a multitude of datasources. Example data sources are relational databases, files, queues and websockets. Logstash ships with a large number of filter plugins, with these we can process data to exclude some fields. We can also enrich data, lookup information about ip addresses, or lookup records belonging to an id in for instance elasticsearch or a database. After the lookup we can add data to the document or event that we are handling before sending it to one or more outputs. Outputs can be elasticsearch, a database, but also queue’s like Kafka or RabbitMQ. In the later releases logstash started to add more features that a tool handling large amounts of data over longer periods need. Things like monitoring and clustering of nodes were introduced and also persisting incoming data to disk. By now logstash in combination with Kibana and Elasticsearch is used by very large companies but also by a lot of start ups to monitor their servers and handle all sorts of interesting data streams. Enough of this talk, let us get our hands dirty. First step install everything on our developer machines.
Installation
I’ll focus on the developer machine, if you want to install it on a server please refer to the extensive logstash documentation.
First download the zip or tar.gz file and extract it to a convenient location. Now create a folder where you can store the configuration files. To make the files small and to show you that you can split them, I create three different files in this folder: input.conf, filters.conf and output.conf. The most basic configuration is one with a stdin for input, no filters and stdout for output. Below the contents for the two files
input {
stdin{}
}
output {
stdout {
codec => rubydebug
}
}Time to start logstash. Step into the downloaded and extracted folder with the logstash binaries and execute the following command.
bin/logstash -r -f ../logstashblog/the -r, can be used during development for reloading the configuration on change. Beware, this does not work with the stdin plugin. With -f we tell logstash to load a configuration file or directory. In our case a directory containing the three mentioned files. When logstash is ready it will print something like this:
[2017-10-28T19:00:19,511][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2017-10-28T19:00:19,526][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
Now you can type something and the result is the created document or event that went through the almost empty pipeline. The thing to notice is that we now have a field called message containing the text we entered.
Just some text for input
{
"@version" => "1",
"host" => "Jettros-MBP.fritz.box",
"@timestamp" => 2017-10-28T17:02:18.185Z,
"message" => "Just some text for input"
}Now that we know it is working, I want you to have a look at the monitoring options you have available using the rest endpoint.
http://localhost:9600/
{
"host": "Jettros-MBP.fritz.box",
"version": "6.0.0-rc1",
"http_address": "127.0.0.1:9600",
"id": "20290d5e-1303-4fbd-9e15-03f549886af1",
"name": "Jettros-MBP.fritz.box",
"build_date": "2017-09-25T20:32:16Z",
"build_sha": "c13a253bb733452031913c186892523d03967857",
"build_snapshot": false
}You can use the same url with different endpoints to get information about the node, the plugins, stats and hot threads:
- http://localhost:9600/_node
- http://localhost:9600/_node/plugins
- http://localhost:9600/_node/stats
- http://localhost:9600/_node/hot_threads
It becomes a lot more fun if we have a UI, so let us install xpack into logstash. Before we can run logstash with monitoring on, we need to install elasticsearch and kibana with X-pack installed into those as well. Refer to the X-Pack documentation on how to do it. The basic commands to install x-pack into elasticsearch and kibana are very easy. For now I disable security by adding the following line to both kibana.yml and elasticsearch.yml: xpack.security.enabled: false. After installing x-pack into logstash we have to add the following lines to the logstash.yml file in the config folder
xpack.monitoring.elasticsearch.url: ["http://localhost:9200"]
xpack.monitoring.elasticsearch.username:
xpack.monitoring.elasticsearch.password:Notice the empty username and password, this is required when security is disabled. Now move over to Kibana and check the monitoring tab (the heart shape figure) and click on logstash. In the first screen you can see the events, they could be zero, zo please enter some events. Now move to the pipeline tab. Of course with our basic pipeline, this is a bit stupid, but imagine what it will show later on.
Time to get some real input.
Import the Signalmedia dataset
Signalmedia has provided a dataset you can use for research. More information about the dataset and how to obtain it can be found here. The dataset contains an exact amount of 1 million news documents. You can download the file as a file that contains a JSON document on each line. The JSON document has the following format:
{
"id": "a080f99a-07d9-47d1-8244-26a540017b7a",
"content": "KUALA LUMPUR, Sept 15 (MySinchew) -- The Kuala Lumpur City Hall today issued ...",
"title": "Pay up or face legal action: DBKL",
"media-type": "News",
"source": "My Sinchew",
"published": "2015-09-15T10:17:53Z"
}
We want to import this big file with all the JSON documents as separate documents into elasticsearch using logstash. The first step is to create a logstash input. Use the path to point to the file. We can use the logstash file plugin to load the file, tell it to start at the beginning and mark each line as a JSON document. The file plugin has more options you can use. It can also handle rolling files that are used a lot in logging.
input {
file {
path => "/Volumes/Transcend/signalmedia-1m.jsonl"
codec => "json"
start_position => beginning
}
}That is it, with the stdout plugin and the rubydebug codec this would give the following output.
{
"path" => "/Volumes/Transcend/signalmedia-1m.jsonl",
"@timestamp" => 2017-10-30T18:49:45.948Z,
"@version" => "1",
"host" => "Jettros-MBP.fritz.box",
"id" => "a080f99a-07d9-47d1-8244-26a540017b7a",
"source" => "My Sinchew",
"published" => "2015-09-15T10:17:53Z",
"title" => "Pay up or face legal action: DBKL",
"media-type" => "News",
"content" => "KUALA LUMPUR, Sept 15 (MySinchew) -- The Kuala Lumpur City Hall today issued ..."
}Notice that besides the fields we expected: id, content, title, media-type, source and published we also got some additional fields. Before sending this to elasticsearch we want to clean it up. We do not need the path, host, @timestamp, @version. There is also something with the field id. We want to use the id field to create the document in elasticsearch, but we do not want to add it to the document. If we need the value of id in the output plugin later on, but we do not want to add it as a field to the document we can move it to the @metadata object. That is exactly what the first part of the filter does. The second part removes the fields we do not need.
filter {
mutate {
copy => {"id" => "[@metadata][id]"}
}
mutate {
remove_field => ["@timestamp", "@version", "host", "path", "id"]
}
}With these filters in place the output of the same document would become:
{
"source" => "My Sinchew",
"published" => "2015-09-15T10:17:53Z",
"title" => "Pay up or face legal action: DBKL",
"media-type" => "News",
"content" => "KUALA LUMPUR, Sept 15 (MySinchew) -- The Kuala Lumpur City Hall today issued ..."
}Now the content is ready to be send to elasticsearch, so we need to configure the elasticsearch output plugin. When sending data to elastic you first need to think about creating the index and the mapping that goes with it. In this example I am going to create an index template. I am not going to explain a lot about the mappings as this is not an elasticsearch blog. But with the following code we insert the mapping template when connecting to elasticsearch and we can insert all documents. Do look at the way the document_id is created. Remember we talked about that @metadata and how we copied the id field into it. This is the reason why we did it. Now we use that value as the id of the document when inserting it into elasticsearch.
output {
elasticsearch {
index => "signalmedia"
document_id => "%{[@metadata][id]}"
document_type => "doc"
manage_template => "true"
template => "./signalmedia-template.json"
template_name => "signalmediatemplate"
}
stdout { codec => dots }
}
Notice there are two outputs configured. The elasticsearch output of course, but also a stdout. This time not with the rubydebug codec, this would be way to verbose. We use the dots codec. This codec prints a dot for each document it parses. For completeness I also want to show the mapping template. In this case I positioned it in the root folder of the logstash binary, usually this would of course be an absolute path somewhere else.
{
"index_patterns": ["signalmedia"],
"settings": {
"number_of_replicas": 0,
"number_of_shards": 3
},
"mappings": {
"doc": {
"properties": {
"source": {
"type": "keyword"
},
"published": {
"type": "date"
},
"title": {
"type": "text"
},
"media-type": {
"type": "keyword"
},
"content": {
"type": "text"
}
}
}
}
}Now we want to import all the million documents and have a look at the monitoring along the way. Let’s do it.
Running a query
Of course we have to prove the documents are now available in elasticsearch. So lets execute one of my favourite queries that makes use of the new significant text aggregation. First the request and then parts of the response.
GET signalmedia/_search
{
"query": {
"match": {
"content": "netherlands"
}
},
"aggs": {
"my_sampler": {
"sampler": {
"shard_size": 200
},
"aggs": {
"keywords": {
"significant_text": {
"field": "content",
"filter_duplicate_text": true
}
}
}
}
},
"size": 0
}Just a very small part of the response, I stripped out a lot of the elements to make it better viewable. Good to see that that see dutch as a significant word when searching for the netherlands and of course geenstijl.
"buckets": [
{"key": "netherlands","doc_count": 527},
{"key": "dutch","doc_count": 196},
{"key": "mmsi","doc_count": 7},
{"key": "herikerbergweg","doc_count": 4},
{"key": "konya","doc_count": 14},
{"key": "geenstijl","doc_count": 3}
]Concluding
Good to see the nice ui options in Kibana. The pipeline viewer is very useful. In a next blog post I’ll be looking at Kibana and all the new and interesting things in there.
