Learning Logstash - Part I
Logstash a component of the ELK stack. ELK is the acronym of Elasticsearch, Logstash, and Kibana. I learned how to use it with examples.
- The ELK stack
- Logstash can deal with much more than logs
- Installation
- Acknowledgement and further resources
The ELK stack
Yes, the acronym ELK was chosen to fit the common name of the animal Cervus canadensis, the oldest in the deer family.
In short:
- ElasticSearch provides full-text indexing and search. It is a multi-server implementation of the Lucene engine, which is a Java-based indexing and full-text search engine.
- Logstash is a data-processing pipeline running on the server side that ingests1 data from multiple sources, transforms it, and sends it to a “stash” like ElasticSearch 2.
- Kibana allows user visualize data in ElasticSearch.
Logstash can deal with much more than logs
Logstash can convert virtually any type of data, for instance tab-delimited files, into JSON objects that can be stored in ElasticSearch, which can then be indexed and searched. Therefore, despite the word ‘log’ in its name, Logstash can be applied to any type of structured data that we wish to index and search. This can, for instance, also include gene annotation and phenotype data (or known as metadata) in omics studies.
Installation
It is possible to install Logstash from source files, which can be downloaded at elastic.co. Alternatively, one can install it from package managers, for instance apt on Debian systems, or run it from a Docker container. The download page gives clear instructions.
Now we have installed Logstash, we can run it to convert virtually any data into JSON files. We will explore this in the next part of the tutorial.
Acknowledgement and further resources
I started to learn Logstash thanks to baddila.
- Logstash on Wikitech. It documents how ELK and Logstash is used by Wikipedia.