Logstash a component of the ELK stack. ELK is the acronym of Elasticsearch, Logstash, and Kibana. I learned how to use it with examples.

The ELK stack

Yes, the acronym ELK was chosen to fit the common name of the animal Cervus canadensis, the oldest in the deer family.

An Elk. Photo in the public domain, taken by Wikimedia user MONGO.
An Elk. Photo in the public domain, taken by Wikimedia user MONGO.

In short:

  • ElasticSearch provides full-text indexing and search. It is a multi-server implementation of the Lucene engine, which is a Java-based indexing and full-text search engine.
  • Logstash is a data-processing pipeline running on the server side that ingests1 data from multiple sources, transforms it, and sends it to a “stash” like ElasticSearch 2.
  • Kibana allows user visualize data in ElasticSearch.

Logstash can deal with much more than logs

Logstash can convert virtually any type of data, for instance tab-delimited files, into JSON objects that can be stored in ElasticSearch, which can then be indexed and searched. Therefore, despite the word ‘log’ in its name, Logstash can be applied to any type of structured data that we wish to index and search. This can, for instance, also include gene annotation and phenotype data (or known as metadata) in omics studies.

Installation

It is possible to install Logstash from source files, which can be downloaded at elastic.co. Alternatively, one can install it from package managers, for instance apt on Debian systems, or run it from a Docker container. The download page gives clear instructions.

Now we have installed Logstash, we can run it to convert virtually any data into JSON files. We will explore this in the next part of the tutorial.

Acknowledgement and further resources

I started to learn Logstash thanks to baddila.

  1. ingest means to absorb, take food or drink by swallowing or absorbing it. 

  2. stash means a store or supply of something that is kept hidden or secret.