What is ELK Stack? Elasticsearch, Logstash and Kibana

source: https://elastic-stack.readthedocs.io/

Many Software Applications are deployed on Cloud (i.e AWS, GCP, Azure) which follow the Distributed Architecture. Means Webservers, Database servers, etc are running somewhere in the Cloud on different VMs(Virtual Machines). To monitor the Application and Infrastructure there is a need for a monitoring solution.

Let me give you one more simple example, Lets consider your application is running on 40 VMs on Cloud and they are generating server logs simultaneously and if you are monitoring those logs on a central monitoring platform (i.e ELK) then quickly you can identify the issue if there is any and troubleshoot it on that particular VM only. This way overhead of the engineer is reduced and troubleshooting happened quickly. That’s why monitoring of the Application and Infrastructure is necessary.

To manage and maintain the system in healthier way, we need to analyze full system activities, that’s where monitoring solution like ELK Stack came into the picture.

In this Blogpost, I am going to give you introduction about ELK Stack and some of its features and alternatives of ELK in the market.

ELK Stack is an Open Source Distributed monitoring solution and Log management solution.

ELK is the collection of three open source tools i.e

1. Elasticsearch

2. Logstash

3. Kibana

These three tools were developed and maintained by a Company named ELASTIC. The official website of ELASTIC is https://www.elastic.co/ . ELK Stack gives the ability to aggregate logs from all your system and Applications and also create visualizations for Application and Infrastructure Monitoring.

E stands for ELASTICSEARCH which is a search and analytics engine. Also, it is used to store and index the logs. In fact, it is the main component of ELK Stack.

L stands for LOGSTASH which is a data ingestion tool which allow users to easily ingest data from variety of sources, transform it and send it to desired destination for storing.

K stands for KIBANA which allow user to visualize Data using different charts and visualization techniques to analyze the Logs and Events( A Web Interface to explore the Data present in Elasticsearch).

What is Elasticsearch ? — A Real-time search and Analytics tool

Source: Elastic.co

Elasticsearch is a distributed, open-source search and analytics engine built on the Apache Lucene library and developed in JAVA. It works on Structured, Unstructured, Numerical, and Geospatial Data. The Data is stored in the form of a JSON Document.

Elasticsearch is a NoSQL database. It is distributed, centrally stores your data, allow RESTful searches and this analytical engine is capable of solving different kinds of growing number of use cases. Because of the power of elasticsearch, It has been adopted in search engine platforms for modern web and mobile applications.

Features of Elasticsearch -

• Lightening fast full-text search.
• Security analytics and infrastructure monitoring.
• Can be scaled to thousands of servers and can handle petabytes of data.
• Can be integrated with Kibana to provide real-time visualization of Elasticsearch data for accessing application performance and for monitoring logs and infrastructure metrics data.
• It uses Machine Learning to automatically model the behaviour of your data in real-time.

What is Logstash? — Ingesting and Routing your Data

source: elastic.co

Logstash is a Data collection pipeline tool that accepts inputs from various sources, executes different transformations, and exports the data to various targets. It collects Data Inputs and Feeds it into Elasticsearch.

This virtually any type of log that you manage i.e System logs, Webserver logs, Error logs and App logs.

With Logstash, it’s super easy to take all logs and store them in a central location. The only prerequisite is a java runtime, and it takes just two commands to get Logstash up and running.

Think of Logstash as a pipeline for event processing i.e it takes precious little time to choose the inputs, configure the filters, and extract the relevant, high-value data from your logs.

What is Kibana ? — Visualization Tool

source: Elastic.co

Kibana is a Data Visualization Dashboard. This tool is helpful for DevOps Engineers, Developers, System Admins, Business Managers, Project Managers, etc. This dashboard offers various Interactive diagrams, Geospatial data, and graphs to visualize.

It can be used for search, view, and interact with data stored in Elasticsearch directories.

You can Visualize trends and patterns for data that would otherwise be extremely tedious to read and interpret. Eventually, each business line can make practical use of your data collection as you help them customize their dashboards.

Alternative Distributed Log Management Tools: —

1. Splunk

2. Nagios

3. Zabbix

4. Prometheus

5. LogDNA

6. Garylog

7. Fluentd

Please 👏👏👏👏 the article if you liked it. It will boost my confidence.