SOLVING COMPLEXITY AND IMPROVING STORAGE EFFICIENCY IN MANAGING ELASTICSEARCH CLUSTERS USING KUBERNETES AND ECK OPERATORS

Abstract

Elasticsearch is a popular open-source tool based on Apache Lucene, widely used for search, storing large amounts of data. It is being used in many small startups to large enterprises for Full-text search, log analytics, application search, geospatial data analysis and very recently also used in SIEM. As business needs grows, the Elasticsearch cluster needs to be scaled and often, the operators of Elasticsearch face many complex issues to manage the cluster. In this article, solution is provided on how to reliably manage the clusters at petabyte scale. Also, key information on monitoring, alerting and autoscaling the clusters are provided for the operators of the clusters to easily manage large scale Elasticsearch clusters. This paper aims to provide key information to the administrators and users of Elasticsearch clusters to reliably use to handle large volumes of data and possibly used as a Data Lake. It will also help organizations to have better financial controls, operational efficiency and do more with less by adopting key solutions discussed in this paper.