Case study

Industry
Financial services
Platforms
Elastic Stack, Kafka, Python, Ansible
Duration
About 6 months
of logs ingested per day
overloaded cluster split into three stable ones
downtime during the whole operation
The challenge
A bank ran one large Elastic Stack cluster for application logs, ingesting about 100 TB per day across more than 200,000 shards. The master node could no longer handle the cluster state. The whole cluster suffered: ingestion lagged, searches slowed, and logs arrived late.
The fix had to happen while the service stayed available. Downtime on the logging platform was not an option.
What we found
The root cause was scale concentrated in one place: too many shards for a single cluster state to manage. One discovery shaped the plan: cross-cluster search would keep all data searchable from one place, even after the data was spread across several clusters.
What we did
Designed a multi-cluster architecture to divide the load.
Redirected new data to the new clusters, instead of moving historical data through snapshot and restore.
Used cross-cluster search so teams kept one view of old and new data.
Tested in development, qualification and pre-production environments before going live.
Kept monitoring in place throughout the operation.
Results
One overloaded cluster became three stable ones: from more than 200,000 shards to three clusters of about 70,000 shards each.
A stable cluster state: the lag, search and ingestion issues caused by the overloaded master node were resolved.
Zero downtime during the whole operation.
What it means for the client
The logging platform now has room to grow. The recommended next step is tighter control of shard creation, to keep the clusters healthy as volumes rise.
Next case study


