Case study

Splitting a 100 TB/day Elastic cluster with zero downtime

Splitting a 100 TB/day Elastic cluster with zero downtime

Splitting a 100 TB/day Elastic cluster with zero downtime

How we split a bank's 100 TB/day Elastic cluster into three stable clusters, with zero downtime and every log still searchable.

How we split a bank's 100 TB/day Elastic cluster into three stable clusters, with zero downtime and every log still searchable.

Elastic Cluster Split

Industry

Financial services

Platforms

Elastic Stack, Kafka, Python, Ansible

Duration

About 6 months

100 TB

100 TB

of logs ingested per day

1 → 3

1 → 3

overloaded cluster split into three stable ones

Zero

Zero

downtime during the whole operation

No headings found on page

Facing a similar challenge?

Talk directly with the engineers who do the work.

Subscribe

Subscribe to get the latest insights straight in your inbox

By entering your email, you agree to receive emails from the DinaBridge team.

The challenge

A bank ran one large Elastic Stack cluster for application logs, ingesting about 100 TB per day across more than 200,000 shards. The master node could no longer handle the cluster state. The whole cluster suffered: ingestion lagged, searches slowed, and logs arrived late.

The fix had to happen while the service stayed available. Downtime on the logging platform was not an option.

What we found

The root cause was scale concentrated in one place: too many shards for a single cluster state to manage. One discovery shaped the plan: cross-cluster search would keep all data searchable from one place, even after the data was spread across several clusters.

What we did

  • Designed a multi-cluster architecture to divide the load.

  • Redirected new data to the new clusters, instead of moving historical data through snapshot and restore.

  • Used cross-cluster search so teams kept one view of old and new data.

  • Tested in development, qualification and pre-production environments before going live.

  • Kept monitoring in place throughout the operation.

Results

  • One overloaded cluster became three stable ones: from more than 200,000 shards to three clusters of about 70,000 shards each.

  • A stable cluster state: the lag, search and ingestion issues caused by the overloaded master node were resolved.

  • Zero downtime during the whole operation.

What it means for the client

The logging platform now has room to grow. The recommended next step is tighter control of shard creation, to keep the clusters healthy as volumes rise.

Have a complex engineering problem to solve?

Have a complex engineering problem to solve?

Whether you are modernizing a platform, scaling your infrastructure, improving visibility, or looking for additional engineering capacity, DinaBridge can help you move forward with greater clarity.

Whether you are modernizing a platform, scaling your infrastructure, improving visibility, or looking for additional engineering capacity, DinaBridge can help you move forward with greater clarity.