Start practising

SYSTEM DESIGN · TECHNOLOGIES · 8 OF 12

Elasticsearch: Inverted Indexes, Shards and Near Real-Time Search

How Elasticsearch turns documents into an inverted index, what analyzers do, the refresh and translog write path behind near real-time search, primary and replica shards, query-then-fetch across shards, BM25 relevance, aggregations, keeping it in sync with your database, deep pagination, where it hurts and how to place it in an interview design.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
1 / 16

Press play to hear the narration, or step through at your own pace.

Notes

What it is

Elasticsearch is a distributed search and analytics engine built on Apache Lucene, a search library. You send it JSON documents over HTTP, it indexes every field, and you query with JSON. OpenSearch is an open-source fork started by AWS, now run under the Linux Foundation. With Logstash and Kibana it forms the well-known ELK stack for log search and dashboards.

Elasticsearch in one line
  • distributed search and
  • analytics engine
  • built on Apache Lucene
  • stores JSON documents in
  • indexes; REST + JSON queries
  • fork: OpenSearch (AWS)
  • part of the ELK stack
  • (Elasticsearch, Logstash,
  • Kibana) for logs

Inverted index

The core structure is the inverted index. Instead of storing which words each document contains, it stores, for every word, the list of documents that contain it, like the index at the back of a book. A search for red jacket looks up two short lists and combines them, instead of reading every document. That is why searching millions of documents takes milliseconds.

The inverted index
doc 1: "red running shoes"
doc 2: "blue running jacket"
doc 3: "red jacket"

term      → doc ids (postings)
red       → 1, 3
running   → 1, 2
jacket    → 2, 3

query "red jacket": 1 ∩ 3 → 3 best

Analyzers

Before indexing, text goes through an analyzer. It splits the text into tokens, lowercases them, reduces words to stems so running matches run, and can add synonyms. The same analyzer runs on the query, so both sides match. In the mapping, a text field is analyzed for search, while a keyword field is stored exactly for filters and sorting. Autocomplete usually uses edge n-grams, indexing r, ru, run and so on.

What an analyzer does
input:  "Running Shoes, Men's"

1. tokenize  Running | Shoes | Men's
2. lowercase running | shoes | men's
3. stem      run | shoe | men
4. synonyms  shoe → sneaker

mapping: "text" field = analyzed
         "keyword" field = exact
autocomplete: edge n-grams

Write path

Here is a write. The document id is hashed to pick a shard. The primary copy of that shard puts the document in an in-memory buffer and appends it to a transaction log for durability, then forwards it to replica shards. Every second, by default, a refresh turns the buffer into a new searchable segment. So search is near real-time: a new document appears about a second later. Segments are immutable and merged in the background.

Near real-time
  • 1. route: hash(_id) % shards
  • 2. primary shard writes to
  • an in-memory buffer +
  • translog (durability)
  • 3. copy to replica shards
  • 4. refresh (every 1 s):
  • buffer → new segment,
  • now searchable
  • segments are immutable;
  • background merges

Shards

An index is split into primary shards, here three, each with one replica on a different node, so any one node can fail. The number of primaries is set when you create the index; changing it means reindexing or a split operation. Replicas can be added any time for more read capacity. Each shard is a full Lucene index. A common guideline is ten to fifty gigabytes per shard, because too many tiny shards waste memory.

Primary and replica shards
  • index products: 3 primaries,
  • 1 replica each = 6 shards
  • primaries: fixed at creation
  • (change = reindex or split)
  • replicas: change any time,
  • add read capacity + safety
  • each shard = one Lucene
  • index; aim 10-50 GB each

Query path

Searching is scatter and gather. The coordinating node sends the query to one copy of every shard. Each shard scores its matches and returns only the top ten ids and scores. The coordinator merges them into a global top ten, then fetches just those ten documents. Two phases keep network traffic small. Your latency is set by the slowest shard, so one overloaded node slows every search.

Query then fetch
  • query phase: each shard
  • scores locally, returns top
  • N ids + scores
  • coordinator merges into a
  • global top N
  • fetch phase: get the full
  • documents for those N
  • latency = slowest shard

Relevance + aggregations

Results are ranked with BM25: a match on a rare word counts more than a common one, more matches score higher, and a match in a short field beats one in a long field. Boolean queries combine must, should and filter clauses; filters do not score and are cached, so put exact conditions there. Aggregations compute counts per brand or a price histogram, which is exactly the facet sidebar of a shop.

Scoring and analytics
  • BM25 score: rare terms and
  • more matches score higher;
  • long fields score lower
  • bool query: must, should,
  • filter (no score, cached)
  • boost: title^3, recency
  • aggregations: counts per
  • brand, price histogram
  • (the facets in a shop sidebar)

Keeping in sync

Elasticsearch should not be your source of truth. Keep data in the database and stream changes into the index: change data capture or an outbox publishes events to Kafka, and a worker writes them in bulk. Avoid having the app write to both, because one write will eventually fail and they drift. To change a mapping, build a new index from the database and switch an alias to it, with no downtime.

Sync from the database
  • never the only copy of data
  • CDC (Debezium) or outbox
  • → Kafka → indexer worker
  • → _bulk API
  • avoid dual writes from the
  • app: they drift on failure
  • rebuild: reindex from DB
  • into a new index, swap alias

Pagination

Pagination has a trap. With from and size, page one thousand of ten results forces every shard to return ten thousand hits for the coordinator to sort, so deep pages get slow, and the default cap is ten thousand. Instead, use search after: pass the sort values of the last hit you saw and get the next page cheaply, at any depth. Combine it with a point in time so results do not shift while you page.

Paging through results
  • from + size: page 1000 means
  • each shard returns 10,000
  • hits to merge; capped at
  • 10,000 by default
  • search_after: pass the sort
  • values of the last hit;
  • cheap at any depth
  • point in time (PIT): a stable
  • snapshot while you page

Great at

It shines at relevance-ranked text search with typo tolerance, stemming and synonyms, at autocomplete, at faceted filtering for shopping sites, and at searching and charting huge volumes of logs and events. It also does geo queries and vector nearest-neighbour search, so you can blend keyword and semantic ranking in one engine.

Where it shines
  • • full-text search with
  • typos, stems, synonyms
  • • autocomplete
  • • facets and filters on
  • e-commerce listings
  • • log and event search
  • (ELK, observability)
  • • geo queries, vector (kNN)
  • search, hybrid ranking

Where it hurts

Where it hurts. No transactions and no real joins. Data is searchable only after a refresh. The primary shard count is fixed, so growth means reindexing. Every update rewrites the whole document internally. Letting users create arbitrary field names explodes the mapping. Clusters need care with memory, garbage collection and shard counts. And it is not built to be the only copy of your data.

Pain points
  • • no transactions or joins
  • • ~1 s until searchable
  • • primaries fixed: reindex
  • • updates rewrite the doc
  • • mapping explosion from
  • arbitrary field names
  • • JVM heap, GC, shard count
  • • not a system of record:
  • keep the DB as truth

Use it when

Use Elasticsearch when users type free text and relevance ranking matters, when you need facets or autocomplete, or when you search logs at volume. Avoid it as a primary store, for exact lookups by id that any database does, and for transactional data. For a small catalogue, Postgres full-text search with a GIN index may be enough, and is one less system to run.

Use it when / avoid it when
  • USE when
  • users type free text
  • relevance ranking matters
  • facets, autocomplete
  • searching logs at volume
  • AVOID when
  • exact lookups by id (DB)
  • money, inventory (SQL)
  • small data: Postgres full-
  • text may be enough

Numbers

Numbers to remember. The refresh interval defaults to one second. Aim for shards of ten to fifty gigabytes. From plus size is capped at ten thousand. New indexes get one primary shard by default. Give the JVM heap about half the machine's memory but stay under about thirty-one gigabytes so pointers stay compressed. Typical searches take ten to a hundred milliseconds, and bulk requests of a few megabytes index fastest.

Numbers worth knowing
refresh interval        1 s (default)
shard size target       10-50 GB
max from + size         10,000
default primaries       1 (since 7.0)
heap                    ≤ ~31 GB, 50% RAM
search latency          10-100 ms typical
bulk request size       ~5-15 MB
replicas                1+ in production

Say it like this

How to say it. Products live in Postgres, the source of truth. Change data capture streams updates through Kafka to an indexer that bulk-writes to Elasticsearch. Title and description are analyzed text fields; brand and category are keywords for filters and facets. Search results are about a second behind writes, which is acceptable for a shop. That answer shows the index, the sync and the trade-off.

In the interview
  • "Products live in Postgres.
  • Debezium streams changes
  • through Kafka to an indexer
  • that bulk-writes to Elastic-
  • search. Title and description
  • are analyzed text, brand is
  • a keyword for facets. Search
  • is about a second behind,
  • which is fine for a shop."

Interview questions

What is an inverted index?

A map from each term to the list of documents that contain it, often with positions and frequencies. A query looks up the lists for its terms and combines them instead of scanning every document.

Why is Elasticsearch called near real-time?

New documents sit in an in-memory buffer and translog and only become searchable when a refresh creates a new segment, which happens every second by default.

How does a search query run across shards?

The coordinating node sends the query to one copy of every shard; each returns its top hits with scores, the coordinator merges them into the global top N and then fetches those documents. Latency depends on the slowest shard.

How do you keep Elasticsearch in sync with your primary database?

Treat the database as the source of truth and stream changes through change data capture or an outbox into a queue such as Kafka, with a worker that bulk-indexes them. Rebuild by reindexing into a new index and swapping an alias.

What is the difference between a text and a keyword field?

A text field is analyzed into terms for full-text search and relevance scoring; a keyword field is stored as one exact value for filtering, sorting and aggregations.

Why is deep pagination slow and what do you use instead?

With from and size each shard must return from + size hits for the coordinator to merge, so cost grows with page depth and is capped at 10,000 by default. search_after with a point in time pages cheaply by continuing from the last sort values.

How would you build autocomplete?

Index the field with an edge n-gram analyzer or use a completion suggester so prefixes match, keep the query analyzer standard, and rank by popularity; debounce requests on the client.

Can you change the number of primary shards?

Not in place. You use the split or shrink APIs, which create a new index with a multiple or factor of the shard count, or reindex into a new index with the desired count.