Elasticsearch stateless source map

Stateless data flow from write to object store to search

This static guide follows how the stateless plugin wires object-store durability, indexing commits, translog replication, search cache hydration, allocation, relocation, and cluster-state persistence. All code links are pinned to commit 4b8f87a3bdfbe8e1c8fc556911d605bbcd95bb77.

End-to-end path

1. Plugin startsCreates object-store, cache, commit, translog, recovery, and role-aware services.
2. Indexing node writesIndexEngine writes Lucene and appends serialized operations into a node translog buffer.
3. Remote durabilityTranslog files and batched compound commits are uploaded through ObjectStoreService.
4. Commit notificationSearch shards are told about the newest available commit and what BCC backs it.
5. Search shard refreshesSearchEngine updates SearchDirectory metadata, opens readers, and retains files needed by PIT/open readers.
6. Cache fills on demandBlobStoreCacheDirectory opens IndexInput through cache readers backed by shared blob cache and object store.
What makes the flow stateless

Shard data is treated as object-store state. Indexing nodes can write and upload Lucene commit content plus translog snapshots; search nodes read through cached object-store-backed Lucene directories. Allocation does not search for a previous local copy because local disk is not durable state.

StatelessExistingShardsAllocator: object store replaces previous local shard lookup StatelessAllocationDecider: shard role must match index/search node role
Main runtime boundary

The runtime split is in the engine factory. Promotable shards get IndexEngine, a replicated translog config, and commit tracking. Searchable non-promotable shards get SearchEngine, which consumes commit notifications and reads through cache-backed directories.

if (config.isPromotableToPrimary()) {
    ... return newHollowOrIndexEngine(indexSettings, newConfig);
} else {
    return new SearchEngine(...);
}
StatelessPlugin.getEngineFactory
Data flow summary

Indexing side

  1. The plugin creates ObjectStoreService, StatelessCommitService, TranslogReplicator, and HollowShardsService.
  2. IndexEngine receives operations and the engine factory injects a translog callback to TranslogReplicator.add(...).
  3. Flush/refresh commits are turned into StatelessCommitRef instances and appended into virtual batched compound commits.
  4. Uploads are pushed to the blob store and search nodes are notified only when the data is safe for search visibility.

Search side

  1. SearchEngine.onCommitNotification receives the latest commit and updates the known latest uploaded BCC.
  2. The prefetcher can warm missing BCC ranges into shared blob cache before reader refresh.
  3. SearchDirectory.updateCommit(...) swaps metadata so Lucene file opens resolve to blob ranges.
  4. BlobStoreCacheDirectory.openInput(...) returns an IndexInput backed by cache readers.
Component pages
default-open details mobile horizontal flow white-space: pre code commit-pinned links
  • Bootstrap explains plugin service creation and index/search engine selection.
  • Object Store explains repository-backed blob containers and upload APIs.
  • Write Path explains operation buffering, translog sync, flush, and recovery reads.
  • Commit Lifecycle explains VBCC creation, upload, notification, and retention.
  • Search Cache explains notification processing, prefetch, reader refresh, and cached IndexInput.
  • Allocation & Recovery explains role-aware placement, relocation, hollowing, and target activation.
  • Cluster State explains lease-backed election and blob-store cluster-state commits.
Generated from local source at commit 4b8f87a3bdfbe8e1c8fc556911d605bbcd95bb77.