TSDB Write Path

type fanout struct {
    logger     *slog.Logger
    primary    Storage    // local TSDB — error here aborts the write
    secondaries []Storage // remote write — errors logged but ignored
}

// fanoutAppender.Commit() order:
//  1. primary.Commit()   ← must succeed
//  2. secondary[i].Commit()  ← best-effort, logged on failure

The fanout decouples local durability from remote delivery. A remote write failure never drops a local sample.

type DB struct {
    dir    string
    locker *tsdbutil.DirLocker

    logger    *slog.Logger
    opts      *Options
    compactor Compactor

    mtx    sync.RWMutex
    blocks []*Block      // persisted, immutable blocks sorted by time

    head *Head           // mutable in-memory block

    compactc chan struct{} // signal to trigger compaction
    stopc    chan struct{}
    donec    chan struct{}

    autoCompact bool
    ...
}

// initAppender defers creation of the real headAppender until the first
// Append() call so that the mint/maxt are known.
type initAppender struct {
    app  storage.Appender    // nil until first append
    head *Head
    ...
}

func (a *initAppender) Append(ref storage.SeriesRef, lset labels.Labels,
    t int64, v float64) (storage.SeriesRef, error) {
    if a.app != nil {
        return a.app.Append(ref, lset, t, v)
    }
    // First append: create real headAppender
    a.app = a.head.appender()
    return a.app.Append(ref, lset, t, v)
}

The WAL provides durability. Every sample is persisted to disk before it enters the in-memory Head. On crash recovery Prometheus replays the WAL to rebuild the Head.

type WL struct {
    dir            string
    logger         *slog.Logger
    segmentSize    int    // default 128 MiB
    mtx            sync.RWMutex
    segment        *Segment  // active segment file
    donePages      int
    page           [pageSize]byte  // 32 KiB page buffer
    actorc         chan func()
    stopc          chan chan struct{}
    compress       CompressionType // snappy or zstd
    ...
}

Record Types

Segment Layout

data/wal/
├── 00000001    ← completed segment (128 MiB)
├── 00000002    ← completed segment
└── 00000003    ← active segment (being written)

data/wal/chunks_head/
├── 000001      ← m-mapped head chunks

type Head struct {
    chunkRange atomic.Int64   // maximum time range for a chunk (default 2h)
    numSeries  atomic.Uint64
    minTime, maxTime atomic.Int64

    wal, wbl *wlog.WL  // WAL and Write-Behind Log (OOO samples)

    exemplars ExemplarStorage

    // Hash-stripe sharded map: 512 stripes to reduce lock contention.
    series *stripeSeries

    // Pools to recycle slices without GC pressure.
    floatsPool          zeropool.Pool[[]record.RefSample]
    histogramsPool      zeropool.Pool[[]record.RefHistogramSample]
    ...
}

The stripeSeries structure is a 512-way hash-sharded map that maps HeadSeriesRef → *memSeries. This dramatically reduces lock contention under high write concurrency.

type memSeries struct {
    // Immutable after construction — no lock needed.
    ref       chunks.HeadSeriesRef
    shardHash uint64

    sync.Mutex  // guards everything below

    lset labels.Labels

    // mmappedChunks: completed chunks flushed to disk (memory-mapped).
    // Pointer arithmetic tracks firstChunkID to handle compaction shifts.
    mmappedChunks []*mmappedChunk
    firstChunkID  chunks.HeadChunkID

    // headChunks: linked list of in-memory chunks still being written.
    // headChunks → headChunks.prev → ... (most recent first)
    headChunks *memChunk

    ooo *memSeriesOOOFields  // out-of-order sample state
    ...
}

Chunk Lifecycle for a memSeries

1. First sample → allocate memChunk with XOR encoder; attach as headChunks
2. Samples appended to active headChunk via XOR encoding (~120 samples max)
3. Chunk full or time range exceeded → flush to chunks_head/ m-map file via chunkDiskMapper
4. Flushed chunk pointer stored in mmappedChunks; memory freed
5. Compaction moves mmappedChunks into a Block; firstChunkID advances

XOR Chunk Encoding

Prometheus uses the Gorilla XOR encoding for float samples, adapted from Facebook's paper:

Implementation: tsdb/chunkenc/xor.go

The headAppender collects samples in memory for a single scrape batch, writes WAL records, then appends to memSeries — all under a single lock lifecycle.

func (a *headAppender) Append(ref storage.SeriesRef,
    lset labels.Labels, t int64, v float64) (storage.SeriesRef, error) {

    // 1. Look up existing series by ref or by label hash.
    s := a.head.series.getByID(chunks.HeadSeriesRef(ref))
    if s == nil {
        // 2. New series: register it, get a new ref.
        var created bool
        s, created, err = a.head.getOrCreate(lset.Hash(), lset)
        if created {
            // 3. WAL SERIES record scheduled.
            a.series = append(a.series, record.RefSeries{
                Ref:    s.ref,
                Labels: lset,
            })
        }
    }

    // 4. Accumulate sample for batch WAL write.
    a.samples = append(a.samples, record.RefSample{
        Ref: s.ref, T: t, V: v,
    })
    a.sampleSeries = append(a.sampleSeries, s)
    return storage.SeriesRef(s.ref), nil
}

func (a *headAppender) Commit() error {
    // 5. Write WAL SERIES + SAMPLES records atomically.
    // 6. For each sample: s.append(t, v, ...) → XOR encode into headChunk.
    // 7. If chunk full: enqueue for m-map flush.
    ...
}

When the Head accumulates more than chunkRange * 3/2 (default 3h) of data, a compaction is triggered. The oldest portion of the head is written to a new immutable on-disk Block.

// Block directory layout after compaction:
data/
├── 01HJXXXXXXXXXXXXX/       ← ULID (time-sortable unique ID)
│   ├── chunks/
│   │   ├── 000001           ← raw XOR-compressed chunk data
│   │   └── 000002
│   ├── index                ← inverted index: label → posting list
│   ├── tombstones           ← delete intervals
│   └── meta.json            ← {ulid, minTime, maxTime, stats, compaction}
└── 01HJYYYYY.../

Block Merge (Level Compaction)

Since Prometheus 2.39, OOO samples (arriving with timestamps older than the current Head maxTime) are buffered in a separate Write-Behind Log (WBL) and the memSeriesOOOFields structure, then merged at compaction time.

type memSeriesOOOFields struct {
    oooMmappedChunks []*mmappedChunk  // flushed OOO chunks
    oooHeadChunk     *oooHeadChunk    // current in-memory OOO chunk
    firstOOOChunkID  chunks.HeadChunkID
}

// OOO write path:
//  headAppender.Append() → detects t < s.maxTime
//  → s.appendOOO(t, v)
//  → written to wbl (Write-Behind Log, separate WAL)
//  → OOO compaction merges into regular blocks