# Durability & Storage

> **Related:** [Access Control](https://wavehouse.dev/access-control.md) · [API Reference](https://wavehouse.dev/api.md) · [Architecture](https://wavehouse.dev/architecture.md) · [Claude Code & AI agents](https://wavehouse.dev/claude-code.md) · [Configuration](https://wavehouse.dev/configuration.md) · [Deployment](https://wavehouse.dev/deployment.md) · [Development](https://wavehouse.dev/development.md) · [Getting Started](https://wavehouse.dev/getting-started.md) · [Ingest Pipeline](https://wavehouse.dev/ingest-pipeline.md) · [Named Pipes](https://wavehouse.dev/pipes.md) · [Behind a reverse proxy](https://wavehouse.dev/reverse-proxy.md) · [TypeScript SDK](https://wavehouse.dev/sdk.md) · [Settings Directory](https://wavehouse.dev/settings-directory.md) · [Why WaveHouse?](https://wavehouse.dev/why-wavehouse.md)
> **Also:** [HTML version](https://wavehouse.dev/durability) · [Docs index](https://wavehouse.dev/llms.txt)

---
By default (`mq.backend: embedded`), WaveHouse buffers every ingested event in embedded NATS JetStream before the [ingest worker](/ingest-pipeline) drains it into ClickHouse. That buffer lives on disk at `<data_dir>/nats`, and **WaveHouse runs the embedded JetStream in its strictest durability mode**: every publish is `fsync`'d to non-volatile storage before the producer is acknowledged. An [external NATS cluster](#with-an-external-nats-cluster) is durable through its replicas instead, and WaveHouse does not require it to `fsync` every publish.

This is a deliberate, strong guarantee — but it makes your ingest latency a direct function of your storage's `fsync` latency. On managed cloud block storage that is effectively free; on some commodity or virtualized substrates the `fsync` tail balloons into seconds and ingest visibly suffers. This page explains the contract, where it is cheap versus expensive, and how to measure your storage before you trust it.

## The durability contract

The embedded server is started with `SyncAlways: true` (`internal/mq/embedded.go`). Concretely:

> When a client receives `200` from `POST /v1/ingest`, the event has already been `fsync`'d to disk on the WaveHouse node.

Ingestion is still [asynchronous](/architecture) end-to-end — the `200` means *durably buffered in JetStream*, not yet *written to ClickHouse* (the worker flushes to ClickHouse later, and the [worker's own ack](/ingest-pipeline#backpressure-and-durability-knobs) is what records "now in ClickHouse"). But the buffering step itself is hard-durable: an event that got a `200` survives an immediate, uncontrolled power loss on the node.

This is the strongest mode JetStream offers. It is stronger than the default, where a publish is acked once the write reaches the OS page cache and the data is flushed to disk later by a periodic background sync — fast, but a hard crash can lose the not-yet-flushed window.

| Mode | Ack means | Crash exposure | Throughput |
| --- | --- | --- | --- |
| **`SyncAlways` (the embedded broker)** | data is `fsync`'d to disk | none for acked events | bounded by `fsync` latency |
| Periodic group commit (default JetStream) | data is in the OS page cache | up to one sync interval of acked-but-unflushed events | bounded by memory/CPU |

WaveHouse does not currently expose a knob to relax this — `SyncAlways` is always on for the embedded broker. Exposing a configurable group-commit interval (`mq.sync_interval`) is tracked in [#139](https://github.com/Wave-RF/WaveHouse/issues/139).

## With an external NATS cluster

Under [`mq.backend: nats`](/deployment#external-nats) the buffer is your NATS cluster, not `<data_dir>/nats`, and the `200` means the partition stream has acked the publish:

- **At 3 or more replicas,** the stream acks only once a Raft quorum of its servers has stored the event. WaveHouse does not require `sync_always` there, and the shipped Helm values do not set it: a quorum spread across failure domains (zones, racks or hosts) survives losing any one of them, which is the durability WaveHouse relies on. What can lose an acked event is losing a quorum's servers at once, before they sync.
- **At one replica,** the server's `sync_interval` (2 minutes unless you set it) governs: this is the periodic group-commit row in the table above, and a crash of that server loses what it stored since its last sync. Boot reports a stream with fewer than 3 replicas as a `recommended` finding and still starts, so a one-server development cluster works. In production, run 3 replicas, or set `sync_always` on a one-server cluster and the rest of this page applies to its disk. A stream with `persist_mode: async` (one replica only) flushes in the background even under `sync_always`, so a crash of the server process alone can lose acked events; boot refuses it on an ingest partition.

WaveHouse does not choose the cluster's storage settings; the rest of this page describes the embedded server. What does not change is that no limit drops an event before it is written: the partition streams have no age limit, and a full one refuses new events with `503` rather than dropping old ones. A full partition refuses every tenant whose events it holds, not one tenant. The replay history is a separate stream whose `max_age` you set. It is best effort: the partitions republish each row to it, so a row republished while it is unavailable reaches ClickHouse but not SSE replay, and it never holds up ingest. Every tenant's parked rows share one dead-letter stream.

## Why the fsync tail is your ingest floor

Because the publish blocks on `fsync`, **your typical ingest latency is your storage's typical `fsync` latency, and your worst-case publish is your storage's worst-case `fsync`.** When that tail is healthy (sub-millisecond to single-digit milliseconds) the guarantee is essentially free. When it is not, the same code path that handles every production message stalls:

- Publishes block for the duration of the `fsync`, so a multi-second `fsync` tail is a multi-second ingest tail.
- The embedded server's consumer setup at boot and every publish run under the JetStream client's request timeout, and opening or resizing a tenant's queue — and joining the consumers to one that opens while the server runs — under ten-second budgets of WaveHouse's own; a slow-enough substrate makes them exceed it. The symptom when a tenant's queue first opens — at the boot or reload that first serves the tenant — is `open dlq stream: ... context deadline exceeded`, or `open ingest stream: ...` (the two share the budget); a boot that finds every queue already at its budget writes nothing, so there the first publish is where it shows.
- If the worker cannot drain to ClickHouse faster than producers publish, a tenant's stream fills toward its [`mq.max_bytes_gb`](/settings-directory#message-queue) and the API returns `503` to that tenant ([backpressure by construction](/ingest-pipeline#backpressure-and-durability-knobs)).

## Where `SyncAlways` is cheap vs. expensive

The strict guarantee translates well to managed cloud infrastructure — the presumed production target — and to enterprise-grade local disks. It is the commodity and virtualized substrates that bite.

**Healthy — `SyncAlways` is effectively free:**

| Substrate | Why |
| --- | --- |
| Cloud block storage (gp3/io2 EBS, GCP pd-ssd, Azure Premium SSD) | Battery-backed cache acks sync writes from non-volatile DRAM, not NAND. Sub-millisecond at typical load. |
| Enterprise NVMe with power-loss protection (Optane, Samsung PM-series, Solidigm D7) | The PLP capacitor lets the controller ack a sync write from DRAM — the `fsync` ≈ `memcpy`. |
| Local `ext4` on consumer NVMe | Single-device journal commit, 1–10 ms typical. Tail spikes under heavy concurrent dirty data, but bounded. |

**Problematic — measure before you trust it:**

| Substrate | Failure mode |
| --- | --- |
| ZFS without a SLOG, consumer NVMe | Every sync write hits the ZIL, gated by transaction-group commit cadence that serializes across all pool consumers. Single-digit-ms idle, **5–25 s under concurrent load**. |
| Loopback / qcow2 on `ext4` inside a VM | Stacks a second journaling layer; often 10× slower than direct `ext4` and highly variable. |
| Spinning disks | Mechanical seek on the NAND-equivalent program path: multi-millisecond baseline, multi-second tail. |

The tell for a commit-cadence problem (ZFS-without-SLOG, noisy-neighbor VM host) is that a single-threaded benchmark looks fine while a concurrent one is far worse — so always benchmark with multiple writers, and benchmark the guest **and** the host if virtualized.

## Deduplication: one more fsync per window

With [deduplication](/settings-directory#deduplication) on, a `200` also means the records' ids were committed to the dedupe store, or, if that commit failed, that the failure was counted by `wavehouse_ingest_dedupe_commit_failed_total` and the ids lapse with their lease. On the embedded Pebble store that commit is an `fsync` of its own. It is taken once per window of up to 256 records of a request, after the window's publishes, rather than once per record: a 1,000-record batch costs four dedupe syncs, not a thousand. Measured with `BenchmarkIngest_DedupBatchOnPebble` on a developer laptop, with the queue stubbed out so only the dedupe store touched disk, the dedupe work for that batch took 24 ms windowed against 5.7 s one record at a time; the JetStream publishes' own fsyncs come on top. A single-record request still pays one sync for its publish and one for its commit.

With a finite `dedupe.retention` and `dedupe.backend: pebble`, expired ids are deleted by a background sweep, an hour apart; on DynamoDB the table's TTL deletes them instead ([Deployment](/deployment#a-shared-dedupe-table-on-dynamodb)). Its deletes are not fsynced (a delete lost to a crash is redone by the next pass), so it adds no sync to the ingest path. It reads 1,024 keys at a time without holding up commits, then re-reads the expired ones and deletes those still expired; a commit waits only for that last step, at most 1,024 point reads and one unsynced write, however many deleted keys the read stepped over. An expired id is already treated as new by the next claim of it, sweep or no sweep, so retention never depends on the sweep having run.

A publish can also fail after JetStream stored the event (a timeout on the ack). The record's id is then left to lapse with its dedupe lease ([`dedupe.lease`](/configuration#dedupe), 30 seconds by default) rather than given back, and every deduped record is published under an idempotency key derived from its tenant, table and id, which each tenant's ingest stream remembers for two minutes after the first publish. A retry after the lease but inside those two minutes is therefore dropped by the stream rather than stored twice; one later than that is stored again. The duplicate window has to cover more than the lease alone: the `503` for an uncertain publish sends the *full* lease as `Retry-After`, so an obedient client's retry can land up to ~2×lease after the original request, and a claim's expiry can itself round up by a further second on some backends — the invariant is `lease + ceil(lease) + 1s ≤ window` (`ceil` rounding up to the whole second, so `2×lease + 1s` for a whole-second lease), not just `lease ≤ window`, and boot refuses a `dedupe.lease` that breaks it with the embedded queue, so at most 59 seconds. Two minutes against the default 30-second lease clears that with room to spare. For the same reason a finite `dedupe.retention` must be at least those two minutes: an id re-sent after a shorter retention ended would be claimed again, then dropped by the stream as a copy while the client was told it was accepted. Settings validation refuses one below it.

## Check your storage before you trust it

Replicate JetStream's exact pattern — a 4 KiB write followed by a flush, in a tight loop — and report the percentiles. The numbers that matter are **p99** and **max**: those are your worst-case publish latency.

On Linux, [`fio`](https://fio.readthedocs.io/) (packaged on every distro) is the honest, standard tool. Point it at the volume that backs `<data_dir>/nats`, ideally before WaveHouse is running:

```bash
# 8 concurrent writers — the variant that surfaces commit-cadence problems
fio --name=jetstream-fsync --directory=/var/lib/wavehouse/nats \
    --rw=write --bs=4k --size=64M --fsync=1 --runtime=30 --time_based \
    --numjobs=8 --group_reporting
```

Run it under representative load, not on an idle box — idle benchmarks understate real-world tails.

Read the measured p99 against these bands, which track WaveHouse's `SyncAlways` default:

| p99 `fsync` | Verdict for `SyncAlways: true` |
| ---: | --- |
| < 1 ms | **Ideal** |
| 1–5 ms | **Good** |
| 5–50 ms | **Workable** — watch bursty load |
| 50 ms – 1 s | **Marginal** — relax durability once `mq.sync_interval` ([#139](https://github.com/Wave-RF/WaveHouse/issues/139)) lands, or move to faster storage |
| > 1 s | **Broken** — opening a tenant's queue (`open dlq stream` / `open ingest stream`) will time out under load; fix the storage substrate |

:::caution[macOS `fsync` lies by default]
A plain `fsync()` on macOS returns once data is in the drive's volatile cache — it does **not** force a flush to NAND; only `fcntl(fd, F_FULLFSYNC)` does (NATS, Postgres, and SQLite all use it). On a Mac, any per-flush number under ~1 ms is almost certainly not a real flush — the gap between plain `fsync()` and `F_FULLFSYNC` can be ~180× on the same consumer NVMe. `fio` on macOS calls plain `fsync()`, so don't trust Mac `fio` numbers for tail-latency planning. This mostly matters when benchmarking a dev machine; production WaveHouse runs on Linux, where `fio` is honest.
:::

A self-contained `wavehouse storage-check` preflight subcommand that bakes this measurement and verdict into the binary — including the per-platform honest flush — is tracked in [#84](https://github.com/Wave-RF/WaveHouse/issues/84).

## Symptoms of storage that can't keep up

If you see any of these, benchmark the `<data_dir>/nats` volume as above:

- `open dlq stream: ... context deadline exceeded`, or `open ingest stream: ...`, when a tenant's queue first opens, at the boot or reload that first serves the tenant.
- `mq queue not reconciled with settings; the next reload retries` with `join the queue: ... context deadline exceeded`, and that tenant's ingest answering `503`, when a tenant's queue opens while the server runs and the ingest worker's or the stream hub's consumer cannot join it in time.
- Ingest p99 latency in the seconds, or occasional `200`s that take multiple seconds to return.
- Intermittent `503 Service Unavailable` from `/v1/ingest` when ClickHouse is healthy (the worker can't drain fast enough because acking is `fsync`-bound).
- Flaky CI or load tests that pass on fast storage and fail on a shared/virtualized host.

## See also

- [Settings Directory → Message Queue](/settings-directory#message-queue) — `mq.max_bytes_gb`, each tenant's queue's disk budget (hot-reloadable); the SSE gap window inside it is [`stream.gap_window_minutes`](/settings-directory#streaming).
- [Deployment → Persistent Storage](/deployment#persistent-storage-required-for-containers) — `data_dir` must resolve to a host-backed volume.
- [Ingest Pipeline → Backpressure and durability knobs](/ingest-pipeline#backpressure-and-durability-knobs) — the worker-side ack cost and the in-flight backpressure layers.