# Settings Directory > **Related:** [Access Control](https://wavehouse.dev/access-control.md) · [API Reference](https://wavehouse.dev/api.md) · [Architecture](https://wavehouse.dev/architecture.md) · [Claude Code & AI agents](https://wavehouse.dev/claude-code.md) · [Configuration](https://wavehouse.dev/configuration.md) · [Deployment](https://wavehouse.dev/deployment.md) · [Development](https://wavehouse.dev/development.md) · [Durability & Storage](https://wavehouse.dev/durability.md) · [Getting Started](https://wavehouse.dev/getting-started.md) · [Ingest Pipeline](https://wavehouse.dev/ingest-pipeline.md) · [Named Pipes](https://wavehouse.dev/pipes.md) · [Behind a reverse proxy](https://wavehouse.dev/reverse-proxy.md) · [TypeScript SDK](https://wavehouse.dev/sdk.md) · [Why WaveHouse?](https://wavehouse.dev/why-wavehouse.md) > **Also:** [HTML version](https://wavehouse.dev/settings-directory) · [Docs index](https://wavehouse.dev/llms.txt) --- Boot config — the YAML file and `WH_*` environment variables on the [Configuration](/configuration) page — is what only a restart can change: resource sizing, listeners, exporters, and the platform's own secrets. This page covers everything else: the **settings directory**, the behavioral configuration a running instance re-reads without restarting — the access-control policy and its role registry, the named pipes, and the tunables including the ClickHouse and auth wiring. The settings directory holds WaveHouse's file-based settings as exactly four JSON documents: [`roles.json`](#rolesjson), [`policies.json`](#policiesjson), [`pipes.json`](#pipesjson), and [`config.json`](#configjson-keys). The files are the only write path — standalone, you edit them on the host; on WaveHouse Cloud the control plane writes them — and there is no API that writes back to them. Every file must exist (an empty document is `{}` — a missing file always means deletion or a wrong path, never "defaults"), and any other entry in the directory is an error, so a typoed filename or a stray backup fails loudly instead of being silently ignored. Dot-prefixed entries are the one exception: editor swap files and the `..data` machinery Kubernetes ConfigMap mounts publish through are ignored. Create one with `wavehouse bootstrap [dir]`: it writes all four files with every key at its default and refuses a non-empty directory, so an existing settings directory is never overwritten. The binary carries no compiled defaults but one — a missing `dedupe.retention` means `"0"`, forever — so the seed is where the defaults live, and what the server adopts is exactly what the files say. The seed ships no policy (`policies.json` is `{}`, `roles.json` and `pipes.json` are empty lists), so a freshly bootstrapped directory boots fail-closed — every request is denied until you write a policy. The container images ship no settings directory: they preset `WH_SETTINGS_DIR=/app/settings` and expect a bind mount there — a host directory you wrote with `bootstrap` (the reference compose file mounts the checked-in `deployments/compose/settings/`, the seed with `clickhouse.addr` pointed at the `clickhouse` service and a permissive `public` trial policy in `policies.json` / `roles.json`). A bind mount, not a named volume: the images are distroless, with no shell to edit files inside a volume. A missing mount refuses to boot rather than running on defaults nobody chose. Check a directory with `wavehouse validate [dir]`. Both commands resolve the directory the same way — the argument, falling back to `WH_SETTINGS_DIR`, and a usage error (exit `2`) with neither — so the path you seed is the path you validate, and inside the container images (which preset `WH_SETTINGS_DIR=/app/settings`) both work with no argument at all. `validate` validates without starting the server (JSON syntax including unknown fields and duplicate keys, per-file shape rules including the required keys, and cross-file role references), prints every finding in one pass, and exits `0` for valid (warnings allowed), `1` for invalid, `2` for usage — so operators and CI can gate a settings change before it reaches a running instance. Where the directory lives is the one boot-config key involved: `settings.dir` / `WH_SETTINGS_DIR`, documented in [Configuration — Settings Directory](/configuration#settings-directory). It's required with no default; the container images preset it to `/app/settings`. ## Loading and hot reload The server validates and adopts the directory at boot — a missing or invalid directory refuses to start, so a typo, a missing mount, or an invalid policy surfaces immediately instead of silently denying every request (run `wavehouse validate` to reproduce the findings, `wavehouse bootstrap` to create a starter directory). A [nested directory](/deployment#the-nested-settings-directory) that would serve no tenant — every folder rejected, or none named by a tenant id — refuses to start the same way. Boot config has no equivalent command and needs none: it only takes effect through a restart, and the restart refuses on an undeclared YAML key, an unbound `WH_*` variable, or an unusable `data_dir` — see [Configuration — Loading Order](/configuration#loading-order). A running instance re-validates and re-adopts on any of three triggers, all funneling through one serialized reload path: - **File change** — the directory is watched; a burst of writes (an editor save, a Kubernetes Secret republish — a Secret rather than a ConfigMap, since `config.json` carries the tenant's secrets) coalesces into one reload, and a directory that is deleted and recreated (or renamed over) is picked up again. - **`SIGHUP`** — `kill -HUP `. - **`POST /v1/ops/settings/reload`** (admin-only) — returns `{"adopted": bool, "findings": [...]}`; `200` when adopted, `422` when rejected. A reload that fails validation is logged (and reported by the endpoint) and the previous good settings stay in effect — an operator mid-edit, a deleted file, even a vanished directory degrades to a log line, never to a broken server. Warnings don't block adoption, matching `wavehouse validate`. "Previous good settings" is the in-memory snapshot of the running process, nothing more: there is no persisted copy of the files. A restart re-validates the directory from scratch and refuses to start on the same findings the reload rejected, so bad files never survive a restart silently — fix them (or run `wavehouse validate`) before bouncing the server. A directory that holds one folder per tenant instead of the four files is [a nested settings directory](/deployment#the-nested-settings-directory): each folder is everything this page describes, but it is not watched, a rejected folder stops its tenant being served rather than keeping the previous settings, and the keys the whole process shares are not read from the tenant's own folder: they come from tenant `0`'s — that section lists which keys. Every adoption — boot and every reload — goes through the same `Validate`, so the policy, the roles, and the pipes are checked with the current rules each time they are read; there is no stored copy that can skip validation. All four files are adopted as one snapshot: a request is evaluated against the policy, pipes, and tunables of a single adoption, never a mix. Every file is decoded strictly: an unknown key, a duplicate key, a UTF-8 byte order mark, an all-whitespace file, or a top-level `null` is an error, so a misspelled rule can never load as "no rule". ## `roles.json` The role registry — every role a policy grant or a pipe allowlist may name: ```json { "roles": ["admin", "analyst", "public"] } ``` Rules: each name must be non-empty, without surrounding whitespace, and unique. Cross-file, every role referenced elsewhere must be declared here: `policies.json`'s `default_role` and `admin_role`, every role keyed under a table's grants, and every entry in a pipe's `allowed_roles`. An undeclared reference is an error, so a typo in a grant fails validation instead of silently matching nobody. Does the admin role belong in this list? Only if `policies.json` names it: setting `admin_role` (or using the admin role as `default_role`) requires declaring that role here, like any other reference. Leaving `admin_role` unset — the built-in default `admin` — requires nothing; the reference compose seed's `roles.json` is just `["public"]` and validates clean for exactly this reason. Naming the admin role inside a table grant or a pipe's `allowed_roles` never requires a declaration either — admin is an [unconditional bypass](/access-control#admin_role--the-privileged-role), so such an entry is dead config and only warns. ## `policies.json` The access-control policy — one [policy document](/access-control#anatomy-of-a-policy) (`default_role`, `admin_role`, `tables`): ```json { "default_role": "public", "admin_role": "admin", "tables": { "events": { "public": { "select": { "allow_columns": ["*"] }, "insert": { "allow_columns": ["*"] } } } } } ``` An empty document (`{}`) means **no policy**: it validates with a warning, and the server adopts it fail-closed — every request is denied, including one carrying the admin role; only the [operator key](/access-control#operator-key) gets through. A non-empty document must pass the full [policy validation](/access-control#anatomy-of-a-policy) (column and row rules, the claim-template grammar, resource limits), and its roles must be declared in `roles.json`. The layout is **role-first** — `tables...select` — and a document still using the pre-v2 operation-first nesting is rejected, with the message linking straight to [the migration note](/access-control#migrating-from-the-operation-first-layout). A role named after an operation is the shape the two layouts most visibly collide in, and it gets its own message rather than the migration one: `tables.
.select.select` grants the same access under either reading and is accepted, but `tables.
.select.insert` means a read-only role named `insert` under the old layout and a write-only role named `select` under the new one — so validation refuses to guess and asks you to rename the role. Three states warn rather than fail: `default_role` equal to the admin role (every roleless request is admin — dev only), a grant keyed by the admin role (admin is an unconditional bypass, so the grant has no effect), and a grant that sets neither `select` nor `insert` (the role gets no access to the table — most often a half-finished migration). A leftover pre-v2 `"select": {}` block decodes into a grant keyed by a role literally named `select`. Whether you see this warning or an error then depends on `roles.json`: if `select` is not declared there — the usual case — the undeclared-role error fires first and you never reach the warning; if it *is* declared, you get the warning and the document adopts. A reload applies to the very next request: the policy is read per request off the adopted snapshot, and a live `GET /v1/stream` connection's role is re-evaluated against the new policy on the next event. ## `pipes.json` The [named pipes](/pipes): ```json { "pipes": [ { "name": "top_pages", "description": "Most-visited pages over a window", "sql": "SELECT page, count() AS hits FROM events WHERE ts > now() - INTERVAL {{hours:24}} HOUR GROUP BY page ORDER BY hits DESC LIMIT {{limit}}", "parameters": [ { "name": "hours", "type": "number", "default": 24 }, { "name": "limit", "type": "number", "required": true } ], "allowed_roles": ["analyst"] } ] } ``` Each pipe carries `name`, `sql`, optional `description`, optional `parameters` (`name`, `type`, `required`, `default`), and `allowed_roles`. Rules: names unique and non-empty; `sql` non-empty; parameter names non-empty and unique within the pipe; `type` one of `string`, `number`, `boolean`, `array` (or omitted); every `allowed_roles` entry non-empty and declared in `roles.json`. A `default` on a `required` parameter warns (it is never used), and listing the admin role in `allowed_roles` warns (admin can always execute). A pipe with no `allowed_roles` authorizes nobody but the admin role. Table and column existence are not checked — that is schema discovery's job at execution time. Pipes are read per request, so a reload changes what the next `GET /v1/pipes/{name}` executes and what `GET /v1/ops/pipes` lists. ## `config.json` keys The tenant tunables. Every key is required (a missing one is a validation error) except `dedupe.retention` and the per-table overrides; the "Seed" column is what `wavehouse bootstrap` writes: | Key | Seed | Description | | --- | ---- | ----------- | | `clickhouse.addr` | `localhost:9000` | Native-protocol `host:port` — schema discovery, structured queries, pipes, `/readyz`. See [ClickHouse](#clickhouse). | | `clickhouse.http_port` | `8123` | HTTP interface port on the same host (ingest `INSERT`s and the raw-SQL proxy), `1–65535`. | | `clickhouse.http_scheme` | `http` | `http` or `https` for that HTTP hop — one of the two *outbound* TLS switches, with `tls.enabled` for the native hop; unrelated to your clients' TLS. | | `clickhouse.database` | `default` | Database tables are discovered from. | | `clickhouse.username` | `default` | Connection user. | | `clickhouse.password` | `""` | The user's password; empty for a passwordless user. A tenant's own secret, kept here with the user it pairs with: a change is a new connection tuple, so a reload moves the tenant to a pool on the new password (see [ClickHouse](#clickhouse)). | | `clickhouse.query_timeout` | `30` | Seconds (`>= 1`) a ClickHouse call may take on the query paths — structured queries, pipes (a write pipe included) and `/v1/ops/query`. On `/v1/query` under a role's `max_execution_time`, the smaller of the two is sent to ClickHouse as `max_execution_time`; otherwise, on the native paths, it bounds the client deadline, from which the driver derives a server-side `max_execution_time`; on `/v1/ops/query` it is the HTTP request's deadline. | | `clickhouse.tls.enabled` | `false` | Switches the native-protocol hop (`addr`) to TLS. The HTTP hop's switch stays `http_scheme`; the rest of the `tls` block applies to whichever hop uses TLS. See [ClickHouse](#clickhouse). | | `clickhouse.tls.ca_file` | `""` | PEM bundle the server certificate is verified against; empty uses the system roots. A path, read when the connection is built and re-read when the `tls` block changes — validation does not open it. | | `clickhouse.tls.cert_file` | `""` | Client certificate for mutual TLS, PEM; set together with `key_file` or not at all. | | `clickhouse.tls.key_file` | `""` | The client certificate's private key, PEM. | | `clickhouse.tls.insecure_skip_verify` | `false` | Skips server certificate verification. `true` validates with a warning: both hops then accept any certificate. | | `clickhouse.tls.server_name` | `""` | Name the server certificate is verified against; empty derives it from the host in `addr` (native) or the URL (HTTP). | | `clickhouse.headers` | `{}` | Extra request headers for the HTTP interface (ingest `INSERT`s, `POST /v1/ops/query`). WaveHouse's own `Content-Type` and credential headers are set after them and win; naming `X-ClickHouse-User`, `X-ClickHouse-Key` or `Authorization` is a validation error, and so are two spellings of one name (names are case-insensitive). See [ClickHouse](#clickhouse). | | `clickhouse.max_open_conns` | `10` | Native connection pool size (`>= max_idle_conns`); tenants sharing a pool size it to the largest ask among them. The open pools together must not exceed the boot config's [`clickhouse.max_total_conns`](/configuration#clickhouse) when one is set. See [ClickHouse](#clickhouse). | | `clickhouse.max_idle_conns` | `5` | Idle native connections kept open (`>= 1`). | | `auth.jwks_url` | `""` | JWKS endpoint (absolute `http(s)` URL). When set, JWKS is the **sole** verifier and `jwt_secret` is ignored (validation warns when both are set). See [Authentication](#authentication). | | `auth.jwt_secret` | `""` | HMAC secret (`HS256`/`HS384`/`HS512`) tokens are verified with while `jwks_url` is empty. A tenant's own secret, kept here with the verifier wiring: a change rebuilds the verifier on reload, and tokens signed with the previous secret stop validating at once. Empty with `jwks_url` empty validates no token — every request is the policy `default_role` — which boot warns about per tenant. See [Authentication](#authentication). | | `auth.role_claim` | `role` | Dot-separated JWT claim path the role is read from (e.g. `app_metadata.role`). | | `dedupe.enabled` | `false` | Turn deduplication on; a reload opens or closes this tenant's store — see [Deduplication](#deduplication). | | `dedupe.id_field` | `event_id` | Dedup key field — see [Deduplication](#deduplication). | | `dedupe.require_id` | `false` | Reject rows missing the id field — see [Deduplication](#deduplication). | | `dedupe.retention` | `"0"` | How long a committed id stays a duplicate, as a duration (`"720h"`); `"0"`, or leaving the key out, keeps it forever — see [Deduplication](#deduplication). | | `dedupe.tables.
.{id_field, require_id, retention}` | `{}` | Optional per-table overrides; each entry overrides only the fields it names and inherits the rest. | | `dlq.enabled` | `true` | Park poison rows — those ClickHouse still rejects after row-by-row isolation, and every row of a batch whose tenant has no ClickHouse connection — on the tenant's dead-letter stream (`DLQ_{tenant}`) (`false`: leave them unacked for redelivery — except an envelope the worker cannot read, which is dropped and counted) — see [Dead Letter Queue](#dead-letter-queue). | | `dlq.tables.
.enabled` | `{}` | Optional per-table override of the switch. | | `query.timestamp_bucket_seconds` | `60` | Bucket (seconds, `>= 0`) that a structured query's relative time range is truncated to, so near-identical queries share a cache entry; `0` disables bucketing. Read per query. | | `query.default_max_rows` | `10000` | Fallback result `LIMIT` (`>= 1`) applied to a structured query when the caller and policy specify none. A result-**shaping** default, not a resource limit — server-wide limits (memory, rows scanned, execution time) belong in ClickHouse, see [Server-side resource limits](/configuration#server-side-resource-limits). | | `schema.refresh_interval` | `60` | Seconds between ClickHouse table-schema re-discoveries (`>= 1`), per tenant; a reloaded value takes effect from the next refresh cycle, and the first periodic refresh lands at a random point within the interval. Schemas are also refreshable on demand via `POST /v1/ops/schema/refresh` (admin-only). | | `stream.keepalive_interval` | `30` | Seconds (`>= 1`) a quiet `GET /v1/stream` connection may go without a write before the server sends a `:` keepalive comment — keep it under your proxy's idle timeout; see [Streaming](#streaming). | | `stream.keepalive_buckets` | `3` | Load-spreading (`1` to `100`): connections are spread across N buckets so each tick nudges ~1/N of live streams. Most deployments leave it. | | `stream.gap_window_minutes` | `15` | Minutes (`>= 0`) of written-to-ClickHouse history the Active Sweeper keeps in NATS for `Last-Event-ID` gap-fill; applies from the next sweep. Under [`mq.backend: nats`](/deployment#external-nats) the history stream's `max_age` decides instead, and the API processes check at boot and after each reload, warning once per tenant and window about a tenant whose window is longer. | | `mq.max_bytes_gb` | `50` | Disk budget (GB, `>= 1`) for the tenant's embedded NATS ingest stream (`INGEST_{tenant}`); its dead-letter stream (`DLQ_{tenant}`) gets a tenth of it. A reload updates the live streams in place. See [Message Queue](#message-queue). | | `cors.allowed_origins` | `["*"]` | Allowed CORS origins, applied per request. `"*"` allows any browser origin. WaveHouse is a Bearer-token API — `Access-Control-Allow-Credentials` is intentionally never sent, so this allowlist controls *which origins can read responses*, not cookie scope. Tighten to your frontend's exact origin(s) in production (e.g. `["https://dashboard.example.com", "http://localhost:3000"]`). An empty list `[]` denies every browser origin (no `Access-Control-Allow-Origin` is ever sent); `"*"` is the only allow-all spelling. Over [a nested settings directory](/deployment#the-nested-settings-directory) each tenant's list decorates its own responses, the preflight included; which list answers a preflight, the tenant-exempt routes, and a refused request is [spelled out there](/deployment#multi-tenant-deployments). | ```json { "clickhouse": { "addr": "clickhouse.internal:9440", "http_port": 8443, "http_scheme": "https", "database": "analytics", "username": "wavehouse", "password": "s3cret", "query_timeout": 30, "tls": { "enabled": true, "ca_file": "/etc/wavehouse/clickhouse-ca.pem", "cert_file": "", "key_file": "", "insecure_skip_verify": false, "server_name": "" }, "headers": {}, "max_open_conns": 10, "max_idle_conns": 5 }, "auth": { "jwks_url": "https://auth.example.com/.well-known/jwks.json", "jwt_secret": "", "role_claim": "app_metadata.role" }, "dedupe": { "enabled": false, "id_field": "event_id", "require_id": false, "retention": "720h", "tables": { "clicks": { "id_field": "click_id", "retention": "24h" } } }, "dlq": { "enabled": true, "tables": { "audit_log": { "enabled": false } } }, "query": { "default_max_rows": 10000, "timestamp_bucket_seconds": 60 }, "schema": { "refresh_interval": 60 }, "stream": { "keepalive_interval": 30, "keepalive_buckets": 3, "gap_window_minutes": 15 }, "mq": { "max_bytes_gb": 50 }, "cors": { "allowed_origins": ["*"] } } ``` What stays in boot config is only what cannot change under a running process — the implementation each layer runs on (`mq.backend`, `cache.backend`, `dedupe.backend`, `coord.backend`) and a shared backend's connection (`mq.nats`, `coord.nats`, `cache.redis`, `dedupe.dynamodb`), how a dedupe claim behaves (`dedupe.lease`, `dedupe.reserve_concurrency`), the process's `roles`, resource sizing (`data_dir`, `cache.l1_max_cost`, `clickhouse.max_total_conns`), the listeners, the observability exporters — and the **platform's own secrets**: `auth.operator_key` and `cache.redis.password`, which stay in the environment and need a restart to rotate. A tenant's secrets — `clickhouse.password`, `auth.jwt_secret` — are keys here, beside the user and the verifier they belong to, so they reload with them; validation warns, at boot and on every reload, when `config.json` carries one of them but is world-readable (see [Deployment — Persistent Storage](/deployment#persistent-storage-required-for-containers)). See [Configuration](/configuration). Everything else lives here and reloads. ## Deduplication Every per-tenant dedupe knob lives here. Where the seen ids are kept (`dedupe.backend`) and how long a claim is held (`dedupe.lease`) are [boot config](/configuration#dedupe), the same for every tenant. The switch and its fields are resolved per record from one snapshot (table override → global value): - `dedupe.enabled` (seed default `false`) — turns deduplication on. Hot-reloadable: a reload that flips it opens or closes this tenant's store (in the embedded Pebble instance at `/pebble`, or its share of the DynamoDB table under `dedupe.backend: dynamodb`), so no restart is needed; seen ids persist across an off/on cycle. If the store fails to open on a reload, the failure is logged and ingest fails closed (`503 dedupe store unavailable`, `Retry-After: 5`) until it opens — the files asked for dedupe, so publishing un-deduped is not a fallback. With `dedupe.backend: pebble` that is the next reload or restart, and at boot a failed open over a flat directory refuses to start, like every other store; with `dynamodb` it is the background retry described below. A record that lands in the instant of the flip itself is published un-deduped: if the settings already say on but the store is not yet open, it's counted by `wavehouse_ingest_dedupe_disabled_total`; in the reverse case (settings already say off, store still open) the handler skips dedupe like any other disabled record and nothing is counted. That counter should only ever tick during a reload, so a steadily climbing rate means the store and the settings have come apart. Over [a nested directory](/deployment#the-nested-settings-directory) with `dedupe.backend: pebble`, every tenant's seen ids live in that one instance, each key led by its tenant and table, and it is open while any tenant's switch is on: each tenant's store follows its own folder's `dedupe.enabled` the same way; a tenant's seen ids are never another's; a rejected or removed folder closes its tenant's store and keeps its seen ids for the folder that restores it; and if that instance fails to open, at boot or on reload, every tenant with dedupe on fails closed — its ingest answers `503 dedupe store unavailable` (`Retry-After: 5`) until a reload opens it — while the tenants with dedupe off carry on. Under `dedupe.backend: dynamodb` the table check plays the instance's part, in either shape: the table is checked whether or not any tenant's switch is on, and a table that fails it fails every tenant with dedupe on closed until the check, retried in the background and at once after every reload, passes. Only a misconfigured table (missing, the wrong key schema, access denied) over a flat directory whose tenant has dedupe on refuses boot instead ([Configuration](/configuration#dynamodb-dedupe)). - `dedupe.id_field` (seed default `event_id`) — JSON field name in the ingest body used as the dedup key. An id is a duplicate only within its own tenant and table: the same value in two tables is two ids. An id longer than 1,024 bytes once escaped (every byte but an ASCII letter, digit, `_` or `-` takes three) is stored as its SHA-256, counted by `wavehouse_dedupe_hashed_id_total`. While its record is being published, an id is held for its lease ([`dedupe.lease`](/configuration#dedupe), 30 seconds by default): another request carrying the same id meanwhile gets `503` (`a request with the same dedupe id is in flight`) with the lease, in whole seconds, as `Retry-After` — see [the ingest errors](/api#post-v1ingesttabletable--ingest-data). An id is committed only after its record is published; if that commit fails (counted by `wavehouse_ingest_dedupe_commit_failed_total`, which should stay at zero), the record is still answered `ok` and the id lapses with its lease: a retry of it before then answers in-flight, one inside the ingest queue's two-minute duplicate window is dropped there by its idempotency key, and one after that is stored again. - `dedupe.require_id` (seed default `false`) — controls what happens to a row missing `id_field`, or carrying it as `null` (which can't be deduped, so idempotency wouldn't apply to it). Such a row is always logged at `WARN` and counted by `wavehouse_ingest_dedupe_missing_id_total`, in both modes. `false`: it is then published un-deduped. `true` rejects it instead (`400` for a single insert; a per-record failure in a batch) — a server-side tripwire for producers that must guarantee the id. - `dedupe.retention` (optional; seed default `"0"`) — how long a committed id stays a duplicate, as a Go duration string: `"24h"`, `"720h"` (30 days), `"90m"`. There is no day unit. `"0"` keeps every id forever, which was the only behavior before this key existed, and a `config.json` without the key means the same. Once an id's retention has ended, the next record carrying it is published as new. With `dedupe.backend: pebble`, a background sweep over the shared Pebble instance deletes the expired id: first about a minute after the instance opens (when the first tenant switches dedupe on), then hourly while any tenant keeps it on, counted by `wavehouse_dedupe_swept_keys_total{reason="expired"}`. With `dynamodb`, no sweep runs: the table's TTL on `ex` deletes the item ([Deployment](/deployment#a-shared-dedupe-table-on-dynamodb)). A finite retention must be at least `"2m"`, the embedded ingest queue's duplicate window (under `mq.backend: nats`, keep it at least the partitions' `duplicate_window` too, which boot warns about but this file cannot check; see [External NATS](/deployment#create-the-topology)): every deduped record is published under an idempotency key derived from its id, so an id re-sent after a shorter retention would be claimed again and then dropped by the queue as a copy, while the client was told it was accepted. A retention below that is refused, not raised to the minimum; so are a negative value and anything that is not a duration, such as `"30d"`, a number with no unit (`"300"` needs one: `"300s"`; `"0"` is the one exception), or a JSON number rather than a string. Hot-reloadable: a change applies to ids committed after the reload, and an id already committed keeps the expiry it was stored with. - `dedupe.tables.
.{id_field, require_id, retention}` — per-table overrides; each entry overrides only the fields it names and inherits the rest, so a table with no `retention` keeps the tenant's (forever when the tenant sets none). A table can keep ids for a shorter time than its tenant, or for longer, or forever (`"retention": "0"`) under a finite tenant retention. ## ClickHouse The `clickhouse` block is the connection wiring, password included. A reload applies it to every consumer (schema discovery, structured queries, pipes, `/readyz`, the ingest worker's HTTP `INSERT`s, and the raw-SQL proxy — the HTTP-side ones re-read the target per request). A change to `addr`, `database`, `username`, `password` or the `tls` block moves the tenant to the pool of its new tuple, opened for it when no served tenant has that tuple; a change to `max_open_conns` or `max_idle_conns` resizes its pool; a change to `http_port`, `http_scheme`, `headers` or `query_timeout` needs no new connection. A replaced connection stays open for the longest `query_timeout` among the tenants that were on it, so in-flight queries finish. The change is **unconditional**: the adopted settings are the authority, so an address that isn't reachable is applied all the same and shows up where reachability already does — schema discovery retries and logs, `/readyz` fails, queries return errors — until the next reload fixes it. Two things are refused instead: a `tls` block whose certificate files cannot be loaded (unreadable, not PEM, or a `cert_file` and `key_file` that do not pair), and a pool that would put the process over the boot config's [`clickhouse.max_total_conns`](/configuration#clickhouse) — a `max_open_conns` raised above it keeps the pool at its size, and a new connection tuple that would cross it is not opened. An option the ClickHouse driver refuses to open is refused the same way, though validation already excludes every one it knows of. A refused tenant keeps the pool and wiring it had until the next reload, or, newly served with no pool to keep, stays on none and its queries answer `503`. The reload itself still reports `adopted` — the refusal is an `ERROR` log line, not a finding — and the next reload retries it; at boot either refuses to start. Validation checks shape only (`host:port`, port range, scheme, non-empty database and user, timeout `>= 1`, the `tls` block's shape without opening its paths, header names and values, pool sizes); reachability is a runtime concern, so `wavehouse validate` needs no ClickHouse and no certificate files. At boot the address is dialed lazily, as before: an unreachable ClickHouse degrades `/livez` and retries rather than refusing to start. **TLS.** There are two hops and two switches: `tls.enabled` puts the native-protocol connection (`addr`) on TLS, and `http_scheme: "https"` does the same for the HTTP interface (`http_port`). The rest of the `tls` block — the authority bundle, a client certificate, `insecure_skip_verify`, `server_name` — applies to whichever hop uses TLS, so a ClickHouse behind a private authority needs `ca_file` once for both. The certificate files are read when a pool opens: at boot, and on a reload that names a tuple no open pool has — a changed `tls` block, `addr`, `database`, `username` or `password`. A reload that changes only `http_port`, `http_scheme`, `headers`, `query_timeout` or the pool sizes reuses the material already loaded, so a file replaced in place is picked up the next time a pool for its tuple opens, or by a restart. Both switches move the hop to ClickHouse's TLS listeners, so the ports move too: `addr` to the secure native port (`9440` by default) and `http_port` to the HTTPS one (`8443`), as the example above does. Validation warns when only one hop is on TLS: a plaintext HTTP hop carries the credentials on every insert and raw-SQL query, and a plaintext native hop sends the password in its handshake. In the container images these are container paths: bind-mount the bundle (`-v /srv/clickhouse-ca.pem:/etc/wavehouse/clickhouse-ca.pem:ro`) and make it readable by UID 65532, like the settings directory. **Headers.** `clickhouse.headers` rides on every HTTP-interface request: routing or identification metadata for a proxy or gateway in front of ClickHouse. Values are stored in the file as written, so a credential placed there is only as protected as the settings directory itself — which already carries the tenant's secrets (`clickhouse.password`, `auth.jwt_secret`), so protect the directory accordingly (see [Deployment — Persistent Storage](/deployment#persistent-storage-required-for-containers)). WaveHouse's own headers — `Content-Type` and the `X-ClickHouse-User` / `X-ClickHouse-Key` credentials — are set after them and win. Naming `X-ClickHouse-User` or `X-ClickHouse-Key` is a validation error, and so is `Authorization`: ClickHouse's HTTP interface reads it as Basic credentials, a second and conflicting credential path, so a gateway that authenticates that way needs a header name of its own. The native protocol carries no headers. **Existing directories.** Every key is required, so a `config.json` written before these keys existed fails validation until `tls`, `headers`, `max_open_conns` and `max_idle_conns` are added. The seed values — `tls.enabled: false` with the other `tls` keys empty, `headers: {}`, `10` and `5` — change nothing; `wavehouse bootstrap` in an empty directory writes them for reference. **Per tenant.** Over a [nested settings directory](/deployment#the-nested-settings-directory) each folder's `clickhouse` block is that tenant's own. The process opens one native connection pool per distinct `addr`, `database`, `username`, `password` and `tls` tuple among the tenants being served — ClickHouse authenticates per connection, so tenants with different credentials never share one — and tenants naming the same tuple share one pool, sized to the largest `max_open_conns` and the largest `max_idle_conns` among them. `http_port`, `http_scheme`, `headers` and `query_timeout` stay each tenant's own whatever it shares: the HTTP hop is addressed per request and the deadline is per query. A reload reconciles the pools: a tenant whose tuple changed moves to the pool that tuple names, opened if it is new; a pool no tenant names any more closes once the longest `query_timeout` among the tenants it had has passed; a pool whose largest ask changed is resized the same way. The boot config's [`clickhouse.max_total_conns`](/configuration#clickhouse) bounds the open pools **together**: at boot, pools that would add up to more refuse to start, naming the sum and the ceiling; on a reload, a resize above it is refused and the pool keeps its size, and a tuple that cannot be opened — the ceiling, a `tls` block whose files cannot be loaded, or options the driver refuses — leaves its tenants on the pool they had, or on none for a tenant that had none. A tenant on no pool answers `503` with `Retry-After` on every route that reaches its ClickHouse until a reload opens one. The refusals are `ERROR` log lines, not findings, and the next reload retries. Schema discovery is per tenant too: each tenant's tables are discovered from its own `database` over its own pool, on its own `schema.refresh_interval`, with the first periodic refresh at a random point within the interval so tenants adopted together do not refresh together; until a tenant's first discovery, its table lookups answer `503` with `Retry-After` rather than `404` — the table may well exist. Moving to a different ClickHouse address or database starts the tenant's schema discovery over with the reload, not at its next `schema.refresh_interval`: its table lookups answer `503` with `Retry-After` until the new database's tables are discovered, retried with backoff while it cannot be reached, and are never answered from the previous database's schema. A tenant whose address and database did not change keeps its schema. Events already accepted stay in the queue, and the worker's next flush uses the new target. ## Authentication The `auth` block is the verifier wiring, HMAC secret included; only the operator key is boot config. `jwks_url` (absolute `http(s)` URL, or `""`), `jwt_secret` and `role_claim` reload live: a reload that changes any of them (the `jwt_secret` only while `jwks_url` is empty — a JWKS tenant ignores it, so editing it there keeps the verifier it has) builds a new verifier and swaps it in atomically — a rotated `jwt_secret` takes effect on the next request, and tokens signed with the previous one stop validating — so a request never sees a JWKS key source paired with the HMAC algorithm allowlist; one that changes none of them keeps the verifier it has. The rebuild does not depend on the JWKS being fetchable (the adopted wiring wins), and neither boot nor a reload waits on the endpoint: the key set is fetched in the background, and until a fetch has succeeded the verifier is *pending* — a request carrying a token answers `503 {"error": "token verifier not ready: the tenant's JWKS has not been fetched yet"}` with `Retry-After: 30`, never an evaluation under the policy `default_role` that could accept its data under a lesser role, while a request carrying no token is unaffected and served as the `default_role`. Each failed fetch logs `jwks refresh failed; no token validates until it succeeds` with the URL and the cause; once a set has been fetched, a later failed refresh keeps that set in use, tokens go on validating against it, and the line reads `jwks refresh failed; the previously fetched key set stays in use` instead. Until the first fetch succeeds it is retried with backoff — one second, doubling to a minute — so an endpoint that comes back is picked up within a minute, without a reload; a fetched set is then refreshed hourly and, rate-limited to one per five minutes, on an unknown key id. While that limiter is closed, a token naming a key id the set does not hold is refused as invalid at once (the request falls back to the policy `default_role` like any other unverifiable token), unless the next refetch is under a minute away, in which case it waits for it (at most a minute) and is then checked against the newly fetched set. A response over 1 MiB (1048576 bytes) is refused as if unreachable, with `jwks response exceeds 1048576 bytes` as the cause — a real key set is a few hundred bytes per key, so if a provider's set ever grows past that, this log line is the symptom to look for. Switching between HMAC (`jwt_secret` set, `jwks_url` empty) and JWKS is a file edit. Over [a nested directory](/deployment#the-nested-settings-directory) each tenant's folder wires its own verifier, HMAC secret included; the operator key is boot config, shared by every tenant. ## Dead Letter Queue A batch insert ClickHouse rejects is retried row by row; a row ClickHouse rejects again on its own is a poison row. A ClickHouse that cannot take the insert at all — down, unreachable, overloaded, read-only, refusing the credentials — makes no poison rows, whatever this switch says: the batch stays in the ingest queue and is retried with backoff (see [Ingest Pipeline](/ingest-pipeline#when-clickhouse-cannot-take-an-insert)). A batch whose tenant has no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and every row of it is a poison row. `dlq.enabled` (seed default `true`) decides what happens to it, resolved per table (`dlq.tables.
.enabled` → global) at the moment of the failure, so a reload applies to the next poison row: - `true` — the row is published to the tenant's dead-letter stream (`DLQ_{tenant}`) under `dlq.{tenant}.{table}` (`0` for a directory that holds the four files) with the failure in its headers, and its original is acked. Inspect it with `GET /v1/ops/dlq/stats` (admin-only; `?tenant=` names the tenant). - `false` — the row is left unacked, so NATS redelivers it and it retries until it inserts or the switch is flipped back. For every row the worker **can read**, nothing is ever dropped either way — the choice is *park it* versus *keep retrying*. **One exception:** an envelope the worker cannot read *at all* — malformed JSON, an unknown `format`, or `columns` and `row` that do not pair — can never insert, so redelivering it forever would wedge the consumer. With the DLQ off for the table it is acked and **dropped**, logged at `ERROR` and counted by `wavehouse_ingest_poison_total` with `disposition="dropped"` (also labeled by `table` and `reason`; an envelope parked on the DLQ carries `disposition="parked"`). See [Ingest Pipeline](/ingest-pipeline). For a tenant no longer served — its folder removed or rejected — there is no switch to read: its rows are always parked, so none of them sits unacked in its ingest queue, redelivered for as long as the tenant is away and stopping the [Active Sweeper](/ingest-pipeline#the-active-sweeper) purging that queue. A tenant's dead-letter stream is opened when the tenant is first served (an empty stream costs nothing) and the stats endpoint is always registered — the switch is purely behavioral, which is what makes it safe to reload. ## Message Queue - `mq.max_bytes_gb` (seed default `50`) — not applied under [`mq.backend: nats`](/deployment#external-nats), where the partition streams' limits are the operator's; everything below is the embedded queue. The disk budget for the tenant's embedded JetStream ingest stream (`INGEST_{tenant}`), which buffers its ingested events until the worker writes them to ClickHouse; its dead-letter stream (`DLQ_{tenant}`) gets a tenth of it. Each tenant's pair of streams is its own, opened when the tenant is first served and kept, at the budget it last had, when its folder is rejected or removed. The ingest stream runs `DiscardNew`, so when it's full new publishes are rejected and `POST /v1/ingest` returns `503` for that tenant alone — [backpressure by construction](/ingest-pipeline#backpressure-and-durability-knobs). A reload updates both streams' limits in place without touching what's buffered: growing takes effect immediately; shrinking below what's currently on disk makes the ingest stream refuse new publishes until the Active Sweeper purges it back under the limit — what it purges is what is both written to ClickHouse and past the tenant's `stream.gap_window_minutes`, and nothing already accepted is dropped — and a dead-letter stream holding more than a tenth of the new budget is kept at what it holds rather than shrunk, since shrinking it would delete its oldest parked rows; that is logged, and the stream then makes room for each new row by dropping its oldest, as a full one always does. If NATS rejects the update, the rest of the reload is still adopted, the failure is logged, and the next reload retries it. A queue NATS will not open at all refuses boot, like every other store; over [a nested directory](/deployment#the-nested-settings-directory) it costs that tenant alone, at boot or on reload — its ingest answers `503`, each reload trying the queue again, and so does a publish, at most once every five seconds — while every other tenant carries on. A queue that opens while the server runs but that a consumer — the ingest worker's or the stream hub's — cannot join is treated the same way: it costs that tenant alone, its ingest answering `503` until a reload or a publish joins the consumer, so no row is accepted that the worker would not write or the tenant's `GET /v1/stream` connections would not see. The two streams are resized as a pair: a failed DLQ resize undoes the ingest one so both stay on the previous budget, but if that undo fails too the ingest stream keeps the new limit and the DLQ the previous one until a later reload succeeds — the log line says which happened. **Sizing the volume.** Nothing checks the budget against the disk — neither one tenant's nor what the tenants' add up to ([#138](https://github.com/Wave-RF/WaveHouse/issues/138)) — so keep within the free space of the `/nats` volume every tenant's budget plus its dead-letter stream's cap: a tenth of the budget, or what the stream held when a smaller budget arrived, if that is more. Count every tenant ever served on the volume, not only those served now: a rejected or removed tenant's queue is kept and nothing deletes it, so what it holds goes on holding disk — a rejected tenant's replay history, and the rows parked on either one's dead-letter stream, up to that cap. A disk that fills before a budget does fails every tenant's writes, not one: the failed write is logged, the publish goes unanswered until it times out, and ingest answers `500` (`publish failed`) for every tenant on that volume, not the `503` with `Retry-After` of a full budget. It does not clear on its own: a stream that failed a write refuses every later one until WaveHouse restarts, so free the space and then restart. ## Streaming `GET /v1/stream` keepalives and gap-fill are tuned here: - `stream.keepalive_interval` (seed default `30`) — seconds a quiet connection may go without a write before the server sends a `:` keepalive comment. It exists to stay under whatever idle timeout sits between WaveHouse and the client; the default clears the common 55–60s proxy windows with margin, and a tighter edge (Azure Application Gateway 20s, CloudFront 30s) wants a lower value — see [Behind a reverse proxy → Idle timeouts](/reverse-proxy#idle-timeouts-by-provider). A reload rebuilds the keepalive wheel in place: live connections stay open and are redistributed across the new ring, each getting at most one full new period before its next keepalive. - `stream.keepalive_buckets` (seed default `3`) — spreads the keepalive writes across the interval (one bucket fires every `keepalive_interval ÷ keepalive_buckets`) so the server nudges ~1/N of connections per tick instead of all at once. It changes only how the writes are spread in time, never the period. At most `100`: the wheel ticks once per bucket per interval and holds a set per bucket, so the cap bounds each tenant's tick rate and ring, and a hundred buckets already spreads the writes to 1% of the tenant's streams per tick. - `stream.gap_window_minutes` (seed default `15`) — minutes of already-written-to-ClickHouse history the Active Sweeper keeps in NATS so a reconnecting client's `Last-Event-ID` replay can bridge the gap; a drop longer than this resumes with a hole. Bounded by the tenant's disk budget, [`mq.max_bytes_gb`](#message-queue). A reload applies from the next sweep (every minute). Under [`mq.backend: nats`](/deployment#external-nats) the history stream's `max_age` decides instead, and the API processes warn once per tenant and window about a tenant whose window is longer.