Watchpost

Docs · Deploy

Data and retention

Watchpost preserves enough operational history to explain state without allowing unbounded telemetry to exhaust the node.

Stored records

Time and provenance

Observations distinguish event time from ingestion time. Logs retain their post, source, severity, and observation time. This lets operators distinguish a delayed collector from a delayed event.

Bounds

Requests, label counts, field sizes, log messages, evidence sets, OID profiles, history queries, and federation payloads are bounded. High-cardinality data can exhaust a node before raw sample count does, so rejected or truncated data must remain visible.

Default retention policy

observations:            30 days
check results:           90 days
logs:                    30 days
changes:                 2 years
resolved alerts:         365 days after resolution
delivered notifications: 30 days
pairing tokens/requests: 7 days after terminal state
federation inbox:        7 days
federation outbox:       30 days after delivery
orphaned conversations:  180 days
incidents, audit, action requests: retained until operator policy removes them

Every category is configurable through WATCHPOST_RETENTION_<CATEGORY>; a zero value keeps that category forever. Expired sessions are always pruned. Deletion runs in bounded batches (default 1,000 rows, capped per pass) so a backlogged table cannot monopolise the database connection or grow the write-ahead log without limit.

Active alerts are preserved regardless of age: the current pending/firing/acknowledged/suppressed row per rule and post is never pruned. Superseded transition rows age out after the resolved window unless they are linked to an incident or cited by an investigation. Evidence cited by an investigation is preserved through an immutable snapshot: after raw evidence is pruned, the citation still resolves with an explicit purged by retention status rather than silently vanishing.

The storage and recovery behaviour below is exercised by the local backup, restore, and corruption drill described in Verification and hardening.

Capacity protection

The node measures its total SQLite footprint, including write-ahead and shared-memory sidecars, against a configurable cap and a free-disk floor. At capacity, telemetry and log ingestion are rejected explicitly with HTTP 507 and an immediate retention pass reclaims space; collectors and agents retry inside their bounded queues and any eventual queue loss is counted and displayed. The storage report is an authenticated operational diagnostic and is also surfaced as a warning in the interface.

SQLite boundary

The current node uses SQLite with foreign keys and versioned migrations. Use watchpost backup for a consistent online snapshot and watchpost restore to restore after validation:

watchpost backup --output daily.wpbk --passphrase-file /secure/backup-pass
watchpost restore --input daily.wpbk --passphrase-file /secure/backup-pass --data-dir /var/lib/watchpost --force

Encrypted backups use a random salt and a versioned, authenticated header (KDF, work factor, salt, nonce, version), are written atomically, and never contain the passphrase or master key that protects them. Scheduled backups run with WATCHPOST_BACKUP_DIR and WATCHPOST_BACKUP_SCHEDULE. Verify restores by comparing schema version, post counts, incident counts, and representative evidence.

Exports and privacy

CSV history exports remain authenticated. Operators are responsible for the retention and access policy of exported files. Diagnostic output must not become a side channel for secrets or sensitive telemetry.