HailDeck self-hosting

Supported deployment

The supported production stack is infra/docker-compose.yml: the HailDeck web/API container, Caddy TLS proxy, LiveKit SFU, coturn, and authenticated Redis. Persistent application state lives in named volumes, while finalized MP4 recordings live under the host-mounted .haildeck-data/recordings directory shared by the application and Egress; only Caddy and media ports are public.

Copy infra/.env.example to infra/.env and replace every example secret with independently generated random values of at least 48 characters.
Set public DNS for HAILDECK_DOMAIN and LIVEKIT_DOMAIN to the host. Open TCP 80/443/3478/5349/7881 and UDP 443/3478/49160-49260/55000-55100 as required by the compose file.
Run docker compose --env-file infra/.env -f infra/docker-compose.yml config --quiet.
Run docker compose --env-file infra/.env -f infra/docker-compose.yml up -d --build.
Confirm https://$HAILDECK_DOMAIN/ready returns HTTP 200, check the container health status and logs, then initialize the first owner in the browser.

The Compose application uses a non-root user, a read-only container filesystem, bounded logs, and a writable data volume mounted at /app/.haildeck-data. Core chat state is explicitly stored at /app/.haildeck-data/state.json; the other JSON stores and uploaded files use the same volume. Existing volumes must be writable by application UID 1001. Keep recording storage writable by the application and Egress. Validate these permissions before replacing an existing container.

For a single Windows test host, run scripts/validate-config.ps1 followed by scripts/start-portal.ps1. This binds only to 127.0.0.1:4173.

Boot-time configuration validation

The server validates its environment at boot (apps/server/src/config.ts) before binding a port. With NODE_ENV=production it exits non-zero, listing every problem, if:

AUTH_SIGNING_KEY, DATA_ENCRYPTION_KEY, or INTEGRATION_TOKEN_PEPPER is missing, shorter than 48 characters, or still a placeholder from an .env.example file;
ALLOWED_ORIGINS is missing, empty, contains *, or contains anything other than exact HTTP(S) origins (no trailing slash, path, credentials, query, or fragment);
PORT is not an integer in 1–65535, HOST is empty, or LIVEKIT_URL is not a ws:///wss:// URL;
ENABLE_DEV_SESSION=true while HOST is a non-loopback address (the test identity must never be network-reachable).
HAILDECK_STORE=json without an explicit persistent HAILDECK_DATA_FILE or legacy FLEET_DATA_FILE path.

Invalid backend combinations, unsafe proxy trust settings, and invalid API rate budgets refuse boot in every environment. The production JSON persistence requirement prevents the core store from silently running in memory. A successful boot does not prove a volume is backed up; verify its mount and restore process separately.

Identity policy on network binds: with a non-loopback HOST, password-only sign-in is refused and sessions require a passkey assertion (set WEBAUTHN_RP_ID to the public hostname and WEBAUTHN_ORIGIN to the public origin). HAILDECK_ALLOW_PASSWORD_LOGIN=1 relaxes this for non-production LAN testing only — production ignores it. The test identity endpoint is refused in production and otherwise requires HAILDECK_TEST_IDENTITY=1 (or ENABLE_DEV_SESSION=true in development). Users recover locked-out accounts with their one-time recovery codes ("Use a recovery code" on the sign-in page); admins can direct users to regenerate codes from Security settings.

In development, secret, origin, and basic bind-setting problems become [config] warnings; missing secrets are replaced with loud ephemeral keys. Backend and proxy validation remains strict. On success the server prints one structured log line summarizing the effective non-secret config (mode, host, port, data file, allowed-origin count, proxy count, API budget, whether SMTP is configured). HAILDECK_DATA_FILE selects the persistent data file; legacy FLEET_DATA_FILE still works but logs a deprecation warning.

Reverse proxy trust and request limits

The server trusts no forwarded client addresses unless HAILDECK_TRUSTED_PROXIES lists the immediate proxy IP addresses or CIDRs. The value accepts a comma-separated list such as 172.30.83.2/32; booleans, hop counts, hostnames, and zero-prefix networks are rejected. Only list proxies you operate, and keep the application port private. Fastify resolves the client from the right side of the forwarding chain until it reaches an untrusted address. See Fastify proxy trust.

Compose assigns Caddy HAILDECK_PROXY_IP on the dedicated HAILDECK_PROXY_SUBNET and trusts that single /32 address. Defaults are 172.30.83.2 and 172.30.83.0/24; change both together if they overlap host, VPC, VPN, or Docker networks. Caddy overwrites incoming X-Forwarded-For and X-Real-IP with the connecting client's address. This configuration assumes Caddy is the public edge. Adding a CDN or another load balancer requires an explicit review of that trust boundary; otherwise clients will share the additional proxy's rate budget.

HAILDECK_API_RATE_LIMIT sets ordinary API requests per client IP per minute, defaulting to 600. It accepts integers from 1 to 100000. Authentication has separate limits: owner setup and recovery allow five requests per 15 minutes; login, invitation acceptance, passkey options/verification, and MFA confirmation/disable allow ten per 15 minutes for each endpoint. Limits apply to attempts, including malformed and unsuccessful requests. Clients sharing a public IP share those budgets; IPv6 clients are grouped by /64.

API responses expose limit/remaining/reset headers; HTTP 429 includes Retry-After. Health probes and static web assets do not consume the API budget and remain available after throttling. When HAILDECK_EVENT_BUS=redis, replicas share rate counters in Redis under haildeck:rate-limit:. Redis is checked at boot and requests fail closed if it becomes unavailable. With local, counters are in memory and reset on restart. The Compose stack selects Redis by default. See the rate-limit plugin documentation.

Invitation email (optional SMTP)

Unset SMTP variables keep today's copy-link flow. Setting any SMTP variable in production without a complete pair refuses boot.

| Variable | Required when | Example | | --------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | HAILDECK_SMTP_URL | sending mail | smtp://localhost:1025 (Mailpit/Mailhog) or smtps://user:pass@mail.example.com:465 | | HAILDECK_SMTP_FROM | HAILDECK_SMTP_URL is set | HailDeck <noreply@example.com> | | HAILDECK_PUBLIC_URL | recommended with SMTP | https://portal.example.com — origin used in accept links. If unset, the request Origin header or first ALLOWED_ORIGINS entry is used. |

Local test: run Mailpit or Mailhog on port 1025, set HAILDECK_SMTP_URL=smtp://localhost:1025 and HAILDECK_SMTP_FROM=HailDeck <noreply@localhost>, create an invite, and open the captured message. Production: point HAILDECK_SMTP_URL at your relay (STARTTLS on 587 via smtp://, implicit TLS on 465 via smtps://). Incomplete production SMTP (URL without FROM, or the reverse) is a boot failure so operators never think mail is sending when it is not.

Invite creation never rolls back if the relay is down: the API still returns the token and delivered: false. Sending is capped at 20 messages per tenant per hour. Logs record the invitation id, recipient, and origin — never the raw token or the full accept URL.

Persistence, event bus, and file storage backends

The standalone server defaults to JSON metadata, a local event bus, and filesystem storage. Production JSON requires an explicit persistent data-file path. Compose supplies this path and selects Redis for realtime delivery and rate counters. Backend switches are validated at boot; invalid combinations refuse to start:

| Variable | Values | Default | Notes | | ---------------------------------------------------------------------------------------------------------- | -------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------- | | HAILDECK_STORE | json \| postgres | json | Application metadata store. postgres requires DATABASE_URL (postgres://…); migrations run and connectivity is verified at boot. | | DATABASE_URL | postgres:// URL | — | Credential — never logged. | | HAILDECK_EVENT_BUS | local \| redis | local | Realtime fan-out. redis requires REDIS_URL (redis:// or rediss://); connectivity is verified at boot. | | REDIS_URL | redis:// URL | — | Credential — never logged. | | HAILDECK_STORAGE | fs \| s3 | fs | Uploaded file bytes. s3 works with MinIO (self-host) or AWS S3. | | HAILDECK_S3_ENDPOINT, HAILDECK_S3_BUCKET, HAILDECK_S3_ACCESS_KEY_ID, HAILDECK_S3_SECRET_ACCESS_KEY | — | — | All required for s3. The secret key is a credential — never logged. | | HAILDECK_S3_REGION | region name | us-east-1 | Optional. | | HAILDECK_S3_FORCE_PATH_STYLE | true \| false | true | Keep true for MinIO; false enables virtual-hosted addressing for AWS. |

Before changing backends, back up every application data store and verify a restore on a separate instance. PostgresStore.importJsonSnapshot imports only the core chat/incident snapshot (covered by apps/server/test/store-contract.test.ts) and refuses a non-empty target database. It does not migrate the separate identity, collaboration, operations, governance, or other JSON stores. A complete production migration must account for every store before traffic is switched.

Scaling out (multiple replicas)

Single node remains the supported default — scale out only when one node is no longer enough. Multi-replica operation requires all three shared backends: HAILDECK_STORE=postgres, HAILDECK_EVENT_BUS=redis, HAILDECK_STORAGE=s3. The compose overlay infra/docker-compose.scale.yml runs two replicas behind Caddy with Postgres + MinIO added to the stack; the Helm chart (infra/k8s) defaults to replicaCount: 2 with the same backends selected in values.yaml and credentials in the haildeck-api-secrets Secret (database-url, redis-url, s3-access-key-id, s3-secret-access-key).

How the pieces behave across replicas:

Realtime: every replica publishes events to Redis and fans them out to its own WebSocket clients, so a message sent through one replica reaches clients connected to any other. Event cursors, the bounded replay log, and client-op-id dedupe live in Redis — reconnecting to a _different_ replica resumes correctly from the last seen cursor. Presence and typing are ephemeral and propagate over the same bus (never persisted).
Sticky sessions: not required when all three shared backends are selected. With HAILDECK_STORE=postgres, sign-in, collaboration, notifications, operations, file metadata/quotas, governance, integrations, media metadata, and the tenant catalog are shared. A request may land on any healthy replica; the Helm Service therefore uses sessionAffinity: None. File bytes remain in shared object storage. Recording files must likewise use storage mounted at the same HAILDECK_RECORDING_DIR on every replica if recording download is enabled.
Files: bytes live in the bucket (uploads, downloads, scan verdicts, quarantine, quotas, and retention behave identically on fs and s3). Downloads are always proxied through the authenticated API with tenant checks — the bucket must stay private (the compose overlay's minio-init enforces mc anonymous set none).
Deployments: containers expose /ready for dependency readiness and /health for liveness. Docker Compose recreates changed containers; its health checks do not provide an ordered rolling update or guarantee continuous availability. Use a maintenance window for Compose replacement. A Kubernetes Deployment can coordinate rolling replacement with readiness probes; verify the selected rollout strategy and capacity. See Docker Compose update behavior.
Proxy health: the supplied Caddy configuration probes the haildeck-web:4173 service address every ten seconds with a three-second timeout. Two failures remove that upstream; one success restores it. With multiple Docker DNS addresses, this remains one logical Caddy upstream, so it does not independently track each replica. Configure distinct upstreams or use an orchestrator-managed service before relying on per-replica routing guarantees.

Failure modes:

Redis down: boot fails fast. At runtime /ready returns 503 and API requests fail closed because shared throttling is unavailable. Existing WebSocket connections may continue node-local delivery, but cross-replica fan-out and resume are unavailable until Redis recovers. The core /v1/events feed and collaboration history are separate domains; reconnecting clients must reload the history appropriate to their feature.
Postgres down: boot fails fast. At runtime /ready returns 503 and affected requests return errors; direct /health stays available. Restore database connectivity and verify readiness before returning the replica to traffic.
S3/MinIO down: uploads and downloads fail with a generic 5xx (no internals leaked); metadata is untouched, so files reappear when storage returns.

Backups and restore

scripts/backup-portal.ps1 creates a ZIP under backups/ with a SHA-256 manifest. Copy backups to encrypted offline storage. Test restores routinely with scripts/restore-portal.ps1 -Backup <absolute-path> -Force; the script verifies every checksum and retains the pre-restore data directory. Run scripts/verify-data.ps1 after restore.

For Docker, quiesce writes and take a consistent backup of the haildeck-data, redis-data, caddy-data, and caddy-config volumes plus the configured recording directory. For PostgreSQL/S3 deployments, also back up the database and object storage; JSON files alone are not a complete export. Restore data before starting services, preserve file ownership, and test recovery separately from the live instance. Never use docker compose down -v during an upgrade.

Upgrade and rollback

scripts/upgrade-portal.ps1 -NoBrowser creates a data backup and build snapshot, installs the lockfile, runs typecheck/tests/build, then restarts only after all gates pass. scripts/rollback-portal.ps1 -NoBrowser restores the newest prior build. Data formats are versioned and backward-compatible unless release notes explicitly require a migration.

For containers: build or pull a versioned image, run the test gates, verify the effective configuration, and back up durable state before replacement. Retain the previous image and configuration. During the maintenance window, update the image and use docker compose up -d --wait --wait-timeout 120; confirm direct readiness, public HTTPS, authentication, and a representative persisted workflow afterward. --wait reports health after replacement; it does not make the replacement a rolling deployment. Roll back with the prior image and compatible configuration. Restore data only through a deliberate recovery procedure so writes made after the backup are not silently discarded.

SIGTERM and SIGINT trigger graceful shutdown: the server stops accepting new work, closes WebSockets and HTTP connections, and releases Redis and PostgreSQL resources. A 30-second application deadline exits nonzero if draining stalls; Compose allows 35 seconds before forcibly stopping the container. Monitor shutdown errors and allow this grace period when using other process managers.

Health, logs, and incident response

Direct /health is process liveness only. /ready checks identity-store access, the shared rate-limit Redis connection when selected, and, for the runtime JSON backend, read/write access to the state file and its parent directory. It returns 503 on failure or after two seconds; concurrent probes share an in-flight check. Readiness does not currently test S3, LiveKit, SMTP, or every JSON file, so monitor those separately.
Container health checks and Caddy use /ready. Caddy can return 503 for public requests, including public /health, after removing an unhealthy upstream. Diagnose process liveness by probing inside the application container; use public HTTPS checks to monitor the complete edge path.
Windows logs: .haildeck-data/portal.out.log and .haildeck-data/portal.err.log.
Container logs: docker compose -f infra/docker-compose.yml logs --since 30m haildeck-web livekit turn.
Application request logs retain method, path, and resolved client IP while excluding query strings and authorization/cookie headers. Configuration summaries include backend selections, proxy count, and API budget. Keep all logs access-controlled; invitation-delivery logs can include recipient addresses.
If authentication keys are exposed, rotate AUTH_SIGNING_KEY, restart, and revoke active sessions. Rotate DATA_ENCRYPTION_KEY only through a planned data re-encryption maintenance window.
If a file is quarantined, only a tenant owner or security administrator can release it after independent review.

Routine maintenance

Daily: health checks and backup completion. Weekly: restore drill, audit-chain verification, failed webhook review, quarantined-file review. Monthly: dependency audit, secret rotation review, retention purge, and capacity review. Quarterly: disaster-recovery and TURN connectivity tests from external networks.

source: docs/OPERATOR-GUIDE.md — rendered at build time; the repository copy is authoritative.