Production deployment

Carabase runs on your always-on Mac, reachable only over Tailscale — no public ports, no cloud dependency, no Docker-in-a-VM on the Mac itself. Deploys are a local source build, restarted via launchd.

pnpm deploy:prod is the only supported way to change prod. It is the one path that keeps three things in lockstep: the checked-out code, the applied database migrations, and the running process. Skip it — git pull and a manual restart — and those three can drift apart, which is how production breaks.

Components

Script	Runs where	What it does
scripts/install-prod-service.sh	prod Mac	One-time: installs the launchd LaunchAgent
scripts/deploy-prod.sh	prod Mac	Every deploy: backup → pull → migrate → build → restart → smoke → auto-rollback
scripts/rollback-prod.sh	prod Mac	Manual rollback to a specific sha/tag
pnpm smoke:prod --url <host>	anywhere	Read-only HTTP smoke against a running host URL

All four scripts load .env.production via scripts/load-env.sh, so they share a single, tested env-resolution path.

The launchd service

pnpm install-prod-service writes ~/Library/LaunchAgents/dev.carabase.host.prod.plist with:

• RunAtLoad=true — start on login
• KeepAlive.SuccessfulExit=false — auto-restart on crash
• ThrottleInterval=10 — don’t flap if something is badly broken
• StandardOutPath=~/.carabase/logs/prod.log
• StandardErrorPath=~/.carabase/logs/prod.err
• ProgramArguments = ./scripts/with-env.sh prod -- node dist/server.js

The installer refuses if .env.production or dist/server.js don’t exist — a flapping service that can’t start is worse than a missing service.

What pnpm deploy:prod does

Every run, in order:

1. Loads .env.production, classifies DATABASE_URL, and refuses if the db name isn’t *_prod / *_production.
2. Acquires a lock file at ~/.carabase/deploys/.lock so two deploys can’t race (a stale lock can be removed manually).
3. Records the current git sha (PRE_SHA) and branch, and writes audit metadata to ~/.carabase/deploys/<ts>.json.
4. Runs pnpm backup:prod — every change gets a backup first. --skip-backup exists but prints a loud warning and is for emergencies only.
5. git fetch origin --tags + git checkout --detach <target> (default origin/main; override with --tag v0.2.0 or --ref 1a2b3c).
6. pnpm install --frozen-lockfile.
7. pnpm db:migrate:prod (Drizzle applies forward migrations only).
8. pnpm build.
9. Defensive port-kill: cleanly terminate any process still listening on the prod port, then force-kill survivors, refusing to proceed if the port stays occupied. launchctl kickstart -k only signals the launchd-tracked parent and can leave the real node process owning the port, which a naive restart would silently “pass” the smoke against.
10. launchctl kickstart -k gui/<uid>/dev.carabase.host.prod — atomic restart.
11. Poll http://127.0.0.1:$PORT/api/v1/health until 200, or 30s.
12. Run pnpm smoke:prod --url <live-url> — the read-only prod smoke.

On any failure after the restart: auto-rollback. git checkout $PRE_SHA → pnpm install → pnpm build → port-kill → restart → best-effort health check → exit 1. The audit metadata is updated with status: rolled_back and the failed/restored shas.

--dry-run prints the plan without executing. Every run tees its full output into ~/.carabase/deploys/<ts>.log.

Drift prevention — atomic deploys only

pnpm deploy:prod exists because three things must stay in lockstep: the code on disk, the migrations applied to the database, and the running process. The most common way to break production is to break that lockstep — advancing the working tree and restarting without running migrations, so newer code hits a missing column and 500s on every request.

• Never git pull + manually restart on the prod host. The service runs straight from the working tree, so a raw pull silently advances the code ahead of the schema. Always use deploy:prod (pull → migrate → build → restart, fail-closed, with auto-rollback).
• Boot-time schema-drift guard. On startup the host compares the migrations the code expects against the migrations actually applied to the database. If the schema is behind, it logs a single SCHEMA_DRIFT: N migration(s) pending and exits before serving any request — rather than crash-looping on per-query 500s. The fix is pnpm db:migrate:<env>, then restart. An ALLOW_SCHEMA_DRIFT=1 override exists but is unsafe (the app will 500 on missing columns).
• Database credential-rotation discipline. Rotating a Postgres role password is a two-write operation: change it in the database and update DATABASE_URL in .env.production, in the same step. If only one is done, the boot guard reports a distinct DB_AUTH_FAILED so the cause is obvious in the log instead of an opaque crash loop.
• Postgres log hygiene. An auth-failure crash loop can balloon the Postgres log file, and Homebrew’s Postgres has no built-in rotation. On the prod host, add a newsyslog.d rule to cap and rotate it.

Release pipeline

A tagged release (v* tag push) runs through a CI pipeline that gates publishing on quality: tests (against PostgreSQL + pgvector), lint and type-check, a dependency audit, and the E2E smoke against a throwaway database — all of which must pass before the build is published. Publishing then builds and signs a container image (cosign keyless via GitHub OIDC), generates and signs a CycloneDX SBOM, composes release notes from the CHANGELOG plus commits since the previous tag, and publishes the GitHub Release. You cannot tag something broken — the quality gates run first.

Daily workflow

# On your dev machine — make changes, open a PR, merge to main
git push
# → CI runs lint + test + smoke:e2e
# → PR merged to main

# On the prod Mac (or ssh in over Tailscale)
cd ~/Code/carabase-host
pnpm deploy:prod
# → backup → pull main → migrate → build → restart → smoke
# → auto-rollback on any failure
# → audit log at ~/.carabase/deploys/<ts>.log

To pin prod to a specific tagged version:

pnpm deploy:prod --tag v0.2.0

To emergency-roll back to the previous tag (this does not run migrations — restore from backup if the schema needs to move):

pnpm rollback:prod --to v0.1.0

Read-only prod smoke

pnpm smoke:prod is the post-deploy gate. Unlike the E2E smoke (which creates a scratch DB, seeds it, and spawns a subprocess), this one:

• Takes --url <host-url> and hits it with real HTTP requests.
• Does not create, seed, or mutate any DB state.
• Does not require a known workspace id to run basic liveness.
• With --workspace-id <uuid>, additionally runs read-only scoped checks (e.g. model-routing GET, folios list, canonical entities list).

The unauthenticated checks always run:

• GET /api/v1/health → 200
• GET /health → 200
• GET /api/v1/folios without the auth header → 400 (confirms the auth middleware is wired — a critical security regression otherwise)

Security model recap

• No public ports. The host binds to the local interface; all external access is through Tailscale, which is WireGuard underneath.
• No shared secrets between environments. Each .env.<env> has its own HOST_MASTER_KEY. Compromising dev doesn’t leak prod.
• Refusal-by-default. db:reset, eval seeding, and deploy:prod all refuse to touch a *_prod URL unless explicitly opted in.
• Auto-rollback on smoke failure. The deploy script only leaves prod in the new state if the post-deploy smoke is green.
• Audit trail. Every deploy writes a <ts>.log (full output) and <ts>.json (pre/post sha, status, rollback info) under ~/.carabase/deploys/.

First deploy

For the complete one-time setup checklist — prereqs, env files, first build, backup cron install, service install, first deploy, first tag, desktop reconnection, and a backup/restore stress test — see docs/FIRST_DEPLOY.md in the repo.