Production deployment

Carabase runs on your always-on Mac, reachable only over Tailscale — no public ports, no cloud dependency, no Docker-in-a-VM on the Mac itself. Deploys are a local source build, restarted via launchd.

pnpm deploy:prod is the only supported way to change prod. It is the one path that keeps three things in lockstep: the checked-out code, the applied database migrations, and the running process. Skip it — git pull and a manual restart — and those three can drift apart, which is how production breaks.

Components

Script	Runs where	What it does
scripts/install-prod-service.sh	prod Mac	One-time: installs the launchd LaunchAgent (running ~/.carabase/releases/current) and the watchdog
scripts/deploy-prod.sh	prod Mac	Every deploy: backup → build a release → migrate → switch → restart → smoke → auto-rollback. --bootstrap creates the release layout
scripts/rollback-prod.sh	prod Mac	Manual rollback: --previous or --to <sha/tag>
pnpm smoke:prod --url <host>	anywhere	Read-only HTTP smoke against a running host URL

All of them load .env.production via scripts/load-env.sh, so they share a single, tested env-resolution path.

The launchd service

pnpm install-prod-service writes ~/Library/LaunchAgents/dev.carabase.host.prod.plist with:

• RunAtLoad=true — start on login
• KeepAlive.SuccessfulExit=false — auto-restart on crash
• ThrottleInterval=10 — don’t flap if something is badly broken (deploys never restart inside it — see step 10 below)
• StandardOutPath=~/.carabase/logs/prod.log
• StandardErrorPath=~/.carabase/logs/prod.err
• ProgramArguments = cd ~/.carabase/releases/current && exec ./scripts/with-env.sh prod -- node --import tsx dist/server.js — the compiled server, exec’d so launchd tracks the process that owns the port

Plist changes only apply when the installer is re-run: launchctl kickstart restarts the loaded job without re-reading the plist.

Prod never runs from a working checkout. Each deployed commit is built into its own directory, ~/.carabase/releases/<sha>, and launchd runs whatever the current symlink points at — so switching branches in your development checkout can never change what prod runs, and a rollback is a symlink switch. All releases share one env file, ~/.carabase/releases/shared/.env.production. Create the first release with ./scripts/deploy-prod.sh --bootstrap before installing the service; the installer refuses if current or its .env.production doesn’t exist — a flapping service that can’t start is worse than a missing service.

The companion watchdog also rotates prod.log / prod.err by size (gzip generations, truncating in place so the running host keeps logging without a restart) and warns with a macOS notification when free disk space drops below a floor (default 25 GiB free or 92% used), before the volume fills.

What pnpm deploy:prod does

Every run, in order:

1. Loads .env.production, classifies DATABASE_URL, and refuses if the db name isn’t *_prod / *_production.
2. Acquires a lock file at ~/.carabase/deploys/.lock so two deploys can’t race (a stale lock can be removed manually).
3. Records the current release (PRE_SHA) and writes audit metadata to ~/.carabase/deploys/<ts>.json.
4. Runs pnpm backup:prod — every change gets a backup first. --skip-backup exists but prints a loud warning and is for emergencies only.
5. Fetches origin --tags and resolves the target (default origin/main; override with --tag v0.2.0 or --ref 1a2b3c).
6. Builds ~/.carabase/releases/<sha>: a detached worktree, pnpm install --frozen-lockfile, pnpm build. If this fails, prod is untouched.
7. pnpm db:migrate:prod (Drizzle applies forward migrations only).
8. Switches the current symlink to the new release.
9. Stop the old host: cleanly terminate whatever listens on the prod port, then force-kill survivors, refusing to proceed if the port stays occupied.
10. Let launchd respawn it from the new current (its KeepAlive restarts the job immediately); kickstart only if nothing is running. Killing that fresh respawn with kickstart -k used to trip launchd’s ThrottleInterval and hold port 3000 closed for ~35 s, long enough for MCP clients to give up. The deploy log records the measured restart gap.
11. Poll http://127.0.0.1:$PORT/api/v1/health until 200.
12. Run pnpm smoke:prod --url <live-url> — the read-only prod smoke.

On any failure after the switch: auto-rollback. current is switched back to the pre-deploy release (still built on disk — no reinstall or rebuild) → port-kill → restart → best-effort health check → exit 1. The audit metadata is updated with status: rolled_back and the failed/restored shas. Old releases are pruned after a successful deploy (three are kept by default). The rollback is skipped if the deploy migrated the database past what the previous release understands, because that release would refuse to boot. Instead the new release stays up and you get a notification to intervene. rollback-prod.sh applies the same check.

--dry-run prints the plan without executing. Every run tees its full output into ~/.carabase/deploys/<ts>.log.

Drift prevention — atomic deploys only

pnpm deploy:prod exists because three things must stay in lockstep: the code on disk, the migrations applied to the database, and the running process. The most common way to break production is to break that lockstep — advancing the working tree and restarting without running migrations, so newer code hits a missing column and 500s on every request.

• Never repoint current or restart onto new code by hand. Always use deploy:prod (build → migrate → switch → restart, fail-closed, with auto-rollback), or the code runs ahead of the schema.
• Boot-time schema-drift guard. On startup the host compares the migrations the code expects against the migrations actually applied to the database. If the schema is behind, it logs a single SCHEMA_DRIFT: N migration(s) pending and exits before serving any request — rather than crash-looping on per-query 500s. The fix is pnpm db:migrate:<env>, then restart. An ALLOW_SCHEMA_DRIFT=1 override exists but is unsafe (the app will 500 on missing columns).
• Database credential-rotation discipline. Rotating a Postgres role password is a two-write operation: change it in the database and update DATABASE_URL in .env.production, in the same step. If only one is done, the boot guard reports a distinct DB_AUTH_FAILED so the cause is obvious in the log instead of an opaque crash loop.
• Postgres log hygiene. An auth-failure crash loop can balloon the Postgres log file, and Homebrew’s Postgres has no built-in rotation. On the prod host, add a newsyslog.d rule to cap and rotate it.

Release pipeline

A tagged release (v* tag push) runs through a CI pipeline that gates publishing on quality: tests (against PostgreSQL + pgvector), lint and type-check, a dependency audit, and the E2E smoke against a throwaway database — all of which must pass before the build is published. Publishing then builds and signs a container image (cosign keyless via GitHub OIDC), generates and signs a CycloneDX SBOM, composes release notes from the CHANGELOG plus commits since the previous tag, and publishes the GitHub Release. You cannot tag something broken — the quality gates run first.

Daily workflow

# On your dev machine — make changes, open a PR, merge to main
git push
# → CI runs lint + test + smoke:e2e
# → PR merged to main

# On the prod Mac (or ssh in over Tailscale)
~/.carabase/releases/current/scripts/deploy-prod.sh
# → backup → build release → migrate → switch → restart → smoke
# → auto-rollback on any failure
# → audit log at ~/.carabase/deploys/<ts>.log

That runs the deploy scripts of the last deployed release, which is right for an ordinary deploy. When the change you are deploying modifies the deploy or install scripts themselves, run pnpm deploy:prod from a checkout at origin/main instead (git checkout --detach origin/main), then re-run ~/.carabase/releases/current/scripts/install-prod-service.sh if the launchd plist changed. A checkout parked on an old commit runs old scripts.

To pin prod to a specific tagged version:

pnpm deploy:prod --tag v0.2.0

To emergency-roll back (this does not run migrations — restore from backup if the schema needs to move):

~/.carabase/releases/current/scripts/rollback-prod.sh --previous
~/.carabase/releases/current/scripts/rollback-prod.sh --to v0.1.0

Read-only prod smoke

pnpm smoke:prod is the post-deploy gate. Unlike the E2E smoke (which creates a scratch DB, seeds it, and spawns a subprocess), this one:

• Takes --url <host-url> and hits it with real HTTP requests.
• Does not create, seed, or mutate any DB state.
• Does not require a known workspace id to run basic liveness.
• With --workspace-id <uuid>, additionally runs read-only scoped checks (e.g. model-routing GET, folios list, canonical entities list).

The unauthenticated checks always run:

• GET /api/v1/health → 200
• GET /health → 200
• GET /api/v1/folios without the auth header → 400 (confirms the auth middleware is wired — a critical security regression otherwise)

Security model recap

• No public ports. The host binds to the local interface; all external access is through Tailscale, which is WireGuard underneath.
• No shared secrets between environments. Each .env.<env> has its own HOST_MASTER_KEY. Compromising dev doesn’t leak prod.
• Refusal-by-default. db:reset, eval seeding, and deploy:prod all refuse to touch a *_prod URL unless explicitly opted in.
• Auto-rollback on smoke failure. The deploy script only leaves prod in the new state if the post-deploy smoke is green.
• Audit trail. Every deploy writes a <ts>.log (full output) and <ts>.json (pre/post sha, status, rollback info) under ~/.carabase/deploys/.

First deploy

For the complete one-time setup checklist — prereqs, env files, first build, backup cron install, service install, first deploy, first tag, desktop reconnection, and a backup/restore stress test — see docs/FIRST_DEPLOY.md in the repo.