Edge harvesters

Some signal can only be read from the device it lives on. Granola meeting transcripts sit in a local cache file, Apple Notes lives in the on-device NoteStore, Safari history is a per-user SQLite database, Apple Photos and Files are local, and iMessage lives in chat.db. The host can’t reach any of these over the network.

The edge harvester pattern inverts the usual data flow: instead of the host pulling, the device pushes to the host, authenticated with a per-device bearer token. Everything runs over the private Tailscale mesh and is isolated per workspace by row-level security.

The pairing flow

A workspace can register multiple devices per edge connector — each gets its own bearer token. Pairing happens once:

Desktop                                    Host
   │                                         │
   │ 1. POST /api/v1/harvester-devices/      │
   │    pairing-codes                        │
   │    { connector, device_name }           │
   │ ──────────────────────────────────────► │
   │                                         │
   │ ◄────────────────────────────────────── │
   │    { pairing_code: "ABCD-1234",         │
   │      expires_at }                        │
   │                                         │
   │ 2. User reads/types the code on the     │
   │    host to confirm the device           │
   │                                         │
   │ 3. POST /api/v1/harvester-devices/claim │
   │    { pairing_code: "ABCD-1234" }        │
   │ ──────────────────────────────────────► │
   │                                         │
   │ ◄────────────────────────────────────── │
   │    { push_token: "<plaintext, only      │
   │       returned once>", device_id }       │
   │                                         │
   │ 4. Push data with Authorization: Bearer │
   │    POST /api/v1/granola/sync            │
   │ ──────────────────────────────────────► │

Pairing codes are short-lived (they expire after a few minutes) and held in memory, never persisted. Only connectors whose manifest declares an edge or streaming auth type can be paired.

What the host stores

The harvester_devices table stores only the sha256 of each device’s push token. The plaintext is returned at claim time and never persisted. If a device is lost, you can soft-revoke it via DELETE /api/v1/harvester-devices/:id (which sets revoked_at); the next push from that device gets a 401. Listing paired devices (GET /api/v1/harvester-devices?connector=<name>) never returns token material.

The auth check

Every push route authenticates the device before doing any work. The presented bearer is hashed and compared timing-safely against every non-revoked device row for that workspace and connector, so probing for valid tokens via timing isn’t viable. A successful sync stamps the device’s last_seen_at and last-sync stats.

Two behaviours matter for first-run and for native iOS clients:

• Migration ramp. Tokenless pushes are accepted only until the first device is paired for a connector. Once any device exists, a tokenless push is a hard 401 — so an attacker can’t bypass auth simply by omitting the header. This lets a fresh install start syncing (for example Granola) before going through pairing.
• Tailnet-trust path. Some native clients (such as the iOS HealthKit push) hold no bearer and have no pairing flow. For these, a tokenless push is the intended path — but only when the request comes from a trusted tailnet/loopback peer. The host does not trust a forwarded-for header, so the peer IP can’t be spoofed; a non-tailnet peer is rejected before any device lookup.

The HARVESTER_DEVICE_AUTH_REQUIRED environment flag (default off) forces strict enforcement: when on, any push without a valid bearer token is rejected, regardless of pairing history. Flip it on once all your devices have completed pairing.

Historical backfill

Pairing covers ongoing sync. To pull history there are two distinct mechanisms, and they shouldn’t be conflated.

Account-level pull (OAuth/API connectors)

For connectors the host can reach directly (Gmail, GitHub, Calendar, and so on), backfill is a cost-gated, signed pull: you preview an estimate (POST /api/v1/backfill/estimate), then start the job (POST /api/v1/backfill/start). The estimate mints a short-lived HMAC token bound to the exact workspace, connector, account, date range, and cost cap; start re-validates it against current state, so a drifted range or cost cap is rejected with a specific reason. Jobs are observable and controllable: list and poll them, cancel at the next batch boundary, or resume a job that paused because it hit its budget. Crashed jobs are recovered at boot and by a periodic stuck-job sweep.

Edge backfill control channel

When the data lives on a device (Apple Photos/Notes/Files, iMessage, Safari history), the host can’t pull it — so it asks the device to sweep its own store and push it back. The host writes a backfill_requests row; the desktop polls for it, claims it, sweeps locally, and pushes the historical items through the connector’s normal /<connector>/sync route carrying an X-Carabase-Backfill-Request-Id header so the host can attribute progress. The host records progress and finalizes the request, picking the right paired device for the connector.

Both paths are idempotent: every ingest derives a deterministic idempotency key so a retried or re-pushed batch never creates duplicates.

AI-session capture (default-OFF)

Edge harvesters also carry AI coding sessions (Claude Code, Codex, Gemini) into the mesh as project evidence. This is off by default and is never auto-enabled on pairing — you turn it on per workspace from the AI Sessions connector page.

When enabled, a session reaches the host two ways: the desktop pushes transcripts to POST /api/v1/ai-sessions/sync (authenticated as a harvester device), or an agent commits one directly via the carabase_commit_session MCP write tool. Both run through a single ingest core that gates on the default-OFF flag, resolves which project folio(s) the session belongs to, builds the record, scrubs secrets, and commits or enqueues it. The sync route returns a per-session skip envelope so the device can tell the difference between “capture disabled” and “empty payload” without reading host logs.

Each session binds to every matching project folio — by explicit project binding, by a repo link on the folio, or by a slug match on the git remote or working-directory name — and is committed as a pinned folio member. Secret scrubbing runs at the host as defense-in-depth on top of the device’s own redaction: credentials are stripped from the git remote and from the body before it’s stored. A one-line summary is synthesized at commit time to drive the desktop work map (GET /api/v1/sessions/feed).

The host side of this contract is live. The shipping desktop sidecar that streams transcripts is the open work; its wire shape is published in the desktop sidecar contract.

How connectors describe themselves

Edge harvesters are self-describing: an edge connector declares its edge auth type alongside the human-facing setup details — a setup summary, its push endpoint, and how often it syncs. The Admin SPA’s Edge Harvesters tab and the Desktop pairing UI render directly from those fields, so there’s no hardcoded per-connector UI — pairing a new edge connector works the moment it’s available in your Carabase Host.

Security recap

• Push tokens are stored sha256-only, returned in plaintext exactly once, and compared timing-safely.
• Tailscale + RLS are the primary boundaries; rate limiting is defense-in-depth. Every query filters explicitly by workspace.
• Backfill HMAC tokens are bound to workspace, connector, account, range, and cost cap, and re-validated against current state at start.
• AI-session capture is default-OFF and never auto-enabled; secrets are scrubbed at the host on top of edge redaction.
• All ingest is idempotent — retried or re-pushed batches never duplicate.