# Deploying Immiscible

Source: https://immiscible.fly.dev/docs/guides/deploy

# Deploying Immiscible

One Node 22 process, no runtime dependencies, one SQLite file. This page
covers the image, the three ways to run it, backups (online copies and
Litestream), the health endpoints, logs, shutdown, smoke testing a live
deployment, the demo seed, and every environment variable the code reads.

With `DATABASE_URL=postgres://...` the server runs on Postgres instead;
see [postgres.md](https://immiscible.fly.dev/docs/guides/postgres.md), and [self-host.md](https://github.com/efr7-7/immiscible/blob/main/docs/self-host.md) for the
Helm chart.

Operator commands on this page (`backup`, `restore`, `integrity`, `db`,
`seed-demo`, `rekey`) belong to `immiscible-server`, the operator CLI in
this repository. In a checkout, run it as `npm run admin -- <command>`;
inside the image, as `node src/cli/immiscible.js <command>`. It is not the
developer CLI (`npx immiscible login`, `init`, `doctor`), which talks to a
server rather than running one.

## The image

`Dockerfile` builds a two-stage image on `node:22-alpine`:

- runs as an unprivileged user with a fixed uid and gid (10001), never root;
- the code under `/app` is owned by root and cannot be changed by the
  process; the only writable path is the `/data` volume, so the root
  filesystem can be mounted read-only (`docker run --read-only --tmpfs /tmp`,
  or `read_only: true` in compose);
- `HEALTHCHECK` calls `/readyz` on `$PORT`;
- `STOPSIGNAL SIGTERM`, and the process drains on it (see Shutdown);
- no `npm install` step: nothing is downloaded at build or run time.

### Upgrading the data volume

Images before this one created the user without a fixed uid. A volume
written by an older image is owned by that uid, and the new user (10001)
cannot write to it. Once, before starting the new image:

```sh
docker compose run --rm --user root --entrypoint sh immiscible -c "chown -R 10001:10001 /data"
# Fly: fly ssh console -C "chown -R 10001:10001 /data"
```

## Three ways to run it

### docker compose (one host, your VPC)

```sh
cp .env.example .env        # fill in the required values below
docker compose up -d                                          # the app on 127.0.0.1:8787
docker compose --profile https --profile litestream up -d     # production
```

| Profile | Adds | Needs |
| --- | --- | --- |
| (none) | the app, bound to `127.0.0.1:${PORT}` | `.env` |
| `https` | Caddy on ports 80 and 443 with automatic HTTPS (Let's Encrypt), HTTP/3, compression, unbuffered streaming | `DOMAIN` pointing at this host; `PUBLIC_URL=https://$DOMAIN` |
| `litestream` | continuous replication to S3-compatible storage, and restore-on-empty before the app starts | the `LITESTREAM_*` variables |

Needs Docker Compose 2.20 or later (optional `depends_on`, used so the app
waits for a Litestream restore only when that profile is on).

The app container runs read-only, with every Linux capability dropped and
`no-new-privileges`, `init: true` for signal handling, and a 30 second stop
grace period. Caddy waits until the app reports healthy.

### Fly.io

`fly.toml` runs one machine with a volume at `/data` and checks `/readyz`.
Fly snapshots volumes daily; for a recovery point better than a day, run
Litestream (below) or a scheduled `immiscible-server backup` shipped off the machine.
Keep to one machine: SQLite is a single writer. For billing, run
`node scripts/stripe-setup.mjs` and paste the `fly secrets set` line it prints
([setup-stripe.md](https://github.com/efr7-7/immiscible/blob/main/docs/setup-stripe.md)).

Deploys of the hosted service run from CI, not from a laptop: see
[Operating the hosted service](#operating-the-hosted-service).

### Regions

The hosted service runs as one deployment per region: **EU** (Fly app
`immiscible`, `fra`, `fly.toml`) and **US** (Fly app `immiscible-us`, `iad`,
`fly.us.toml`). Each region has its own machine, volume, SQLite file, master
key, backup bucket and Ed25519 signing key. No customer data is copied
between regions; a workspace lives where its owner created it.

- **Which region a deployment is:** `IMMISCIBLE_REGION` (`eu` or `us`). Unset,
  the deployment is a single region of its own (self-hosting, development)
  and none of this applies.
- **The hosts:** `config/regions.json` lists each region with its URL, Fly
  app and backup prefix. When the immiscible.ai domain arrives, change the
  two `url` values there, or set `IMMISCIBLE_REGION_URLS=eu=https://eu.immiscible.ai,us=https://us.immiscible.ai`
  on both apps, and nothing else.
- **Sign-up:** the sign-up page says where the new workspace's data will live
  and links to the other region's own sign-up page; a sign-up posted for
  another region is refused with that region's URL (`other_region`), so no
  account or workspace is made in the wrong place. A region is offered only
  when it is live (`"live": true` in `config/regions.json`, or
  `IMMISCIBLE_REGIONS_LIVE=eu,us`).
- **Where it shows:** `/trust`, `trust.json` (`deployment.region`, `regions`)
  and the console under Settings, General, Data region.
- **Backups:** each region writes under its own prefix (`immiscible/backups`
  for EU, `immiscible-us/backups` for US) in its own bucket. Each manifest
  records its region, and a restore refuses a backup from another region
  unless `--allow-other-region` is given (only to move a workspace at its
  owner's written request).
- **Receipt keys:** a key minted in a region has the region in its id
  (`eu-...`, `us-...`); a key minted before regions keeps the id it was
  published under. Each region fetches the other's public keys from
  `/.well-known/immiscible-keys.json?scope=own` every six hours and
  publishes them beside its own, each tagged with its `region`, so a
  receipt from either region verifies offline against either key set.
  `POST /v1/verify` on one region recognises a genuine receipt from the
  other and answers with that region's verify URL (`otherRegion: true`):
  single use and revocation are kept where the receipt was issued, and the
  receipt is never forwarded.

#### Setting up the US region

Approved by the founder on 7 October 2026. That day the app, the 3 GB volume
and the Tigris bucket were created, and the founder set the secrets with
`scripts/ops/set-us-region-secrets.sh` (a new master key via the clipboard
for the password manager, the Resend key read hidden). The launch itself,
the first deploy and `"live": true`, is postponed until the founder says go;
no machine runs yet. The commands below are the record and the rest of the
steps.

```sh
fly apps create immiscible-us
fly volumes create immiscible_us_data --app immiscible-us --region iad --size 3
fly storage create --app immiscible-us --name immiscible-us-backups     # Tigris; sets BUCKET_NAME and AWS_* secrets on immiscible-us only
fly secrets set --app immiscible-us \
  IMMISCIBLE_MASTER_KEY="$(node -e "console.log(require('crypto').randomBytes(32).toString('base64'))")" \
  PUBLIC_URL=https://immiscible-us.fly.dev \
  RESEND_API_KEY=... MAIL_FROM=... FOUNDER_EMAIL=...
# Optional, the same values as the EU app: STRIPE_*, GOOGLE_*, MICROSOFT_*, SLACK_* (each OAuth app needs
# https://immiscible-us.fly.dev/... added as a redirect URI), plus any operator apps from the tables below.
fly deploy --config fly.us.toml
node scripts/smoke.mjs https://immiscible-us.fly.dev
```

The US master key is new and separate: never reuse the EU key. Keep it in the
password manager beside the EU one. Then make the US region live and
redeploy both, so each offers the other at sign-up and fetches its keys:

```sh
# in config/regions.json set "live": true for us, commit, then
fly deploy                                  # EU
fly deploy --config fly.us.toml             # US
# optional, so the EU key id also says its region:
fly ssh console --app immiscible -C "node src/cli/immiscible.js rotate-signing-key --reason 'region key id'"
```

Estimated cost of the US region, at Fly's published prices on 7 October 2026
(iad is Fly's base price):

| Item | Monthly |
| --- | --- |
| One `shared-cpu-2x` machine with 1 GB (512 MB included, 512 MB more at $6 a GB) | about $7.40 |
| 3 GB volume at $0.15 a GB | $0.45 |
| Volume snapshots (first 10 GB a month free) | $0 |
| Tigris backups, under 1 GB of encrypted copies | under $0.10 |
| Outbound data at $0.02 a GB, a few GB | under $0.20 |
| Shared IPv4 and IPv6 (no dedicated IPv4) | $0 |
| **Total** | **about $8 a month (under $10)** |

A dedicated IPv4 address adds $2 a month; a custom certificate for
`us.immiscible.ai` is free up to Fly's certificate allowance.

### Render

`render.yaml` is the same shape: a Docker web service with a persistent disk
at `/data` and `/readyz` as its health check.

## Backups and restore

The evidence ledger lives in the database, so a backup is evidence too.

### Online copy

```sh
node src/cli/immiscible.js backup backups/immiscible-2026-10-04.db
```

`VACUUM INTO` takes a transactionally consistent copy while the server keeps
serving, then the copy is integrity-checked before the command reports
success. A backup never overwrites a file. `npm run backup` (scripts/backup.mjs)
does the same into `$DATA_DIR/backups` and keeps the newest `BACKUP_KEEP`
(default 14). Copy backups off the machine.

### Integrity check

```sh
node src/cli/immiscible.js integrity                # the live database
node src/cli/immiscible.js integrity backups/x.db --json
```

Checks the file (`PRAGMA integrity_check`), foreign keys, that the schema is
not newer than this build, and every workspace's evidence chain: sequence,
links, and each record's hash recomputed from its content. Exit code 1 on any
problem.

### Restore

```sh
# stop the server first
node src/cli/immiscible.js restore backups/immiscible-2026-10-04.db
```

The backup is integrity-checked first; the database it replaces is kept as
`immiscible.db.pre-restore-<time>`; the old `-wal` and `-shm` are removed so they
cannot be replayed onto the restored copy. If the live `-wal` is not empty
the command assumes a server is still running and refuses (`--force` if you
are sure). Start the server afterwards: it migrates an older copy forward.

### Litestream (continuous)

`litestream.yml` streams the database to any S3-compatible bucket about once
a second, snapshots daily, and keeps 30 days. With the `litestream` compose
profile, an empty volume is restored from the newest replica before the app
starts, so recovering a lost host is: provision a new one, copy `.env`, run
`docker compose --profile https --profile litestream up -d`.

Point-in-time restore by hand:

```sh
docker compose run --rm litestream restore -config /etc/litestream.yml \
  -timestamp 2026-10-04T09:00:00Z -o /data/restored.db /data/immiscible.db
```

Give the bucket versioning or object lock, and the key write access without
delete, where the provider allows it.

### Built-in off-machine backups (Fly and anywhere without Litestream)

On Fly a volume attaches to one machine at a time, so a scheduled machine
cannot mount `immiscible_data` while the app runs. The server backs itself up
instead when `IMMISCIBLE_BACKUP_BUCKET` (or Fly's `BUCKET_NAME`) and
credentials are set:

```sh
fly storage create --app immiscible       # Tigris; sets BUCKET_NAME and AWS_* secrets
```

Every `IMMISCIBLE_BACKUP_INTERVAL_MINUTES` (default 60), in a worker thread
so requests are not held up: `VACUUM INTO` a copy, run the integrity check
and recompute every workspace's evidence chain on it, gzip it, encrypt it
with AES-256-GCM under a key derived from `IMMISCIBLE_MASTER_KEY`, upload it
with a manifest (sizes, SHA-256 of both files, schema, every chain's head),
and never delete anything: old copies are expired by a lifecycle rule on the
bucket, or pruned by the operator with a key of their own (`immiscible-server
backups prune`), so the server's key needs no delete permission (see
[Backups a stolen key cannot delete](#backups-a-stolen-key-cannot-delete)).
A run is skipped if the disk lacks room for the copy. `/readyz` reports the
last good backup's age; `backup_failed` is logged at error level, and a
failed or stale backup emails the operator (see
[Backup alerts](#backup-alerts)).

```sh
immiscible-server backups                          # what is in the bucket
immiscible-server backup --upload                  # one now, from the machine
immiscible-server restore --from-bucket latest --to /data/drill.db   # a restore drill: fetched, decrypted, every chain checked; live data untouched
immiscible-server restore --from-bucket latest     # the real thing (server stopped)
```

In a hosted region the prefix is that region's own (see Regions), and a
restore refuses a backup whose manifest names another region.

A restore checks the download's hash, the GCM tag, the decrypted file's hash,
the integrity of the file and every evidence chain, and that each chain ends
exactly where the manifest says, before anything touches the live database.
Without the master key the backups cannot be read: keep it in a password
manager. Recovery point: up to one interval (60 minutes by default) of
writes. Recovery time: the download and checks (about a minute per GB) plus a
restart.

A drill (`--to FILE`) records its result, pass or fail, with how long it took,
as `restore-drill.json` beside the backups. The server reads it with each
backup, and `trust.json` (`backup.lastRestoreDrill`), `/readyz` and Security
posture in the console report it. Delete the scratch file afterwards.

Database settings the app applies on every open: WAL journal, `synchronous =
NORMAL`, foreign keys on, `busy_timeout = 5000` ms. The app never truncates
the WAL itself, which is what Litestream requires.

## Health, readiness, logs, shutdown

| Endpoint | Answers |
| --- | --- |
| `GET /healthz` | 200 while the process is alive: `{ ok, version, commit, builtAt, uptimeSec, backup }`; `backup` is `ok`, `stale` or `off` and never changes the status code |
| `GET /version` | `{ version, commit, builtAt }`: the commit this image was built from, or `dev` |
| `GET /readyz` | 200 when the database answers and the schema matches this build; 503 while draining, or if the schema is behind |

Logs: with `LOG_FORMAT=json` (the production default) every line on stdout is
one JSON object with `t` (ISO time), `level` and either `msg` or request
fields (`rid`, `method`, `path`, `status`, `ms`, `ip`). Start-up, config
errors, shutdown and crashes are JSON lines too. `IMMISCIBLE_LOG=off` turns off
request lines only.

Shutdown: on SIGTERM or SIGINT the process marks itself not ready (so a load
balancer stops sending traffic), stops accepting connections, closes idle
keep-alive connections, lets in-flight requests and streams finish, flushes
every workspace, logs `stopped`, and exits 0. If that takes longer than
`IMMISCIBLE_SHUTDOWN_TIMEOUT_MS` (default 25 s), open connections are cut and it
exits anyway. An uncaught exception or unhandled rejection is logged and
starts the same orderly stop with exit code 1, so the supervisor restarts it.

Keep-alive: idle connections stay open 65 s (headers timeout 66 s), longer
than the idle timeout of the usual proxies (Caddy, Fly, cloud load balancers
at 60 s), so the server never closes a pooled socket just as the proxy reuses
it. If your proxy keeps upstream connections longer than 65 s, lower its
idle timeout.

## Smoke testing a live deployment

```sh
node scripts/smoke.mjs https://immiscible.fly.dev --keep
```

Walks the whole product through the public API: sign-up and email
verification, a workspace, a second owner, a provider, a service token, an
agent from a blueprint with standing and allowlists, a gateway model call, a
payment approved by the other owner, the card charge for it through the
issuer webhook, an MCP tool call, a freeze and lift, a drill, the trace view,
and the evidence bundle verified offline against keys pinned at the start.
The same journey runs in CI against a server booted on a temporary database
(`test/e2e-journey.test.js`).

Email links (verification, invitation) need mailbox access. Outbox mode
returns the invitation link in the API answer; otherwise run the smoke on the
server host and pass the database file:

```sh
fly ssh console -C "node scripts/smoke.mjs http://127.0.0.1:8787 --db /data/immiscible.db"
```

Steps that cannot run are reported as skipped, never as passed. The workspace
is deleted at the end unless `--keep`. Variables: `SMOKE_URL` (instead of the
argument), `SMOKE_PROVIDER_KEY` (an OpenAI key to connect a real provider),
`SMOKE_MCP_URL` and `SMOKE_MCP_SECRET` (a real MCP server; against a loopback
URL the script starts its own), `SMOKE_EMAIL_DOMAIN` (default `smoke.test`).
`--json` prints a machine-readable report.

## The demo workspace

```sh
node src/cli/immiscible.js seed-demo --password 'choose-one'
```

Creates Amethyst, a London design and research studio: Jules Moreau
(`jules@amethyst.example`, owner, the demo login), Priya Shah (second owner) and
Sam Okafor (member); five agents with sponsors, kill owners, purposes, end
dates and mandates; model calls through the gateway (the labelled mock, so
no provider key is needed); about twenty payment requests that are allowed,
held or refused with reasons; approvals decided by Priya; a freeze and lift;
a drill; a signed checkpoint; single sign-on set up for amethyst.example
against a sample identity provider, not required, so the demo logins sign in
with a password. `--crypto` adds a sixth agent that pays x402 services in
USDC on Base; the default demo has no crypto in it. It writes to the database
the server would open and is idempotent. Without `--password` (or `SEED_PASSWORD`) a password is
generated and printed.

## Operating the hosted service

How the hosted service at `immiscible.fly.dev` (Fly app `immiscible`, region
`fra`, one machine) is changed, watched and recorded. The scripts named here
live in `scripts/ops`; each one records who ran it, what, when and why in the
operator log before it acts.

### Deploying

**The normal path is CI.** A push to `main` runs `.github/workflows/ci.yml`
(tests, preflight, image build and scan, `npm audit`, CodeQL, TruffleHog,
gitleaks). Only when that whole run succeeds does
`.github/workflows/deploy.yml` deploy the same commit:

```sh
flyctl deploy --remote-only --ha=false --build-arg GIT_SHA=<commit> --build-arg BUILT_AT=<time>
```

The `Dockerfile` turns the two build arguments into `IMMISCIBLE_COMMIT` and
`IMMISCIBLE_BUILT_AT`; `/healthz`, `/version`, `trust.json` and `/trust`
report them. The workflow then reads `/version` until the live service
reports the commit it deployed, and fails if it never does. The Actions run
history and `fly releases -a immiscible` are the deploy log. Anyone can check
what is live:

```sh
curl -s https://immiscible.fly.dev/version     # {"version":"4f1c2a9e0b7d","commit":"4f1c2a9e0b7d...","builtAt":"2026-10-07T10:12:00Z"}
```

The workflow needs one repository secret, `FLY_API_TOKEN`: a deploy token
limited to the one app. To set or rotate it (yearly; the script suggests a
one-year expiry):

```sh
scripts/ops/set-fly-deploy-token.sh --reason "first CI deploy token"
```

It asks you to run `fly tokens create deploy -a immiscible` in another
terminal, reads the token at a hidden prompt and pipes it straight into
`gh secret set FLY_API_TOKEN`. The token is never echoed, written to a file
or put on a command line.

**A laptop deploy is break-glass only**: GitHub Actions is down, or an
incident cannot wait for the pipeline.

```sh
scripts/ops/breakglass-deploy.sh --reason "Actions outage; fix for the 503s on /v1/actions"
```

It refuses a working tree with uncommitted changes, runs the tests (unless
`--skip-tests`, which is recorded), records the reason in the operator log
(or, if the server cannot be reached, in a GitHub issue labelled
`operator-log`), and deploys with the same build arguments as CI. Push the
commit and open a pull request within one working day.

### Branch protection

Branch protection and rulesets are not available for a private repository
on GitHub's free plan; the API answers "Upgrade to GitHub Pro or make this
repository public". The options:

- **GitHub Team** (an organisation, about $4 per user a month): rulesets on
  `main` requiring a pull request and the `test`, `secrets`, `gitleaks` and
  `codeql` checks, no force pushes or deletion. The repository then belongs
  to the company and organisation-wide two-factor can be required. GitHub
  Pro on the personal account (about $4 a month) also allows protection on
  private repositories.
- **Make the repository public**: protection and rulesets become free, and so
  do Actions minutes. The code would be visible to everyone.

Until then there is a local hook, which is **not a real control**:

```sh
git config core.hooksPath scripts/hooks    # once per clone
```

`scripts/hooks/pre-push` runs `npm test` before a push to `main` and refuses
the push if it fails. `git push --no-verify`, another clone or the web editor
skip it, and nothing on GitHub enforces it. What does hold today is the
deploy gate: nothing reaches production through CI unless the whole ci run
passed.

### Watching from outside

`.github/workflows/uptime.yml` runs every hour on GitHub's runners,
outside Fly.io (`scripts/ops/uptime-check.mjs`):

- `/healthz` must answer 200 with `ok: true` and a `backup` that is not
  `stale`;
- the console sign-in page `/login` must answer 200 with a page.

After two failed runs in a row it opens an issue labelled `incident`,
assigned to the repository owner; GitHub emails the assignee (keep email
notifications on for the repository). While it stays down each run comments
on the issue, and the first good run comments and closes it.

Limits, plainly. GitHub's schedule is best effort: runs start late, by
several minutes or more at busy times, and some are dropped, so with an
hourly run expect detection within roughly two hours. A GitHub Actions
outage is also an outage of this monitor. Each run is billed as at least one
minute, so hourly is about 720 of the 2,000 minutes a free private
repository includes; with a spending limit of zero an overflow would also
stop CI and deploys. For faster detection, make the repository public
(Actions are then free) or move to a plan with more minutes and set the
schedule to `*/5`, or add a free external monitor alongside this one.

### Backup alerts

A failed backup, or no good backup for `IMMISCIBLE_BACKUP_STALE_MINUTES`
(default 120, two missed hourly runs), emails `OPERATOR_EMAIL` (else
`FOUNDER_EMAIL`) through the configured mail sender (Resend on the hosted
service): at most one email an hour about failures, and one every six hours
while it stays stale. `/healthz` says `"backup": "stale"`, which fails the
uptime check, and `/trust` shows the last good backup and the build.

```sh
fly secrets set -a immiscible OPERATOR_EMAIL=you@example.com
```

### Security log

Besides stdout, the server keeps these in the `security_log` table:

| Kind | What | Kept |
| --- | --- | --- |
| `request` | method, path, status, time taken, address, request id | `IMMISCIBLE_REQUEST_LOG_DAYS` (30; 0 keeps none) |
| `auth` | sign-ins, failed sign-ins, second factors, sessions | `IMMISCIBLE_SECURITY_LOG_DAYS` (365) |
| `admin` | every other audited admin action: what, who, from where | `IMMISCIBLE_SECURITY_LOG_DAYS` (365) |
| `error` | every error-level log line: message, alert name, the error's first line | `IMMISCIBLE_SECURITY_LOG_DAYS` (365) |

Never kept: query strings, bodies, headers, prompt or answer text, audit
details (they stay in the workspace's own audit log), stack traces, or
anything shaped like a key or token; long opaque path segments become
`:token`. The table is in the database, so the hourly encrypted backups carry
it off the machine; there is no log shipper and no extra vendor. It holds
user ids and addresses, so the retention policy covers it. Export, for an
auditor or an investigation:

```sh
immiscible-server logs export --out /data/security-log-2026-10.jsonl --since 2026-10-01 --until 2026-11-01 --reason "monthly evidence"
# then copy it off: fly ssh sftp get /data/security-log-2026-10.jsonl
```

One JSON object a line; the command prints the file's SHA-256, and the export
itself is recorded in the operator log.

### Operator actions

Every `immiscible-server` operations command (`backup`, `backups`,
`restore`, `integrity`, `rebuild-totals`, `db`, `seed-demo`, `rekey`,
`rotate-signing-key`, `incident`, `logs`) writes a row to the `operator_log`
table before it acts: who (`--by`, else `IMMISCIBLE_OPERATOR`, else the
login name), what (the command and its arguments, with anything secret
redacted), when, where (`FLY_MACHINE_ID` or the host name) and why
(`--reason`). A restore of the live database writes it afterwards, into the
restored copy. In production a command that changes data refuses to run
without `--reason`. `scripts/backup.mjs` and `scripts/smoke.mjs --db` record
themselves too, and the shell scripts in `scripts/ops` record through
`immiscible-server note` over `fly ssh console`.

The table is append-only: triggers refuse any change or deletion, and each
row carries the SHA-256 of the one before, so editing the raw file shows.

```sh
immiscible-server operator-log                 # who did what, when and why
immiscible-server operator-log --verify        # recompute the chain
immiscible-server operator-log --json --since 2026-10-01
immiscible-server note --what "rotated Resend key" --reason "yearly rotation"
```

A shell on the machine, with the reason recorded first:

```sh
scripts/ops/breakglass.sh --reason "quarterly restore drill"
```

**What cannot be captured.** Inside an SSH session, anything not done
through `immiscible-server` (sqlite3, editing files) leaves no record; Fly.io
gives customers no transcript of SSH sessions. Actions in the Fly.io or
Tigris dashboards, who read a secret, and anything Fly.io staff do are not
recorded here either. The record is honour-based for a person with a shell;
the chain makes later tampering visible, not impossible.

**Fly.io's own records.** Fly.io does not offer customers an organisation
audit log (checked 7 October 2026: nothing in `flyctl` or the documentation,
and Fly's community forum says it is not yet available). What it does expose
is saved, dated and hashed, by:

```sh
scripts/ops/export-fly-activity.sh            # ops-evidence/fly-<date>/ (not committed)
```

`fly releases --json --image` (each release: who triggered it, when, the
image), `fly machine list --json`, `fly secrets list --json` (names, digests
and when each was set, never values), `fly tokens list` and `fly orgs show`.
Run it monthly and after any incident, and keep the folder with the
compliance evidence.

### Backups a stolen key cannot delete

The server never deletes a backup, so its key needs no delete permission.
What Tigris offers (checked 7 October 2026 against its documentation and
CLI 3.15.0):

- **Scoped access keys with IAM policies.** Supported actions include
  `s3:PutObject`, `s3:GetObject`, `s3:ListBucket` and, separately,
  `s3:DeleteObject`, so a key can write without deleting.
- **Lifecycle rules** that expire objects after a number of days, by prefix.
- **Soft delete**: deleted objects stay recoverable for 7 to 90 days.
- **Snapshots** of a whole bucket, immutable once taken.
- **Object lock and bucket versioning are not documented** among Tigris's
  supported S3 operations or IAM actions. For write once, read many storage,
  use a second provider that has it (AWS S3, Backblaze B2 and Cloudflare R2
  all do).

One gap remains even with a write-only key: `s3:PutObject` can overwrite an
object under an existing name. Backups have unique names and the server
never reuses one, but a stolen key could. Soft delete does not cover an
overwrite; snapshots and a locked second copy do.

`scripts/ops/backup-hardening.sh` holds the exact commands. Run with no
arguments it prints the plan and changes nothing; each step asks for "yes"
and a reason. In order:

```sh
npm install -g @tigrisdata/cli && tigris login

# 1. A key that can write and read the backup prefix and never delete, set as
#    IMMISCIBLE_BACKUP_ACCESS_KEY_ID / _SECRET_ACCESS_KEY on the app.
#    Policy: Allow s3:PutObject, s3:GetObject, s3:AbortMultipartUpload,
#    s3:ListMultipartUploadParts on <bucket>/immiscible/backups/*, and
#    s3:ListBucket, s3:ListBucketMultipartUploads on <bucket>; Deny
#    s3:DeleteObject, s3:DeleteObjectVersion, s3:DeleteBucket,
#    s3:PutLifecycleConfiguration, s3:PutBucketAcl, s3:PutObjectAcl.
tigris iam policies create immiscible-backup-write-only --document policy.json
tigris access-keys create immiscible-server-backups --env <private temp file> --for aws
tigris access-keys attach-policy <key id> --policy-arn <policy arn>
#    then the id and secret piped into: fly secrets import -a immiscible
scripts/ops/backup-hardening.sh scoped-key --reason "..."

# 2. Prove it: one backup with the new key.
scripts/ops/backup-hardening.sh verify --reason "..."

# 3. Expiry, now that the server does not prune.
tigris buckets lifecycle create <bucket> --prefix immiscible/backups/ --expire-days 35

# 4. Deleted objects recoverable for 30 days.
tigris buckets set <bucket> --soft-delete enable --retention-days 30

# 5. Take the key that can delete off the server (fly storage create set it
#    as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY). Keep one in the password
#    manager for `immiscible-server backups prune` and emergencies.
fly secrets unset -a immiscible AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

# 6. IRREVERSIBLE, the founder's decision: a second, locked copy at another
#    provider, in the EU. Compliance mode means nobody, the account's root
#    user included, can delete a copy or shorten its lock for 35 days, and
#    object lock can never be turned off on that bucket.
aws s3api create-bucket --bucket immiscible-backups-locked-eu --region eu-central-1 \
  --create-bucket-configuration LocationConstraint=eu-central-1 --object-lock-enabled-for-bucket
aws s3api put-object-lock-configuration --bucket immiscible-backups-locked-eu --region eu-central-1 \
  --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":35}}}'
```

Step 6 also needs code that does not exist yet (each run uploading to a
second bucket), an IAM user for it with put, get and list only, and AWS
added to the sub-processors with the notice the DPA promises.

Pruning by hand, with the operator's own key (never set on the server):

```sh
IMMISCIBLE_BACKUP_PRUNE_ACCESS_KEY_ID=... IMMISCIBLE_BACKUP_PRUNE_SECRET_ACCESS_KEY=... \
  immiscible-server backups prune --reason "lifecycle rule not yet set" [--dry-run]
```

### Scanning

- `.github/dependabot.yml`: weekly update proposals for npm (the server and
  each published package), the GitHub Actions in the workflows (all pinned
  by commit SHA) and the Docker base image. Dependabot alerts and automated
  security fixes are on in the repository settings.
- `ci.yml`: `npm audit --omit=dev` (any advisory fails), a Trivy scan of the
  built image that fails on a critical vulnerability with a fix available,
  gitleaks on every push beside TruffleHog, and a weekly scheduled run.

### Operations settings

| Variable | Default | Meaning |
| --- | --- | --- |
| `IMMISCIBLE_COMMIT` | set by the image | The git commit the image was built from (Docker build argument `GIT_SHA`); `dev` when unset. Reported by `/healthz`, `/version`, `trust.json`, `/trust`. |
| `IMMISCIBLE_BUILT_AT` | set by the image | When the image was built (build argument `BUILT_AT`). |
| `OPERATOR_EMAIL` | `FOUNDER_EMAIL` | Where operational alerts go: a failed or stale backup. |
| `IMMISCIBLE_BACKUP_STALE_MINUTES` | `120` | No good backup for this long: `/healthz` reports `stale` and the operator is emailed (minimum 15). |
| `IMMISCIBLE_SECURITY_LOG_DAYS` | `365` | Days sign-in, admin and error lines stay in the security log. |
| `IMMISCIBLE_REQUEST_LOG_DAYS` | `30` | Days request lines stay in the security log; `0` keeps none. |
| `IMMISCIBLE_OPERATOR` | the login name | Who the operator log records, unless `--by` names someone. |
| `LOGNAME` | (the shell's) | Used for the operator's name when `USER` is unset. |
| `FLY_MACHINE_ID` | set by Fly | Recorded as where an operator command ran. |
| `IMMISCIBLE_OPERATOR_REASON` | `scheduled local backup` | The reason `scripts/backup.mjs` records. |
| `IMMISCIBLE_BACKUP_PRUNE_ACCESS_KEY_ID` | (unset) | The operator's own bucket key for `backups prune`, which may delete. Never set on the server. |
| `IMMISCIBLE_BACKUP_PRUNE_SECRET_ACCESS_KEY` | (unset) | Its secret. |

## Environment variables

Every variable the code reads. Required in production are marked **required**;
the server refuses to start without them when `NODE_ENV=production`.
`test/e2e-ops.test.js` fails if code reads a variable this table does not
list.

### Core

| Variable | Default | Meaning |
| --- | --- | --- |
| `NODE_ENV` | (unset) | `production` turns on strict config checks, secure cookies, JSON logs, email verification. |
| `IMMISCIBLE_MASTER_KEY` | dev: generated into `DATA_DIR` | **required.** 32 random bytes, base64. Seals every stored secret (provider keys, signing keys, SSO client secrets, connection tokens, vault fields) with AES-256-GCM, each bound to its workspace. Rotate it with the procedure below, never by just changing it. |
| `IMMISCIBLE_MASTER_KEY_PREVIOUS` | (unset) | During a rotation: the old master key (or several, comma separated). Values it sealed still open; everything new is sealed with `IMMISCIBLE_MASTER_KEY`. Remove it once `immiscible-server rekey` reports nothing left. |
| `PUBLIC_URL` | `http://localhost:$PORT` | **required.** The URL people and agents use; must be https in production. |
| `MCP_REGISTRY_NAME` | from `server.json` when it lists this server, else the host in reverse DNS | The name the MCP server card at `/mcp/server-card` gives, for example `ai.immiscible/immiscible`. |
| `MCP_REGISTRY_AUTH` | unset | The MCP Registry's HTTP domain proof (`v=MCPv1; k=ed25519; p=...`), served at `/.well-known/mcp-registry-auth` for `mcp-publisher login http`. |
| `OPENAI_APPS_CHALLENGE` | unset | The domain token OpenAI's app submission shows, served as plain text at `/.well-known/openai-apps-challenge` while it is set. |
| `IMMISCIBLE_MODE` | `cloud` | `cloud` (multi-tenant, sign-ups) or `selfhost` (one workspace, provider keys from the environment). |
| `BRAND_CONFIG` | `brand.config.json` | Path to another brand file. |
| `PORT` | `8787` | Listen port. |
| `HOST` | `0.0.0.0` | Listen address. |
| `TRUST_PROXY` | `true` in production | Trust the last `X-Forwarded-For` hop (set when behind Caddy, Fly or a load balancer). |
| `FLY_APP_NAME` | set by Fly | With it, the client address is read from Fly's `Fly-Client-IP`. |
| `IMMISCIBLE_REGION` | (unset) | The hosted region this deployment is (`eu`, `us`), from `config/regions.json`. Unset: a single region of its own. See Regions. |
| `IMMISCIBLE_REGION_URLS` | from `config/regions.json` | Each region's host, as `eu=https://...,us=https://...`. Change it when the domain moves. |
| `IMMISCIBLE_REGIONS_LIVE` | from `config/regions.json` | Which regions are live (offered at sign-up, keys fetched), as `eu,us`. |

#### Rotating the master key

1. Make a new key: `node -e "console.log(require('crypto').randomBytes(32).toString('base64'))"`. Keep the old one; you need both.
2. Take a backup (`immiscible-server backup <file>`) and stop the server.
3. Re-seal every stored secret under the new key, with both keys supplied:

   ```bash
   IMMISCIBLE_MASTER_KEY=<new> IMMISCIBLE_MASTER_KEY_PREVIOUS=<old> immiscible-server rekey
   ```

   It lists, table by table, what it re-sealed. Sign-in and connection flows in progress (they live ten minutes) are cleared. If anything could not be opened it says so and exits non-zero; nothing is deleted.
4. Start the server with `IMMISCIBLE_MASTER_KEY=<new>` and, to be safe, `IMMISCIBLE_MASTER_KEY_PREVIOUS=<old>`. Both keys open values, and new ones use the new key.
5. Once it runs well and `rekey` reported nothing left, remove `IMMISCIBLE_MASTER_KEY_PREVIOUS` and destroy the old key.

Receipts and checkpoints already signed stay verifiable: the signing key is re-sealed, not replaced.

#### Rotating the signing key

The Ed25519 key that signs receipts, checkpoints and erasure statements is separate from the master key. To replace it (on a schedule, or if it may be exposed):

```bash
IMMISCIBLE_MASTER_KEY=<the server's key> immiscible-server rotate-signing-key --reason "annual rotation"
```

The new key signs from then on; a running server picks it up within 30 seconds, no restart needed. The retired key stays in `/.well-known/immiscible-keys.json` (and `/.well-known/assay-keys.json`), marked `"retired": true`, so every receipt and checkpoint it signed still verifies. Every live workspace's evidence ledger gets a `signing_key_rotated` record naming both key ids.

#### Incidents on /status

`/status` shows the incident history from the database and 90 days of uptime the server measures on itself (`IMMISCIBLE_HEALTH_SAMPLES`). Write incidents on the server:

```bash
immiscible-server incident add --title "Approvals delayed" --impact minor --note "Looking at the queue."
immiscible-server incident update inc_1a2b3c4d5e --status monitoring --note "Fix deployed."
immiscible-server incident resolve inc_1a2b3c4d5e --note "Back to normal."
immiscible-server incident list
```

Impacts: minor, major, critical, maintenance. There is no web route that writes an incident.

### Data

| Variable | Default | Meaning |
| --- | --- | --- |
| `DATA_DIR` | `./data` | Directory holding `immiscible.db`, its WAL and the dev master key. |
| `DATABASE_FILE` | `$DATA_DIR/immiscible.db` | The SQLite file (`:memory:` in tests). |
| `DATABASE_URL` | (unset) | A `postgres://` URL selects the Postgres backend (postgres.md). Anything else, or the `pg` package missing, or the database unreachable, stops the server at start; it never falls back to SQLite. |
| `TEST_DATABASE_URL` | (unset) | Tests only: a `postgres://` URL makes every database the test suite opens a schema on that server instead of SQLite (`npm run test:postgres`, postgres.md). Ignored when `NODE_ENV=production`. |
| `BACKUP_KEEP` | `14` | Copies `scripts/backup.mjs` keeps. |
| `SEED_PASSWORD` | generated | Password for the Amethyst demo people (`seed-demo`). |
| `LITESTREAM_BUCKET` | (unset) | Bucket for the Litestream replica (compose `litestream` profile). |
| `LITESTREAM_PATH` | `immiscible` | Path inside the bucket. |
| `LITESTREAM_ENDPOINT` | AWS | S3-compatible endpoint (R2, B2, MinIO, Tigris). |
| `LITESTREAM_REGION` | `auto` | Bucket region. |
| `LITESTREAM_FORCE_PATH_STYLE` | `false` | `true` for MinIO and some S3-compatible stores. |
| `LITESTREAM_ACCESS_KEY_ID` | (unset) | Bucket credentials. |
| `LITESTREAM_SECRET_ACCESS_KEY` | (unset) | Bucket credentials. |
| `DOMAIN` | `localhost` | The hostname Caddy serves and gets a certificate for (compose `https` profile). |
| `IMMISCIBLE_BACKUP_BUCKET` | (unset) | Turns on the built-in scheduled, encrypted, off-machine backup to this S3-compatible bucket. Falls back to `BUCKET_NAME`, which `fly storage create` sets. |
| `IMMISCIBLE_BACKUP_ENDPOINT` | AWS | S3-compatible endpoint (Tigris: `https://fly.storage.tigris.dev`). Falls back to `AWS_ENDPOINT_URL_S3`. |
| `IMMISCIBLE_BACKUP_REGION` | `auto` | Bucket region. Falls back to `AWS_REGION`. |
| `IMMISCIBLE_BACKUP_ACCESS_KEY_ID` | (unset) | Bucket credentials. Falls back to `AWS_ACCESS_KEY_ID`. |
| `IMMISCIBLE_BACKUP_SECRET_ACCESS_KEY` | (unset) | Bucket credentials. Falls back to `AWS_SECRET_ACCESS_KEY`. |
| `IMMISCIBLE_BACKUP_PREFIX` | the region's `backupPrefix`, else `immiscible/backups` | Path inside the bucket. |
| `IMMISCIBLE_BACKUP_PATH_STYLE` | `false` | `true` for MinIO and stores that need the bucket in the path. |
| `IMMISCIBLE_BACKUP_INTERVAL_MINUTES` | `60` | How often the server backs itself up (minimum 5). The first run is two minutes after boot. |
| `IMMISCIBLE_BACKUP_KEEP_HOURLY` | `48` | Newest backups `immiscible-server backups prune` always keeps. The server itself never prunes. |
| `IMMISCIBLE_BACKUP_KEEP_DAILY` | `35` | Days for which `backups prune` also keeps the newest backup of the day; the same number of days is the bucket lifecycle rule and what `/trust` states. |
| `IMMISCIBLE_GROUP_COMMIT` | `true` | Gather each step's writes into one transaction (one commit instead of one per statement). |
| `IMMISCIBLE_DURABILITY` | `fsync` in production, else `os` | `fsync`: no answer leaves until the WAL holding its writes is flushed to the disk (one flush shared by every request waiting at that moment). `os`: committed to the operating system, which survives a process crash but not a machine crash. |
| `IMMISCIBLE_HOUSEKEEPING_DAYS` | `30` | Finished webhook deliveries and sent email are deleted after this many days, hourly. Never the evidence ledger or the audit log. |

### Behaviour and limits

| Variable | Default | Meaning |
| --- | --- | --- |
| `IMMISCIBLE_SIGNUPS_OPEN` | `true` | Allow self-service sign-up (cloud). |
| `IMMISCIBLE_REQUIRE_EMAIL_VERIFICATION` | `true` in production | Require a verified email. |
| `IMMISCIBLE_ALLOW_MOCK` | `true` | The deterministic mock answers for providers with no key, labelled everywhere. |
| `IMMISCIBLE_FORCE_MOCK` | `false` | Every call goes to the mock (tests, demos). |
| `IMMISCIBLE_SEED_DEMO` | `true` outside production | Seed the Northwind sample workspace on first boot. |
| `IMMISCIBLE_FX_REFRESH` | `true` (off for in-memory databases) | Fetch the European Central Bank's reference rates at boot and daily, so journal exports and budgets can be shown in pounds or euros from day one. Off, a converted export is refused until a rate is stored. |
| `LOG_FORMAT` | `json` in production, else `pretty` | Log format. |
| `IMMISCIBLE_LOG` | (on) | `off` stops per-request log lines. |
| `IMMISCIBLE_DEBUG` | (unset) | The CLI prints stack traces. |
| `USER` | (the shell's) | Who `rotate-signing-key` records as having rotated the key, unless `--by` names someone. |
| `NO_COLOR` | (unset) | The CLI prints without colour. |
| `IMMISCIBLE_SHUTDOWN_TIMEOUT_MS` | `25000` | How long a drain may take before open connections are cut. |
| `IMMISCIBLE_KEY_RPM` | `1200` | Requests a minute per gateway key. |
| `IMMISCIBLE_AUTH_RPM` | `10` | Sign-in attempts a minute per address. |
| `IMMISCIBLE_FORMS_PER_HOUR` | `20` | Public form submissions an hour per address. |
| `IMMISCIBLE_AGENT_RPM` | `60` | Action authorisations a minute per agent. |
| `IMMISCIBLE_HOOK_RPM` | `300` | Tool calls a minute per agent from the Claude Code hook (`tool.call` with a `claude-code` session), counted apart from other actions. |
| `IMMISCIBLE_WORKSPACE_RPM` | `12000` | Requests a minute per workspace, across its gateway keys and agents, so one tenant cannot take the machine from the others. |
| `IMMISCIBLE_MAX_INFLIGHT` | `256` | Requests in progress at once; past it the server answers 503 with `Retry-After: 1` at once instead of queueing. Health checks are never refused; a streaming answer stops counting once it starts. |
| `IMMISCIBLE_MAX_LAG_MS` | `500` in production, else `0` (off) | When the event loop stays this far behind (CPU bound, callers queueing in the socket buffers), new requests get the same immediate 503 until it recovers. One slow step does not trip it; two samples in a row do. |
| `IMMISCIBLE_OUTBOUND_TIMEOUT_MS` | `15000` | Deadline for any outbound call that does not set its own (mail, billing, sign-in discovery, directory sync, accounting, alerts). |
| `IMMISCIBLE_VERIFY_RPM` | `120` | Public receipt checks a minute per address. |
| `IMMISCIBLE_MAX_BODY_BYTES` | `33554432` | Largest request body on the routes that take large bodies (prompts, exports, webhooks); others are capped at 1 MB. |
| `IMMISCIBLE_RATE_LIMIT_STORE` | `auto` | `memory` or `sqlite` (shared between processes on one database). |
| `IMMISCIBLE_GATEWAY_STATE_STORE` | `auto` | `memory` or `sqlite`, for provider health and in-flight counters. |
| `IMMISCIBLE_MCP_PROXY_TIMEOUT_MS` | `15000` | Timeout for a call to an MCP upstream. |
| `IMMISCIBLE_MCP_PROXY_MAX_BYTES` | `1048576` | Largest MCP upstream answer. |
| `PLAUSIBLE_DOMAIN` | (unset) | Cookieless analytics on the public site. |

### Accounts and sign-in

| Variable | Default | Meaning |
| --- | --- | --- |
| `IMMISCIBLE_SESSION_IDLE_MINUTES` | `1440` | A session ends after this long without use. |
| `IMMISCIBLE_SESSION_MAX_DAYS` | `14` | A session ends this long after sign-in, whatever its use. |
| `WEBAUTHN_RP_ID` | the `PUBLIC_URL` host | Relying party id for passkeys. |
| `GOOGLE_CLIENT_ID` | (unset) | Sign in with Google (with `GOOGLE_CLIENT_SECRET`). |
| `GOOGLE_CLIENT_SECRET` | (unset) | Required when `GOOGLE_CLIENT_ID` is set. |
| `MICROSOFT_CLIENT_ID` | (unset) | Sign in with Microsoft (with `MICROSOFT_CLIENT_SECRET`). |
| `MICROSOFT_CLIENT_SECRET` | (unset) | Required when `MICROSOFT_CLIENT_ID` is set. |
| `MICROSOFT_TENANT` | `common` | Microsoft Entra tenant. |
| `OKTA_CLIENT_ID` | (unset) | Sign in with Okta (with `OKTA_CLIENT_SECRET` and `OKTA_ISSUER`). |
| `OKTA_CLIENT_SECRET` | (unset) | Required when `OKTA_CLIENT_ID` is set. |
| `OKTA_ISSUER` | (unset) | For example `https://your-org.okta.com/oauth2/default`. |

### Workspace security and retention

| Variable | Default | Meaning |
| --- | --- | --- |
| `IMMISCIBLE_IP_ALLOWLISTS` | (on) | `off` suspends every workspace's IP allowlist. An operator's switch for an owner who has locked themselves out; `/trust.json` then says allowlists are not enforced. |
| `IMMISCIBLE_DEV_VERIFY_DOMAINS` | `false` | Development and review only: verifying a single sign-on domain succeeds without a DNS TXT record, so domain capture and SSO enforcement can be tried on a laptop. The server refuses to start with it in production. |
| `IMMISCIBLE_RETENTION_PURGE` | `true` (`false` for an in-memory database) | Run the daily job that removes call logs and prompt metadata past each workspace's retention setting. Signed evidence is never purged. |
| `IMMISCIBLE_ERASURE_GRACE_DAYS` | `30` | Days between an owner deleting a workspace and its records being erased; owners can cancel until then. 0 to 30. |
| `IMMISCIBLE_ERASURE_JOB` | `true` (`false` for an in-memory database) | Run the hourly job that erases deleted workspaces whose grace period has ended, and emails the erasure certificate. |
| `IMMISCIBLE_HEALTH_SAMPLES` | `true` (`false` for an in-memory database) | Record the server's own health check every few minutes, for the uptime shown on `/status`. |
| `IMMISCIBLE_HEALTH_SAMPLE_MS` | `300000` | How often the health sample is taken, in milliseconds (at least 10000). |

Workspace session limits (`sessionIdleMinutes`, `sessionMaxHours`) can only tighten `IMMISCIBLE_SESSION_IDLE_MINUTES` and `IMMISCIBLE_SESSION_MAX_DAYS`, never loosen them. See [Security administration](https://immiscible.fly.dev/docs/guides/security-admin.md).

### Directory sync

SCIM needs no variables: each workspace makes its own token. Google Workspace and Microsoft Entra sync use an OAuth app the operator registers once ([setup-directory.md](https://github.com/efr7-7/immiscible/blob/main/docs/setup-directory.md)). Microsoft reuses `MICROSOFT_CLIENT_ID` and `MICROSOFT_CLIENT_SECRET`.

| Variable | Default | Meaning |
| --- | --- | --- |
| `GOOGLE_DIRECTORY_CLIENT_ID` | the Google sign-in client | Google Workspace sync (with `GOOGLE_DIRECTORY_CLIENT_SECRET`). Unset, `GOOGLE_CLIENT_ID` is used if it is set. |
| `GOOGLE_DIRECTORY_CLIENT_SECRET` | (unset) | Required when `GOOGLE_DIRECTORY_CLIENT_ID` is set. |
| `IMMISCIBLE_DIRECTORY_SYNC_MINUTES` | `60` | How often a connected directory is read (at least 5). |

### Crypto payments

Crypto payments are priced at the moment they are decided from two public, keyless price APIs: Coinbase first, Kraken if Coinbase does not answer. The server needs outbound HTTPS to `api.coinbase.com` and `api.kraken.com`; nothing else is configured. With no answer in time, the payment waits for a person ("Couldn't price this payment right now"); a rate is never guessed. See [docs/site/guides/crypto-rates.md](https://immiscible.fly.dev/docs/guides/crypto-rates.md).

| Variable | Default | Meaning |
|---|---|---|
| `IMMISCIBLE_CRYPTO_RATE_TIMEOUT_MS` | `1500` | How long each price source may take before the next is tried (200 to 10000). |
| `IMMISCIBLE_CRYPTO_RATE_CACHE_SECONDS` | `45` | How long a rate is reused (0 to 60). Its age is recorded with every decision. |

### Self-hosted bootstrap

| Variable | Default | Meaning |
| --- | --- | --- |
| `IMMISCIBLE_ADMIN_EMAIL` | (unset) | **required** for self-hosted production: the first owner. |
| `IMMISCIBLE_ADMIN_PASSWORD` | (unset) | The first owner's password. |
| `IMMISCIBLE_ADMIN_TOKEN` | (unset) | Admin token for the self-hosted admin API. |
| `IMMISCIBLE_ORG_NAME` | `Default` | Name of the single self-hosted workspace. |

### Email

| Variable | Default | Meaning |
| --- | --- | --- |
| `RESEND_API_KEY` | (unset) | Send with Resend. One email provider is **required** in production. |
| `POSTMARK_TOKEN` | (unset) | Send with Postmark instead. |
| `MAIL_FROM` | (unset) | From address; required with a provider. |
| `FOUNDER_EMAIL` | (unset) | Where sign-up and assessment notices go. |
| `MAIL_REPLY_TO` | `FOUNDER_EMAIL` | Reply-To on the welcome email, which invites a reply. With neither set, the welcome does not promise one. |

Without a provider every message is kept in the `outbox` table (and the log),
which is how tests and the smoke read verification links.

### Billing

To set these, run `node scripts/stripe-setup.mjs`. It creates the Stripe
products, prices and webhook from `config/plans.json` (or reuses them on a
rerun) and prints the `fly secrets set` line with every value below except
your secret key, which you paste in yourself. See
[setup-stripe.md](https://github.com/efr7-7/immiscible/blob/main/docs/setup-stripe.md).

| Variable | Default | Meaning |
| --- | --- | --- |
| `STRIPE_SECRET_KEY` | (unset) | Turns on checkout. Must start with `sk_test_` or `sk_live_` (or `rk_` for a restricted key); anything else is a configuration error. |
| `STRIPE_WEBHOOK_SECRET` | (unset) | Required with `STRIPE_SECRET_KEY`. |
| `STRIPE_AUTOMATIC_TAX` | `false` | Stripe Tax on checkout. |
| `STRIPE_PRICE_PLATFORM` | (unset) | Price id for the platform plan. |
| `STRIPE_PRICE_TEAM_AGENTS_YEARLY` | (unset) | Price id for Team billed yearly (£468 a year). Team checkout stays hidden until it is set. |
| `STRIPE_PRICE_TEAM_AGENTS_MONTHLY` | (unset) | Price id for Team paid monthly (£49 a month). |
| `STRIPE_PRICE_TEAM_EXTRA_AGENT` | (unset) | Price id for an extra Team agent (£5 a month). Not charged until the governed-agent meter is verified. |
| `STRIPE_PRICE_BUSINESS_AGENTS_YEARLY` | (unset) | Price id for Business billed yearly (£5,988 a year). Business checkout stays hidden until it is set. |
| `STRIPE_PRICE_BUSINESS_AGENTS_MONTHLY` | (unset) | Price id for Business paid monthly (£599 a month). |
| `STRIPE_PRICE_BUSINESS_EXTRA_AGENT` | (unset) | Price id for an extra Business agent (£4 a month). Not charged until the governed-agent meter is verified. |
| `STRIPE_PRICE_SETUP` | (unset) | Price id for the setup fee. |
| `STRIPE_PRICE_SCALE` | (unset) | Price id for a Scale contract (quoted and invoiced; maps the subscription to the plan). |
| `STRIPE_PRICE_ASSESSMENT` | (unset) | Price id for the assessment. |
| `STRIPE_PRICE_PERSONAL_PLUS` | (unset) | Price id for Personal Plus. |

### Chat and mobile

| Variable | Default | Meaning |
| --- | --- | --- |
| `SLACK_CLIENT_ID` | (unset) | The Slack app's client id (docs/setup-slack.md). |
| `SLACK_CLIENT_SECRET` | (unset) | The Slack app's client secret. |
| `SLACK_SIGNING_SECRET` | (unset) | Verifies requests from Slack. |
| `RAMP_CLIENT_ID` | (unset) | The operator's Ramp Developer app, for one-click Connect Ramp (docs/setup-connections.md). Without it, owners connect Ramp with their own client credentials. |
| `RAMP_CLIENT_SECRET` | (unset) | Required with `RAMP_CLIENT_ID`. |
| `GITHUB_APP_ID` | (unset) | The operator's GitHub App, for Install GitHub App on Integrations (docs/setup-connections.md). Without the six `GITHUB_APP_*` values, GitHub outcomes use a pasted webhook secret. |
| `GITHUB_APP_SLUG` | (unset) | The app's URL name, as in `https://github.com/apps/<slug>`. |
| `GITHUB_APP_PRIVATE_KEY` | (unset) | The app's private key (PEM; `\n` escapes are accepted). Signs the app's RS256 JWT. |
| `GITHUB_APP_WEBHOOK_SECRET` | (unset) | The app's webhook secret; verifies deliveries to `/hooks/github-app`. |
| `GITHUB_APP_CLIENT_ID` | (unset) | The app's client id: the JWT issuer, and confirms who installed the app. |
| `GITHUB_APP_CLIENT_SECRET` | (unset) | The app's client secret, for that confirmation. |
| `MICROSOFT_BOT_APP_ID` | (unset) | The Teams app's bot: its Microsoft Entra application (client) id (docs/setup-teams.md). Without it, Teams uses a Workflows webhook. |
| `MICROSOFT_BOT_TENANT_ID` | (unset) | The operator's Entra tenant id; the bot is single tenant. |
| `MICROSOFT_BOT_APP_PASSWORD` | (unset) | The bot's client secret. Or use a certificate (the next two). |
| `MICROSOFT_BOT_CERTIFICATE_KEY` | (unset) | The certificate's private key (PEM), instead of a password. |
| `MICROSOFT_BOT_CERTIFICATE_THUMBPRINT` | (unset) | The certificate's SHA-1 thumbprint, as Entra shows it. Required with the key. |
| `EXPO_ACCESS_TOKEN` | (unset) | Expo push service token for the mobile app. |
| `IMMISCIBLE_EXPO_PUSH_URL` | Expo's push URL | Override the Expo push endpoint. |
| `IMMISCIBLE_PUSH_HOSTS` | (unset) | Extra allowed Web Push hosts, comma separated. |
| `IMMISCIBLE_MOBILE_REDIRECTS` | (unset) | Extra allowed mobile sign-in redirect URIs, comma separated. |

### Finance and operations integrations

Each one-click card appears only when its app is configured. Register each app once; the steps and redirect URIs are in the guides under `docs/site/guides` (xero, quickbooks, pagerduty).

| Variable | Default | Meaning |
| --- | --- | --- |
| `XERO_CLIENT_ID` | (unset) | The operator's Xero app (Auth Code grant), for Connect Xero. Redirect URI `PUBLIC_URL/connect/xero/callback`. |
| `XERO_CLIENT_SECRET` | (unset) | Required with `XERO_CLIENT_ID`. |
| `INTUIT_CLIENT_ID` | (unset) | The operator's Intuit app (QuickBooks Online Accounting scope), for Connect QuickBooks. Redirect URI `PUBLIC_URL/connect/quickbooks/callback`. |
| `INTUIT_CLIENT_SECRET` | (unset) | Required with `INTUIT_CLIENT_ID`. |
| `INTUIT_ENVIRONMENT` | `production` | `sandbox` sends QuickBooks calls to Intuit's sandbox companies, for testing with development keys. |
| `PAGERDUTY_CLIENT_ID` | (unset) | The operator's PagerDuty app (Classic User OAuth, `write`), for Connect PagerDuty. Redirect URI `PUBLIC_URL/connect/pagerduty/callback`. Without it, a routing key is pasted instead. |
| `PAGERDUTY_CLIENT_SECRET` | (unset) | Required with `PAGERDUTY_CLIENT_ID`. |
| `ZAPIER_CLIENT_ID` | (unset) | The agent inventory's Zapier source (docs/site/guides/agent-inventory.md). Zapier issues these only to an integration published in its App Directory. Redirect URI `PUBLIC_URL/connect/inventory/zapier/callback`. |
| `ZAPIER_CLIENT_SECRET` | (unset) | Required with `ZAPIER_CLIENT_ID`. |

The agent inventory's Microsoft source reuses `MICROSOFT_CLIENT_ID` and `MICROSOFT_CLIENT_SECRET`, and its Google Workspace source the directory or sign-in client; the permissions and redirect URIs they need are in docs/site/guides/agent-inventory.md.

Opsgenie, Datadog and Splunk need nothing from the operator: none of them offers OAuth for sending events, so each workspace pastes a key once and Immiscible checks it live.

### Reports and dashboards

| Variable | Default | Meaning |
| --- | --- | --- |
| `GOOGLE_SHEETS_CLIENT_ID` | the Google sign-in client | The Google OAuth client for the Google Sheets export (scope `drive.file` only, a non-sensitive scope). Redirect URI `PUBLIC_URL/connect/google-sheets/callback`, with the Google Sheets API enabled in its project. Unset, `GOOGLE_CLIENT_ID` is used if it is set and has that redirect URI. |
| `GOOGLE_SHEETS_CLIENT_SECRET` | (unset) | Required with `GOOGLE_SHEETS_CLIENT_ID`. |

The CSV downloads, the board links (`/board/...`), `/openapi.json`, the import files under `/downloads/` and action callbacks need nothing from the operator. Callbacks go only to https on public addresses in production.

### Model providers

In self-hosted mode a provider's key may come from the environment instead
of the console. In cloud mode each workspace stores its own keys, sealed with
`IMMISCIBLE_MASTER_KEY`.

| Variable | Provider |
| --- | --- |
| `ANTHROPIC_API_KEY` | Anthropic |
| `OPENAI_API_KEY` | OpenAI |
| `DEEPSEEK_API_KEY` | DeepSeek |
| `MOONSHOT_API_KEY` | Moonshot |
| `ZHIPU_API_KEY` | Zhipu |
| `MISTRAL_API_KEY` | Mistral |
| `FIREWORKS_API_KEY` | Fireworks |
| `NVIDIA_API_KEY` | NVIDIA |
| `GOOGLE_API_KEY` | Google |
| `COHERE_API_KEY` | Cohere |
| `OPENROUTER_API_KEY` | OpenRouter |
| `AWS_BEARER_TOKEN_BEDROCK` | AWS Bedrock |
| `AZURE_OPENAI_API_KEY` | Azure OpenAI |
| `SELFHOST_API_KEY` | your own OpenAI-compatible server |
| `IMMISCIBLE_SELFHOST_BASE` | base URL of that server (default `http://localhost:8000/v1`) |
| `IMMISCIBLE_BEDROCK_BASE` | Bedrock base URL (default eu-west-1) |
| `IMMISCIBLE_AZURE_BASE` | Azure OpenAI base URL |
| `OPENROUTER_REFERER` | the HTTP-Referer sent to OpenRouter (default `PUBLIC_URL`) |
| `OPENROUTER_TITLE` | the X-Title sent to OpenRouter (default Immiscible; a workspace may set its own) |

### Smoke test

| Variable | Meaning |
| --- | --- |
| `SMOKE_URL` | Server to test, instead of the argument. |
| `SMOKE_PROVIDER_KEY` | An OpenAI key: the smoke connects a real provider with it. |
| `SMOKE_MCP_URL` | A real MCP server for the tool-call step. |
| `SMOKE_MCP_SECRET` | Its bearer secret. |
| `SMOKE_EMAIL_DOMAIN` | Domain for the people the smoke creates (default `smoke.test`). |
| `PG_CHECK_MODULES` | `scripts/pg-check.mjs`: a `node_modules` directory holding `pg` or `@electric-sql/pglite`. |

## Older names

The product was called Assay before it was Immiscible. A deployment set up under the old name keeps working without changes; everything below is read silently, and wherever both names are present the new one wins.

| Older name | Now | What still works |
| --- | --- | --- |
| `ASSAY_*` environment variables | `IMMISCIBLE_*` | Every variable is read under either prefix (`src/platform/env.js`). |
| `x-assay-*` request headers | `x-immiscible-*` | Accepted on every route and allowed by gateway CORS. `x-assay-session`, the session a client names itself, is now `x-immiscible-client-session`; `x-immiscible-session` is the session the server issues. Responses carry only the new names. |
| the `assay` request body field | `immiscible` | Read when it is the only one. Responses carry `immiscible`. |
| `assay/approvalId`, `assay/idempotencyKey` in MCP `_meta` | `immiscible/*` | Accepted by the MCP proxy. Replies carry `immiscible/actionId`. |
| `/.well-known/assay-keys.json` | `/.well-known/immiscible-keys.json` | The same key set is served at both paths, so verifiers that pinned the old path keep working. |
| OAuth scope `assay:agent` | `immiscible:agent` | Accepted when a client asks for it; tokens are issued with the new scope. |
| SSO domain TXT record `_assay-verification` with `assay-domain-verification=` | `_immiscible-verification` with `immiscible-domain-verification=` | A record under the older host and prefix still verifies the domain. |
| `assay.db` | `immiscible.db` | When `immiscible.db` is absent and `assay.db` is present in `DATA_DIR`, the server opens `assay.db`. Litestream and backups follow `DATABASE_FILE`; set it to `/data/assay.db`, or rename the file while the server is stopped, so replication and the database agree. |
| the `assay` command | `immiscible` | Still installed; it prints a one-line notice and forwards. |
| Fly app `assay`, volume `assay_data` | `immiscible`, `immiscible_data` | Only names in `fly.toml`. An existing app keeps its own; set `app` and the mount `source` back to the names you already have. |

Some names are part of signed or hashed formats and never changed, so that evidence and receipts issued before the rename verify byte for byte: the receipt `typ` (`assay-receipt+jwt`), the record schema ids (`assay.evidence.v1`, `assay.evidencepack.v1`, `assay.mandate.v1`, `assay.snapshot.v1`), and the labels that derive keys from `IMMISCIBLE_MASTER_KEY`. They are identifiers, not branding, and stay as they are.