Skip to content
Suite

Deploy a server

One small instance, one Docker host, one app container, six SQLite stores on one volume, Caddy in front, Cloudflare in front of Caddy.

visitor ──▶ Cloudflare (proxy, WAF, cache bypass) ──▶ :443 Caddy ──▶ app:1235 /api, /health, /metrics
                                                          └──────▶ app:1234 /ws
                                                                     │
                                                          /data volume (six stores)

There is no second replica, no shared database and no zero-downtime deploy: rooms, presence, projection state and rate-limit counters live in the app process, and SQLite is a local file. Scale comes from another instance with its own workspaces. Day-two tasks are in operate.md; the quotas are generated into reference/limits.md. Paths below assume the repo at /srv/suite, compose project suite, hostname canvas.example.com.

The AWS path, in order

Section titled “The AWS path, in order”

The sections below are the shape of a deployment in general. This is the same thing as a checklist for one AWS account in us-east-2, with the console page each step lives on, because the order matters: the IAM role has to exist before the instance can assume it, and the bucket has to exist before a backup can land in it.

# What Where Section
1 Bucket for dumps and snapshots, private, versioned, 30-day expiry S3, suite-backups-<account>-use2 §7
2 Instance role from deploy/iam/instance-role-policy.json, named suite-host IAM, Policies then Roles, EC2 use case §2
3 Security group: 80 and 443 from the world, 22 from your address only EC2, Security Groups §4
4 t4g.medium Ubuntu 24.04 arm64, 30 GB gp3 root, user-data from deploy/user-data.sh EC2, Launch instance, Advanced details §1, §2
5 Elastic IP, associated once the instance is running EC2, Elastic IPs §5
6 Data volume, gp3 20 GB, same availability zone, attached and mounted EC2, Volumes §3
7 RDS Postgres db.t4g.micro, private, same VPC, TLS required RDS, Create database below
8 One A record, DNS only, at the Elastic IP Cloudflare DNS §5
9 tailscale up --ssh on the box, then the panel over the tailnet Tailscale admin §5
10 Five alarms and one topic deploy/cloudwatch/setup.sh operate.md
11 Stripe products and prices in test, then live; one 100%-off promotion Stripe dashboard, then /admin/settings below
12 Resend domain and API key, Turnstile site and secret keys Resend, Cloudflare Turnstile, then /admin/settings below
13 docs.suite.run and the marketing page as static builds Cloudflare Pages Docs and marketing

Two of those steps are the ones people get wrong. The first is the key pair: a Tailscale-only box needs none for daily use, but a box with neither a key pair nor a working tailnet has no way in at all, so create one anyway and keep it as the break-glass path. The second is the database URL: it has to name the RDS endpoint, and the compose default names a postgres service that this host does not run. deploy/preflight.sh asks a finished .env for both, and for the rest of the failures that a boot does not name itself (§6).

Every secret in the list lands in one of two places, and neither is a file in the repository. The control plane reads the values the panel owns from settings rows encrypted with SUITE_SETTINGS_KEY, so Stripe, Resend and Turnstile keys go in at /admin/settings and are invisible to docker exec. The rest (the auth secret, the two relay tokens, the database URL) go in /srv/suite/.env with mode 600, written once by hand. deploy/backup.sh reads the same file.

1. Pick a size

Section titled “1. Pick a size”

The process holds every open room in memory. Loading a room and rebuilding a share projection are the expensive operations; a 5 000-card board costs seconds per rebuild, per open link.

Instance Comfortable load Why
1 vCPU / 2 GB (Lightsail, t4g.small) A handful of workspaces, ~10 concurrent editors ~1 GB left for rooms, resident only while someone is connected.
2 vCPU / 4 GB (t4g.medium) Dozens of editors, thousands of workspaces, board shares Rebuilds are CPU-bound and hold the event loop; a second core absorbs them.
4 vCPU / 8 GB (t4g.large) Hundreds of workspaces, heavy share use Still one process. Past this, split workspaces across instances.

Measure instead of guessing: /api/admin/status has store sizes, rooms and clients, and /metrics has suite_rss_bytes, suite_open_rooms and suite_projection_rebuilds_total. mem_limit in compose.yaml turns a leak into a container restart instead of an OOM kill of the box. The volume is gp3, 20 GB minimum and encrypted at creation, because the application has no encryption at rest.

That table is about serving. The same host also builds each release (§9), and the build needs more than the process does: 4 GB of memory and about 30 GB of root disk, on top of the data volume. A 2 GB instance serves a small deployment and then fails pnpm build, so either add swap or resize for the upgrade. §9 has both remedies and the numbers.

2. Create the instance and install Docker

Section titled “2. Create the instance and install Docker”

Lightsail: Ubuntu 24.04, 2 GB, static IP attached. EC2: Ubuntu 24.04 LTS, t4g.small, Elastic IP, security group from step 4. Instance store is not durable; root EBS is. Give the root volume 30 GB, because the build in §9 writes node_modules, the pnpm store and image layers there; the 2 GB Lightsail plan already includes 60 GB.

The instance’s user-data does the installing: deploy/user-data.sh brings up Docker from Docker’s repository, sqlite3, the AWS CLI, Tailscale and a 2 GB swap file, creates the suite user in the docker group, and stops there.

Terminal window
aws ec2 run-instances --region us-east-2 \
  --image-id "$(aws ssm get-parameters --region us-east-2 \
    --names /aws/service/canonical/ubuntu/server/24.04/stable/current/arm64/hvm/ebs-gp3/ami-id \
    --query 'Parameters[0].Value' --output text)" \
  --instance-type t4g.medium --key-name suite-host \
  --iam-instance-profile "Name=suite-host" \
  --security-group-ids sg-… --subnet-id subnet-… --user-data file://deploy/user-data.sh \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=30,VolumeType=gp3,Encrypted=true}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=suite-app}]'

--key-name is the break-glass path and it is not optional by accident: Tailscale is what you use every day, but a tailnet that failed to come up on a fresh box would otherwise leave no way in at all.

Then, once, by hand (both are decisions rather than setup):

Terminal window
sudo -iu suite
git clone <your remote> /srv/suite
sudo tailscale up --ssh        # optional: prints a URL; click it and SSH and /admin arrive over the tailnet

A private repository needs credentials on the host, and the smallest credential that works is a read-only deploy key per repository rather than a personal token that also reads every other one:

Terminal window
sudo -iu suite
ssh-keygen -t ed25519 -N '' -C 'suite host' -f ~/.ssh/suite_deploy
gh repo deploy-key add ~/.ssh/suite_deploy.pub --title 'suite host' --allow-write=false   # from a workstation
git clone git@github.com:<owner>/<repo>.git /srv/suite                                    # on the host

Reaching the operator panel needs no tailnet: SUITE_OPERATOR_ALLOW already admits loopback, so an SSH tunnel terminates inside it.

Terminal window
ssh -N -L 8443:127.0.0.1:443 suite@<host>      # then, locally:
curl -fsS --resolve suite.example.com:8443:127.0.0.1 https://suite.example.com:8443/admin.html

The same request from the public internet answers 403 Not found, because the door only admits the networks named there. That is the cheapest way to prove the panel is not exposed: one probe from each side.

deploy/operator-tunnel.sh opens that tunnel and prints the two URLs, so the command above does not have to be remembered:

Terminal window
deploy/operator-tunnel.sh --host <elastic-ip>          # or SUITE_TUNNEL_HOST
deploy/operator-tunnel.sh --host <elastic-ip> --print  # just show the commands

A browser cannot pass --resolve. Visiting https://127.0.0.1:8443 sends 127.0.0.1 as the hostname, Caddy finds no site block for it, and the panel does not appear. Either add 127.0.0.1 app.suite.run to /etc/hosts while the tunnel is up, or take the simpler route for a person at a known address: put that address in SUITE_OPERATOR_ALLOW and use https://app.suite.run/admin.html directly.

Terminal window
# your own address, from the machine you will browse from
curl -s https://api.ipify.org; echo
# then: SUITE_OPERATOR_ALLOW=127.0.0.1/32 ::1/128 172.16.0.0/12 <that-address>/32
#       docker compose up -d --no-build --force-recreate caddy

That is a list of networks, so it excludes everyone else; drop the entry when the address changes. A home connection’s address moves, which is the argument for the tunnel as the durable path and the allowlist as the convenient one.

Who is an operator is decided by address, not by a role. SUITE_ADMIN_EMAILS names the accounts that may use the panel, and the check compares the signed-in address against that list. There is no grant step and nothing to click: create the account, sign in, and the panel answers. An address that is not on the list gets 403 forbidden and a sentence saying so, which is the honest answer rather than a lockout.

The host needs the checkout because it builds every image it runs: caddy bakes deploy/Caddyfile and the canvas bundle, and app is compiled here from the same tree (§9). Nothing is pulled from a registry, so the clone is part of the deployment.

3. Attach the data volume

Section titled “3. Attach the data volume”

Create the gp3 volume in the same availability zone and attach it. On Nitro it appears as /dev/nvme1n1; confirm with lsblk.

Terminal window
sudo file -s /dev/nvme1n1          # "data" = unformatted, safe to mkfs
sudo mkfs.ext4 /dev/nvme1n1 && sudo mkdir -p /mnt/suite-data
sudo blkid /dev/nvme1n1            # copy the UUID into this fstab line:
# UUID=0f0e… /mnt/suite-data ext4 defaults,nofail,x-systemd.device-timeout=10s 0 2
sudo mount -a && mountpoint /mnt/suite-data
sudo docker volume create --driver local \
  --opt type=none --opt device=/mnt/suite-data --opt o=bind suite_suite-data
docker volume inspect suite_suite-data      # confirms the bind

nofail keeps a slow attach from making the instance unreachable over SSH, and it creates a trap: the box then boots with /mnt/suite-data as an empty directory on the root disk, and a careless start would create six fresh databases there and look like data loss. The mount guard in step 7 catches that. Create the volume before the first docker compose up, because compose names volumes <project>_<volume>.

4. Security group

Section titled “4. Security group”
Port Source Why
80/tcp 0.0.0.0/0 or Cloudflare ranges HTTP to HTTPS redirect and ACME HTTP-01.
443/tcp 0.0.0.0/0 or Cloudflare ranges The only public application port.
22/tcp your IP /32 only SSH.

1234 and 1235 are exposed inside the compose network and never published, so the admin API and /metrics go through Caddy, or from a container in the app network namespace:

Terminal window
docker run --rm --network "container:$(docker compose ps -q app)" curlimages/curl:8 \
  -fsS http://127.0.0.1:1235/metrics | head

5. DNS and Cloudflare

Section titled “5. DNS and Cloudflare”

One proxied A record: canvas pointing at the Elastic or static IP. Before the first certificate exists, leave it DNS-only, or keep it proxied with Always Use HTTPS off. The first Let’s Encrypt validation needs plain HTTP on port 80, and the edge redirect refuses it before Caddy can answer.

  • SSL/TLS: Full (strict), minimum TLS 1.2, then Always Use HTTPS. HSTS comes from Caddy, so leave Cloudflare’s off.
  • WAF rate limit: one rule, since Free allows one, on starts_with(http.request.uri.path, "/api/shares") or starts_with(http.request.uri.path, "/api/admin") at 120 requests per minute per IP, action Block. A challenge page in front of a JSON API is useless to a client.
  • Cache: a Bypass rule for /api/ and /ws, and Browser Cache TTL set to Respect Existing Headers so Caddy’s one-year immutable on /assets/* survives. Never Cache Everything: index.html and admin.html are no-store on purpose.
  • Network: WebSockets and Brotli on; Rocket Loader, Email Obfuscation and Under Attack Mode off. Rocket Loader breaks ES modules and the WASM bundle.
  • Client addresses: with the proxy on, Caddy sees the edge. Leave SUITE_TRUST_PROXY=0 unless you also set Caddy’s trusted_proxies for Cloudflare’s ranges, or every visitor of one edge shares a rate-limit bucket. The operator door below depends on seeing the real address, which is why the app host is DNS-only.

The operator door

Section titled “The operator door”

/admin.html, /admin/* and /api/admin/* are gated by the account, not the network: the panel asks the control plane who is signed in and whether that address is in SUITE_ADMIN_EMAILS, answers 401 without a session and 403 for a signed-in account that is not an operator. That is what lets the panel work from a phone or a hotel, where an address-based rule cannot.

/metrics is the exception and stays address-gated by SUITE_OPERATOR_ALLOW, because it carries room and memory counts and a scraper has no business reaching it from the internet.

The panel used to sit behind the same address rule, and the cost of that decision was an owner who could not reach their own deployment from anywhere but one connection. The trade is discoverability: /admin.html is reachable by anyone, and the lock is the session plus the operator list.

What SUITE_OPERATOR_ALLOW must contain on a compose host is not obvious, and getting it wrong breaks the metrics scrape. Measured on a running stack, twice:

Client Address the door sees
a browser on the internet its own public address, preserved through the port mapping
a request from the host itself, including an SSH tunnel to 127.0.0.1:443 the Docker bridge gateway (172.17.0.1, 172.18.0.1, …), not loopback

So a list of loopback alone refuses the tunnel, and the bridge subnet is both safe and necessary: external addresses keep their identity, which is why 172.16.0.0/12 cannot be spoofed from outside. The compose default already carries it, so leave it there, and delete the tailnet entry only if you are not using one. The probe in deploy/cloudwatch/probe.sh scrapes inside the relay’s namespace, which is what makes the loopback entry sufficient for it.

Tailscale’s personal plan is free and covers this. Install it on the box (the instance user-data does), run tailscale up --ssh, and connect your laptop: ssh and the panel both arrive over the tailnet, so port 22 never has to be open. A local docker compose stack sets SUITE_OPERATOR_ALLOW to the Docker bridge instead, because a browser on that machine arrives through it and is in no tailnet.

An external monitor on https://canvas.example.com/health, expecting 200 {"ok":true}, catches the instance being down; deploy/alert.sh from cron catches it being sick. The control plane’s own /health on :1236 is the money path’s: it runs SELECT 1 on every request and answers 503 with checks: [{name: "database", state: "failed"}] while Postgres is unreachable, so an RDS failover is visible rather than silent. Nothing has to be restarted afterwards, because the next request opens a fresh connection. That retry is why every pool here takes an error listener: an idle client dying is a warning in the log instead of a crash.

6. First boot

Section titled “6. First boot”
Terminal window
cd /srv/suite && cp .env.example .env && chmod 600 .env
openssl rand -base64 32     # SUITE_PROJECTION_TOKEN
openssl rand -base64 32     # SUITE_SOURCE_TOKEN

Set SUITE_DOMAIN and SUITE_ACME_EMAIL. Leave SUITE_IMAGE commented: the build below writes the tag it produced into that line, and a tag pinned by hand before an image exists only makes compose refuse to start. For a hosted relay also set SUITE_HOSTED=1 and SUITE_LICENCE_PUBLIC_KEY, generating and signing on a workstation so the private key never reaches the relay:

Terminal window
pnpm --filter @suite/collab-server licence generate-keypair --out ./licence-private.pem
pnpm --filter @suite/collab-server licence sign \
  --key ./licence-private.pem --tier hosted --workspaces 25 \
  --document-bytes 5242880 --attachment-bytes 26214400 --days 365 --email customer@example.com

SUITE_HOSTED=1 without a key refuses to boot, since a relay that cannot verify a licence is an unlimited one. SUITE_HOSTED=0 is a self-host: no licences, and no quotas unless you set them.

compose.yaml puts all six stores on the one volume (documents, shares, workspaces, settings, licences, attachments) and sets SUITE_REQUIRE_DURABLE_STORES=1, so boot refuses any store that is unset, :memory:, relative, or outside the volume, naming each offender. Leave the flag unset for a local run. Every variable is in reference/environment.md.

Run the preflight first. It reads the .env and names the failures that do not stop a boot: a settings key that makes every panel save answer 503, a Turnstile secret with no site key, a mail transport that prints verification links to a log nobody reads, a licence key pasted as a PEM file.

Terminal window
deploy/preflight.sh                       # exit 1 when a blocker is found

Turnstile is two values and one policy

Section titled “Turnstile is two values and one policy”

The pair is split on purpose. SUITE_TURNSTILE_SITE_KEY is public and inlined into the canvas at build time, so it belongs on the build (deploy/compose.aws.yaml passes it, Dockerfile inlines it); changing it means rebuilding the caddy image. SUITE_TURNSTILE_SECRET is read by the control plane at runtime and only ever leaves it to Cloudflare, so it belongs in .env and a restart picks it up.

Setting the secret makes the challenge mandatory: sign-up with no token answers 400 MISSING_RESPONSE. That is why the site key has to be in the same deployment, and why the policy has to allow the widget. deploy/Caddyfile permits challenges.cloudflare.com in script-src, connect-src and frame-src on the two canvas hosts. Without those three the script is blocked, no token is produced, and every sign-up fails forever while the page itself looks perfect (ADR-0077).

Terminal window
deploy/deploy.sh --no-backup              # builds suite:v1.1.0, starts app, waits for health
docker compose build caddy control-plane  # the other two images, from the same checkout
docker compose up -d --no-build           # Caddy, the control plane and Postgres
docker compose ps && docker compose logs -f caddy     # wait for "certificate obtained successfully"
curl -fsS https://canvas.example.com/health           # {"ok":true}
curl -fsS -o /dev/null -w '%{http_code}\n' https://canvas.example.com/admin

Open the canvas, create a workspace, and sign in. There is no recovery key any more: a workspace belongs to the organization that first opened it, and access is the account’s own sign-in. To keep the panel off the public internet, uncomment the two remote_ip lines in deploy/Caddyfile, list your office or VPN addresses, and rebuild Caddy.

7. Backups and the mount guard

Section titled “7. Backups and the mount guard”

deploy/backup.sh snapshots all six stores with SQLite’s online backup, checks PRAGMA integrity_check, hashes each file, and writes a manifest with per-table row counts and an attachment-id checksum. With SUITE_BACKUP_BUCKET set it uploads one tarball to S3 or R2.

Terminal window
sudo apt-get install -y sqlite3
curl -sS "https://awscli.amazonaws.com/awscli-exe-linux-$(uname -m).zip" -o awscliv2.zip
unzip -q awscliv2.zip && sudo ./aws/install && rm -rf aws awscliv2.zip
sudo aws configure --profile suite-backup      # cron runs as root

Bucket: versioning on, Block Public Access on, SSE-S3 on, and an IAM user allowed only s3:PutObject on arn:aws:s3:::suite-backups/suite/*. The script resolves all six stores from SUITE_DATA_DIR and detects the documents.sqlite naming compose uses, so that variable plus the bucket is the whole configuration.

Terminal window
sudo install -m 600 /dev/null /etc/suite/backup.env
sudo tee /etc/suite/backup.env >/dev/null <<'EOF'
SUITE_DATA_DIR=/mnt/suite-data
SUITE_BACKUP_BUCKET=s3://suite-backups
SUITE_BACKUP_PREFIX=suite
AWS_PROFILE=suite-backup
AWS_DEFAULT_REGION=eu-west-1
EOF
20 3 * * * root set -a; . /etc/suite/backup.env; set +a; /srv/suite/deploy/backup.sh /srv/backups >> /var/log/suite-backup.log 2>&1
40 3 * * * root find /srv/backups -mindepth 1 -maxdepth 1 -name 'suite-*' -mtime +14 -exec rm -rf {} +
*/15 * * * * root mountpoint -q /mnt/suite-data || logger -t suite 'EBS data volume is NOT mounted'
*/5 * * * * root /srv/suite/deploy/alert.sh --url http://127.0.0.1:1235 >/dev/null

Retention (7 daily and 4 weekly copies) belongs to the bucket lifecycle, and the rule plus the restore caveats are in operate.md. Snapshots are unencrypted: plaintext content, uploaded images, share and owner hashes. The bucket is encrypted at rest and the directory is root-only. Keep /srv/backups off the data volume, or a volume loss takes the backups with it.

8. Restore drill

Section titled “8. Restore drill”

Monthly, read-only, because a backup you have never verified is a hope:

Terminal window
/srv/suite/deploy/restore.sh "$(ls -d /srv/backups/suite-*/ | tail -1)" --verify-only
/srv/suite/deploy/restore.sh /srv/backups/suite-20261023T001541Z --force   # quarterly, full drill

In order: verify, refuse a non-empty volume without --force, stop app, delete the six files and their stale -wal/-shm sidecars, copy the snapshot in, start app, and require /health within 90 seconds. Check the workspace you know best and one share link, then record the duration, which is your RTO. Caddy keeps proxying while app is down, so visitors see 502 for the length of the copy.

9. Upgrade by building on the host

Section titled “9. Upgrade by building on the host”

Nothing is pulled: the host builds app from /srv/suite, which is why it needs 4 GB of memory and 30 GB of root disk (§1) and why a Graviton host produces an arm64 image. Neither the steps below nor deploy/deploy.sh depends on .github/workflows/ci.yml or release.yml, which cannot run on this account.

The order is the part that matters. A host build replaces the image it runs, so the image that is running now is tagged before the build starts; that tag is the whole rollback.

  1. Snapshot the six stores. A release that changes a stored format can only be undone from a backup taken before it opened the stores.

    Terminal window
    cd /srv/suite
    sudo bash -c 'set -a; . /etc/suite/backup.env; set +a; /srv/suite/deploy/backup.sh /srv/backups'

  2. Ask the running container which version it is. The answer names the rollback tag; v1.1.0 below stands for it.

    Terminal window
    docker compose ps -q app | xargs docker inspect \
      --format '{{range .Config.Env}}{{println .}}{{end}}' | sed -n 's/^SUITE_VERSION=//p'

  3. Tag the running image before anything can replace it.

    Terminal window
    docker tag "$(docker compose images -q app)" suite:rollback-v1.1.0

    docker compose images -q app prints the image id the container runs from, so the tag names that exact image and keeps it after the build moves suite:local.

  4. Move the checkout onto the release. The build reads the working tree, not a tag.

    Terminal window
    git fetch --tags && git checkout v1.1.1 && git describe --tags --exact-match

  5. Build the image with the tag baked in. SUITE_VERSION is what /metrics and /api/admin/status report. Both -t flags matter: suite:v1.1.1 is the record you can come back to, suite:local is what compose runs when .env names no image.

    Terminal window
    docker build --target runtime --build-arg SUITE_VERSION=v1.1.1 \
      -t suite:v1.1.1 -t suite:local .

  6. Pin the tag and restart app alone.

    Terminal window
    sed -i -E 's|^#?[[:space:]]*SUITE_IMAGE=.*|SUITE_IMAGE=suite:v1.1.1|' .env
    docker compose up -d --no-build app     # Caddy keeps running; 502 for the restart only

    --no-build because the image already exists: without it compose may rebuild and retag over the tag being started.

  7. Check the version, not only health.

    Terminal window
    curl -fsS https://canvas.example.com/health
    # The operator API answers a session: sign in at /admin.html and use its token,
    # or read /api/admin/status from the box while SUITE_METRICS_LOCAL is on.
    curl -fsS http://127.0.0.1:1235/metrics | grep suite_build_info

    A container that reports the previous version is still the previous release, however healthy /health looks.

deploy/deploy.sh runs this sequence: it backs up, tags the running image, builds suite:<tag> and suite:local from the tag at HEAD, writes SUITE_IMAGE into .env, restarts app with --no-build, and waits 90 seconds for the compose healthcheck. A failure leaves the old image tagged and prints the rollback command.

Terminal window
cd /srv/suite && git fetch --tags && git checkout v1.1.1 && deploy/deploy.sh

A host that writes its logs to CloudWatch carries a second compose file, and every docker compose command on that host has to name it. Put it in .env once, and docker compose and deploy/deploy.sh both read it from there:

.env
COMPOSE_FILE=compose.yaml:deploy/compose.aws.yaml

app is the only image deploy/deploy.sh builds, because it is the only one that changes every release. When the control plane changed too, rebuild it and include it in the restart:

Terminal window
docker compose build control-plane
docker compose up -d --no-build app control-plane

Tabs that outlive a deploy

Section titled “Tabs that outlive a deploy”

A canvas tab keeps running the bundle it loaded. index.html is no-store and the hashed chunks under /assets are immutable, so nothing about the deploy reaches a tab that is already open: the old engine keeps running, and a fix that shipped reads as a bug that survived it. Ask the reader to reload before believing a report against a tab that predates the build.

apps/canvas/src/app/useStaleBuild.ts makes that visible instead of remembered. It compares the entry chunk this page loaded with the one the server would hand out now, on mount and whenever the tab comes back into view, and StaleBuildNotice.tsx shows a line with a Reload button when they differ. It never reloads on its own: a document can be mid-edit.

Rollback

Section titled “Rollback”

Rollback restarts on the tag taken in step 3. Nothing is compiled and no old checkout is involved, so it takes as long as a container recreate.

Terminal window
deploy/deploy.sh --rollback suite:rollback-v1.1.0


# the same thing by hand
sed -i -E 's|^#?[[:space:]]*SUITE_IMAGE=.*|SUITE_IMAGE=suite:rollback-v1.1.0|' .env
docker compose up -d --no-build app

deploy/deploy.sh --list prints the tags this host holds. Keep the last two suite:rollback-* and remove older ones with docker image rm: each is a full copy of the image and counts against the root volume.

The tag returns the code; the data moves with it only as far as the schema allows. Stores migrate forward when opened, so a release that bumped PRAGMA user_version or changed a snapshot tag is undone by restoring the step 1 snapshot with deploy/restore.sh --force (operate.md), then starting the older tag. release.md has the rest of that procedure.

What the build needs

Section titled “What the build needs”

The build runs on the same box that serves, so these are facts about the deployment.

  • 4 GB of memory, or swap on 2 GB. pnpm build for the canvas and the Docker build that runs the same tree inside node:22-bookworm each want roughly 1 GB, and the second one runs inside a container whose memory is charged to the same box. On 2 GB (t4g.small, the $5 Lightsail plan) one of them usually ends in JavaScript heap out of memory or a killed vite, mid-upgrade, with the rollback tag already taken and app still on the old image. For 4 GB of swap:

    Terminal window
    sudo fallocate -l 4G /swapfile && sudo chmod 600 /swapfile
    sudo mkswap /swapfile && sudo swapon /swapfile
    echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

    A swap-backed build takes several minutes. Resizing to t4g.medium for the upgrade is faster, and is what §1 recommends past a handful of workspaces.

  • About 30 GB of root volume. The checkout grows with history, node_modules and the pnpm store are several GB, and every image adds layers on top of the node:22-bookworm base. Reclaim space before the build:

    Terminal window
    df -h /                          # the number that decides whether to build now
    docker builder prune             # build cache, 5-10 GB after a few upgrades
    pnpm store prune                 # store entries no current lockfile names
    docker image prune               # dangling layers; --all also drops tags, so check --list first

    The builder cache is the one that surprises people: it keeps the intermediate layer of every stage of every build, and on a 30 GB disk two or three upgrades fill it. deploy/deploy.sh --prune runs the first two for you.

  • Architecture follows the host. The image is built for the machine doing the building: arm64 on Graviton, x86-64 on Intel. That is correct here because the host builds what it serves. An image moved from an x86 workstation with docker save and docker load, or pulled from a registry that built it elsewhere, starts and then dies with exec format error. A build that has to happen off-instance needs --platform linux/arm64 and a builder of that architecture.

A Caddyfile change

Section titled “A Caddyfile change”

A deploy/Caddyfile change needs docker compose build caddy && docker compose up -d caddy, which costs a couple of seconds of 502. Enable the commented caddy-data volume if you recreate Caddy often, or it re-issues certificates until Let’s Encrypt’s five-per-week limit stops you.

When the certificate authority says no

Section titled “When the certificate authority says no”

The limit is real and it is measured per exact hostname set: five duplicate certificates in seven days, then 429 rateLimited until the window rolls over, which is days away and not shortened by asking sooner. Caddy logged this and kept retrying while every host answered nothing, so the shape of the incident was a working origin with no TLS.

The fix is a second authority rather than a wait. SUITE_ACME_CA selects the directory:

Terminal window
# .env on the host, then `docker compose build caddy && docker compose up -d caddy`
SUITE_ACME_CA=https://acme.zerossl.com/v2/DV90
SUITE_ACME_EMAIL=you@example.com     # ZeroSSL mints account credentials from the address

ZeroSSL hands Caddy its EAB credentials when an email is set, so nothing has to be registered by hand; the certificate is a normal 90-day DV certificate and the ACME account lives in caddy-data beside the Let’s Encrypt one. deploy/Caddyfile defaults to Let’s Encrypt, so this is a line in .env, not a change to the file. Two authorities side by side also mean a future rate limit costs one variable instead of an outage.

The other half of the incident was ours: caddy had its state on tmpfs, so every recreate began with an empty /data and requested three fresh certificates. deploy/compose.aws.yaml mounts the caddy-data volume, and without it a handful of deploys in one afternoon spends the weekly allowance.

Publishing to a registry again (optional)

Section titled “Publishing to a registry again (optional)”

Not used today, and nothing above needs it. If a registry becomes available, GitHub Actions on a working account or Codeberg with a container registry, only step 5 changes: build with a registry tag, push it, and point SUITE_IMAGE at the digest it produced. compose.yaml accepts any reference in SUITE_IMAGE, so a pull-based host sets it to ghcr.io/<owner>/suite@sha256:… and then runs docker compose pull app followed by docker compose up -d --no-build app. A host that builds keeps using suite:<tag>, and caddy is never pulled because it packages this checkout’s canvas bundle.

Container hardening

Section titled “Container hardening”

compose.yaml runs every service with the smallest surface it can and still work. The two server processes are the strict case, because they are the ones exposed to a document and to the internet:

Control app control-plane caddy
User 1000:1000 (node, set by the image) same root, with one capability
Root filesystem read_only: true read_only: true read_only: true
Writable paths the suite-data volume at /data, tmpfs on /tmp tmpfs on /tmp tmpfs on /tmp, /config, /data
Capabilities cap_drop: ALL cap_drop: ALL cap_drop: ALL, cap_add: NET_BIND_SERVICE
Escalation no-new-privileges:true same same
Limits mem_limit: 1536m, cpus: "1.0", pids_limit: 256 512m, 0.5, 128 256m, 0.5, 128

The relay never needed a writable root: its six stores are on the volume, and the tmpfs on /tmp is there because Node and tsx want a scratch directory. Caddy is the exception on two counts — ports 80 and 443 need NET_BIND_SERVICE, and its certificate cache would be tmpfs unless the commented caddy-data volume is enabled, in which case a certificate survives a recreate.

Prove it rather than read it. The relay, with every store on the volume and a read-only root:

Terminal window
docker build --target runtime -t suite-relay .
docker run --rm -d --name suite-relay-probe \
  --read-only --tmpfs /tmp:size=32m,mode=1777 \
  --cap-drop ALL --security-opt no-new-privileges --user 1000:1000 \
  -v suite-probe-data:/data \
  -e PORT=1234 -e HEALTH_PORT=1235 \
  -e SUITE_DB=/data/documents.sqlite -e SUITE_SHARES_DB=/data/shares.sqlite \
  -e SUITE_WORKSPACES_DB=/data/workspaces.sqlite -e SUITE_SETTINGS_DB=/data/settings.sqlite \
  -e SUITE_LICENCE_DB=/data/licences.sqlite -e SUITE_ATTACHMENTS_DB=/data/attachments.sqlite \
  -e SUITE_REQUIRE_DURABLE_STORES=1 suite-relay
docker exec suite-relay-probe node -e "fetch('http://127.0.0.1:1235/health').then(async r=>console.log(r.status, await r.text()))"
# 200 {"ok":true}
docker exec suite-relay-probe id
# uid=1000(node) gid=1000(node) groups=1000(node)

The control plane needs Postgres, because it refuses to boot without a database URL:

Terminal window
docker build --target control-plane -t suite-control-plane .
docker run --rm -d --name suite-pg -e POSTGRES_USER=suite -e POSTGRES_PASSWORD=suite -e POSTGRES_DB=suite postgres:17-alpine
docker run --rm -d --name suite-cp-probe --read-only --tmpfs /tmp:size=32m,mode=1777 \
  --cap-drop ALL --security-opt no-new-privileges --user 1000:1000 \
  -e PORT=1236 -e SUITE_CONTROL_PLANE_DATABASE_URL=postgres://suite:suite@<postgres-host>:5432/suite \
  -e SUITE_AUTH_ENABLED=0 -e SUITE_BILLING_ENABLED=0 suite-control-plane
docker exec suite-cp-probe node -e "fetch('http://127.0.0.1:1236/health').then(async r=>console.log(r.status, await r.text()))"
# 200 {"status":"ok",…,"checks":[{"name":"database","state":"ok"}]}

And that the runtime tree carries no toolchain:

Terminal window
docker run --rm suite-relay sh -c 'ls node_modules/.pnpm | grep -E "^(typescript|vitest|@biomejs|@stryker|turbo)@" || echo clean'
# clean

The account service on a hosted launch

Section titled “The account service on a hosted launch”

The relay above is one process on one box. The account service is the part that needs a managed database, because it holds accounts, organizations, seats and subscriptions, and losing it means every sign-in stops. Put it on RDS Postgres and leave the documents on the box.

app.suite.run ──▶ Caddy ──┬──▶ control-plane:1236  /api/auth, /api/account, /admin/admissions
                          ├──▶ app:1235            /api/workspaces, /health, /metrics
                          └──▶ app:1234            /ws
                                   │
                          RDS Postgres (accounts, organizations, billing)
                          EBS volume   (six SQLite stores)

Creating that database, the fields that matter:

Field Value Why
Engine Postgres 17 the migrations are tested against it, and pnpm db:verify names the version if a host drifts
Template Free tier, db.t4g.micro, 20 GB gp3 accounts and subscriptions are small; the documents are not here
Connectivity same VPC, private subnets, no public access only the instance’s security group needs 5432, added as an inbound rule on the database’s group
Encryption on, at creation it cannot be turned on afterwards
Backup as long as the plan allows; deletion protection a lost accounts table is a lost deployment. A FREE-plan account caps automated retention at 1 day and refuses 7 with FreeTierRestrictionError, which is exactly why deploy/backup.sh writes its own pg_dump to S3
Parameters rds.force_ssl = 1 the connection string carries sslmode=require, and this makes it enforced

Nothing migrates by hand. The control plane runs its migrations on the first request that needs identity, so a fresh database fills itself in and a failed migration clears the memo and retries rather than wedging the process. pnpm db:verify (or scripts/db-verify.mjs on the host) is the check that the eleven migrations landed.

Two things RDS does not do for you, both of which fail in ways that look like something else:

  1. The database itself. RDS creates postgres and nothing else, so a connection string ending /suite answers database "suite" does not exist, and only after the certificate verifies, so the message changes as you fix each layer.

    Terminal window
    docker run --rm postgres:17-alpine \
      psql "postgres://suite:PASS@HOST:5432/postgres?sslmode=require" -c 'CREATE DATABASE suite'

  2. The certificate. sslmode=require currently means verify-full, and RDS signs its endpoint with a CA no Node image carries: the control plane answers 503 checks:[{name:"database",state:"failed",detail:"self-signed certificate in certificate chain"}]. sudo deploy/rds-ca.sh fetches the public bundle, and deploy/compose.aws.yaml mounts it and points NODE_EXTRA_CA_CERTS at it. Do not “fix” this with uselibpqcompat=true&sslmode=require, which encrypts without verifying anything.

Set app.suite.run to DNS only in Cloudflare. The proxy’s idle timeout sits on a WebSocket that is supposed to stay open, and a collaboration socket behind it is dropped on a timer.

Four variables decide whether a person who signs up can ever get in:

Variable Why it has to be right
SUITE_CONTROL_PLANE_DATABASE_URL Compose defaults it to the postgres service, which does not exist on a host that uses RDS: the name resolves to nothing and the control plane boots degraded. Point it at the server with ?sslmode=require.
SUITE_AUTH_SECRET Signs the session. Changing it signs everyone out.
SUITE_MAIL_TRANSPORT console cannot deliver a verification link. Use resend with RESEND_API_KEY and SUITE_MAIL_FROM, or smtp with SUITE_SMTP_URL.
SUITE_ADMISSION_REQUIRED 1 holds every new account. Pair it with SUITE_ADMIN_EMAILS or the boot refuses: the queue would have no door.
STRIPE_WEBHOOK_SECRET Without it every delivery fails verification, so no payment is recorded. pnpm stripe --listen prints it locally; on the host it comes from the Stripe dashboard endpoint.
SUITE_AUTH_JWKS_URL The address the relay fetches the key set from, when that is not the public origin. Compose defaults it to http://control-plane:1236/api/auth/jwks; leave it set, or the relay asks the proxy for its own public hostname and no session verifies.
SUITE_ADMIN_EMAILS Who may use the operator surface. Comma-separated, matched against the signed-in address. Unset locks the admissions queue and the starter list to nobody.

Name the first operator before the first boot. SUITE_ADMIN_EMAILS is the only door to the admissions queue and the starter list, and an account becomes an operator by address, so create the account first and add its address to the variable:

Terminal window
# 1. the address is in .env, then restart the control plane so it is read
SUITE_ADMIN_EMAILS=you@example.com
# 2. sign up once at https://app.suite.run, follow the confirmation mail
# 3. sign in at https://app.suite.run/admin.html; the panel says whether you are an operator

If step 3 answers “not an operator”, the variable did not reach the process: it is read through a whitelist (services/auth/src/http/env.ts), and a missing value there mints operator: false rather than refusing to boot.

Admitting the first users, once mail and the queue are live: sign in at https://app.suite.run/admin.html with an address from SUITE_ADMIN_EMAILS and use the Admissions tab. The same list opens /admin/templates for writing and the relay’s /api/admin/*. The account API needs the queue and starter list, which deploy/Caddyfile already routes to the control plane on the app origin, and it routes /admin/settings, /admin/accounts, /api/account* and /api/invites/* the same way: those are control-plane paths that the /admin panel rewrite and the /api/* relay default would otherwise answer with a page or a 404.

An address that is not on the list gets 403 forbidden and a sentence saying so, which is the honest answer rather than a lockout: the panel tells the reader their address needs adding.

A new account cannot sync until it subscribes. Local and offline use is free and needs no account; an account is what buys sync, so the born-with licence grants no workspace and POST /api/workspaces/claim answers 402 plan-required until a subscription replaces the row. The app shows the plan picker rather than a bare refusal. Testers skip payment with a 100%-off promotion code at checkout, which keeps them on the real paid path.

Support access and impersonation

Section titled “Support access and impersonation”

The Accounts tab opens a support session: the operator becomes the account, sees what it sees, and stops when they are done. It is gated on consent the account itself gave, not on the operator’s role.

Fact Where it lives
Who may start one SUITE_ADMIN_EMAILS, read per request
Whether it exists at all the flags.impersonation settings row, on by default
Consent the operator_support_grants table, written from the account’s own session
The second factor required: the admin plugin refuses an impersonation for an account without 2FA

An operator needs a second factor of their own before impersonating: the admin plugin refuses an account that can manage people without one, and the Accounts tab enrols it: the password, then the otpauth:// URI into an authenticator, then one code. That is a condition of the role, not of the panel, so it holds however the request arrives.

The account turns consent on from Support access… in the app’s File menu, and can turn it off there without asking anyone. The operator sees the grant in the Accounts list and may only end it; a request that tries to grant it is refused with consent-is-the-accounts, because a permission the operator can give themselves is not consent.

Turning the flag off is the deployment-wide switch: set flags.impersonation to off in Settings and every impersonation request answers 403, grant or no grant, without a restart.

Every impersonation also writes two rows and leaves one for the account: an operator_audit entry (who, when, and what came of it) and an account_notices sentence the account reads at its next sign-in. Neither names a secret, and the notice deliberately does not name the operator.

Tables the control plane creates

Section titled “Tables the control plane creates”

Four tables appear on first use rather than in a migration, so an older database boots:

Table Written by Read by
suite_settings Settings every process’s boot overlay, and the panel
operator_support_grants the account, or an operator ending one the impersonation gate
operator_role_overrides an operator wearing a role the memberships reader, so the session and the relay’s token agree
operator_audit every operator act the Activity tab
account_notices an impersonation the account’s own dialog and strip

All five live in Postgres, so deploy/backup.sh covers them with the pg_dump it already takes. None has a retention policy: audit rows are kept forever, which is stated rather than assumed.

SUITE_APP_URL is where the admission notice sends someone. Unset admits without sending one.

A held account is not broken: it can sign in, edit, import and export on its own device. What it cannot do is create an organization or open a sync connection, which is what “waiting for approval” means.

Docs and marketing

Section titled “Docs and marketing”

docs.suite.run is the static Astro tree described below. The marketing site is a second static build with no server, so both belong on Cloudflare Pages rather than on the box: a page that is a file does not need a machine that can go down.

Both projects are uploaded as a prebuilt directory, not built by Pages, so nothing about the workspace has to be reproduced in Cloudflare’s builder:

Terminal window
pnpm docs:site:build && pnpm site:build          # apps/docs/dist and apps/site/dist
npx wrangler pages deploy apps/docs/dist --project-name suite-docs
npx wrangler pages deploy apps/site/dist   --project-name suite

Then add the custom domains in the Pages project, and leave those two A or CNAME records proxied: unlike the app, a static origin has no long-lived socket for the proxy’s idle timeout to cut.

_headers in each site’s public/ directory is the Cloudflare equivalent of the Caddy policy, and it is copied into dist at build time. Keep the two in step: the same page should not have a different security posture depending on whether it was served from Pages or from the box. The docs site allows inline scripts because Starlight emits them; the marketing page does not need to.

Docs site

Section titled “Docs site”

The reader-facing docs are a static Astro site in apps/docs, built from docs/. It needs no database, no volume and no relay: the image is the built tree at /srv/docs, which is the default of SUITE_DOCS_ROOT in deploy/Caddyfile and the root the docs.suite.run site block serves.

Terminal window
docker compose --profile docs up -d --build docs
curl -fsS http://127.0.0.1:3001/ > /dev/null
curl -fsS http://127.0.0.1:3001/llms.txt > /dev/null

The image reads the Markdown at build time, so a docs edit needs a rebuild. The container serves that tree with Caddy’s file server on port 3001. The same build runs locally with pnpm docs:site:build and pnpm docs:site:start.

The marketing page is the same shape in apps/site: /srv/marketing, SUITE_MARKETING_ROOT, and pnpm site:build.

Starlight ships inline scripts for the theme, the search dialog and the sidebar, and the dashboard adds one for its tabs and filter. The docs host therefore carries script-src 'self' 'unsafe-inline', and it is the only host that does: there is no user content on that origin, which is what makes the relaxation safe there and nowhere else. The marketing page has no scripts and keeps 'self'.

What not to do

Section titled “What not to do”
  • A second app replica, through --scale app=2, an autoscaling group or a load balancer, corrupts the locking assumptions: two processes on one volume, or two volumes behind one hostname that silently split workspaces.
  • SQLite on a network filesystem, or one EBS volume Multi-Attached to two instances, breaks locking and WAL shared memory.
  • docker compose down -v removes the volume, and instance-store or spot-only data does not survive a replacement.
  • /metrics and /admin stay off the public internet. The token is all-or-nothing, so allowlist or tunnel it.
  • One Caddy container, and nothing else serving /srv. The caching and no-store headers live in deploy/Caddyfile.

Cost

Section titled “Cost”

List prices, one small instance, excluding tax. Check them before quoting.

Item Monthly
Lightsail 2 GB (1 vCPU, 60 GB SSD, 3 TB transfer) ~$12
EC2 t4g.small with 20 GB gp3 and a public IPv4 ~$17
4 GB step (t4g.medium) ~$24–30
EBS snapshots and S3 backups (5 GB) ~$1–2
Cloudflare Free (proxy, TLS, one WAF rule) $0

The cost that bites is not on the table: a root volume filling with images, logs and local snapshots, or a box swapping because one board share rebuilds every few seconds on one core. Ten instances with ten separate workspace sets is the only scale shape, and it costs ten times one.