Observe a deployment

Where to look when something is wrong, and what each surface can and cannot tell you. Written for two people working on one box: the owner and whoever is helping them.

Everything here reads CloudWatch, so it needs AWS_PROFILE=suite and AWS_REGION=us-east-2.

The three surfaces

Surface	Answers	Does not answer
Logs (suite-app)	what happened, in order, with the request id	whether it is still happening
Alarms (Suite/*, AWS/*)	is anything red right now	why
/metrics via deploy/alert.sh	rooms, memory, connections at this instant	what led there

The panel at /admin.html is a fourth, narrow one: it answers “who is signed in, what have they claimed, and what has an operator changed”, and the activity list is the audit trail for every operator write.

Start here

AWS_PROFILE=suite deploy/cloudwatch/queries.sh            # the saved questions
AWS_PROFILE=suite deploy/cloudwatch/queries.sh errors 6   # the last six hours

Six queries are saved, because each has already been asked once under pressure: errors, auth, admin, webhooks, relay and csp. Each names the symptom it addresses in its own output. They run against the suite-app group, where every container’s JSON lines land under a stream named after the service (app, control-plane, caddy).

A line is JSON with level, msg, component and, for a request, path and status. When two lines disagree about what happened, the requestId on both is how they are tied together.

Alarms

aws cloudwatch describe-alarms --profile suite --region us-east-2 \
  --query 'MetricAlarms[].[AlarmName,StateValue]' --output text

Six alarms, and what each one means in practice:

Alarm	Fires when	Usual cause
suite-health	the probe’s /health fetch failed	the relay is down, or a container is unhealthy
suite-missing	the health metric has not reported	the probe is not running, or it cannot publish
ec2-cpu, ec2-status	the instance is saturated or failing its own checks	a build on the box, or a host fault
rds-storage	Postgres is short of free space	attachments or the update log growing
suite-certificate	CertificateDays is under 14, or was never published	the certificate was not renewed, or the origin serves none

The last one treats missing data as breaching, on purpose: “no certificate” is exactly the state that took the site down for an afternoon, and it arrives as an absence rather than as a failure.

CertificateDays in the Suite namespace is the sixth, and the one that was added after an outage taught us why: deploy/cloudwatch/probe.sh reads notAfter from the certificate the browser actually gets and publishes the days left. Without it, a spent ACME rate limit looked exactly like a healthy deployment, because the relay answered /health on loopback the whole time the public site was refusing connections.

The lesson generalises: an alarm on the service cannot see a failure in the layer in front of it. TLS, DNS and the proxy are all that layer.

What is not covered

Client-side exceptions. The app has scrubbed, opt-in crash reporting built (apps/canvas/src/crash/report.ts) and no DSN configured, so nothing leaves a reader’s browser. A JavaScript error in a card editor is therefore visible only through the diagnostics bundle the app can copy, which is a deliberate choice while the deployment has two users and no error tracker.

That is the first thing a hosted tracker buys. Until then, when a browser-only problem is reported, ask for the copied diagnostics: it carries the build, the route, the storage state and the last uncaught error, which is most of what a tracker would have shown.

The metrics probe

deploy/cloudwatch/probe.sh runs from /etc/cron.d/suite every five minutes and publishes Suite/Health and Suite/CertificateDays, exiting non-zero when a check fails. SUITE_PROBE_HOST names the certificate it reads (default app.suite.run), and no certificate at all publishes nothing, so the alarm’s INSUFFICIENT_DATA says so rather than a fake zero. It scrapes /metrics inside the relay’s namespace because the relay only answers a loopback source, and publishes from the host because the instance metadata is unreachable from inside that namespace.

If suite-missing is in ALARM, run the probe by hand first, then check that SUITE_METRICS_LOCAL=1 is in .env on the host. Without it /metrics answers 401 and the probe has nothing to publish, which is precisely how both alarms came to be red while the site was fine.