Test and verify a change

Suite runs three test runners: Vitest for the workspace projects, Playwright for end-to-end journeys, and node --test for @suite/billing, which sits outside the pnpm workspace. Commands run from the repository root unless stated.

The gate

Every change runs this, locally:

pnpm lint && pnpm typecheck && pnpm test && pnpm docs:check && pnpm docs:symbols && pnpm shape

None of those commands needs a runner or an account. pnpm check runs Biome, tsc, vitest run --changed, the node --test suite under services/billing, pnpm docs:symbols and pnpm shape, which is why it belongs after each edit batch; it leaves out pnpm docs:check and the full test run. pnpm verify adds the story suite, the build, and the two fast security scans (security:deps and security:secrets, both described below).

The billing suite is in the edit loop because pnpm test cannot see it: those tests are node --test and Vitest does not collect them. A rename that reached deploy/stripe/fixtures.json and not the tests naming its tiers passed every fast run and only failed at pnpm verify, which is a bad trade for half a second.

Playwright stays outside the gate. No contributor or agent starts it by default, and no change waits on a run. E2E runs when the owner asks for it, or at a commit milestone the owner names. A full pnpm e2e is a milestone action; when a run is asked for, an area script is the cheaper evidence.

ci.yml in .github/workflows has valid YAML, and every run on this account ends in startup_failure with zero jobs, so no workflow is a check on main. A self-hosted runner or an entitlement change would make the workflow usable again.

What each command runs

Command	Covers	Run it
pnpm check	Biome, tsc across the workspace, vitest run --changed, the billing suite, docs symbols, shape	After each edit batch
pnpm lint	Biome over the whole tree	The gate, and after a formatting sweep
pnpm typecheck	tsc across every workspace package	The gate, and after a type-only change
pnpm test	Every Vitest project	The gate, and before you report a result
pnpm docs:check	Relative links and anchors, code fences, banned prose, generated indexes, limits	A prose change, or a change to a limit
pnpm docs:symbols	Backticked paths, root scripts and ADR references that resolve	A page that names a command or a file
pnpm shape	File length, observe call sites, raw Loro writes	A change under apps/ or packages/
pnpm verify	The gate plus the story suite, the build, security:deps and security:secrets	Before a commit you want fully checked, and on a tag
pnpm test:stories	The Storybook suite under jsdom plus axe, in both themes	A .stories.tsx changed, or you touched a component a story renders

Security scans

Four scans, one pnpm script each. Nothing here needs a GitHub workflow, a runner or an account, and each script exits non-zero on a finding, so the same command is a check and a gate.

Command	Reads	Needs
pnpm security:deps	the registry advisories for pnpm-lock.yaml, then every package’s published signature	the network
pnpm security:secrets	every file in the checkout, then the staged diff from the pre-commit hook	Docker
pnpm security:osv	pnpm-lock.yaml, against the OSV database	Docker and the network
pnpm security:image	the package set inside a built image	Docker and a built image

security:deps runs pnpm audit --audit-level high and then pnpm audit signatures. The second half checks that each tarball came from the registry that published it, which is the part a lockfile alone cannot tell you. A high or critical advisory, or a package with no verified signature, fails it.

security:secrets runs gitleaks in a pinned image (zricethezav/gitleaks:v8.28.0) with the rules in .gitleaks.toml. That file allows the gitignored paths a working copy has (.env, dist/, storybook-static/) and names each fixture credential the tests use, one value at a time, so test files are not exempt as a class and a real key pasted into a test still fails. Findings print redacted, with the file, the line and the rule. The same script runs from .husky/pre-commit as pnpm security:secrets --staged, which scans the index: a credential stops the commit instead of becoming a commit that has to be rewritten out of history.

security:osv runs OSV-Scanner (ghcr.io/google/osv-scanner:v2.7.0) against pnpm-lock.yaml. It reads a different feed from pnpm audit, so the two report different sets of packages and different fixed versions, and running both is the point. The switches:

pnpm security:osv --download   # copy the vulnerability database into ~/.cache/suite/osv-scanner
pnpm security:osv --offline    # scan with no network at all, from that cache
pnpm security:osv --licences   # also report packages outside the licence allowlist

--offline fails when the cache is empty rather than reporting nothing, so an offline run is evidence. --licences checks the packages against a list in the script (MIT, Apache-2.0, ISC, BSD, 0BSD, CC0, BlueOak, Python-2.0, Unlicense, Zlib) and it is opt-in: what it currently reports (MPL-2.0 through lightningcss, ten LGPL-3.0-or-later packages, two UNKNOWN) is a licensing decision, and a build gate is not where that decision is made.

security:image runs Trivy (aquasec/trivy:0.76.1) against suite:local, or against the image named as its argument (pnpm security:image ghcr.io/<owner>/suite:v1.1.0). It reports HIGH and CRITICAL only, and only findings with a fixed version, because a finding with no fix cannot be closed by rebuilding the image. It is a release-time command: pnpm verify leaves it out because it needs a built image. Build one with docker build --target runtime -t suite:local ..

Docker is the only prerequisite for three of the four. A missing daemon stops the command with an instruction; no scan is skipped without saying so. git commit --no-verify is the only way past the hook.

E2E on request

An e2e run starts its own canvas dev server on :5173 and a relay on :1235, with rate limits raised and a database per run. Every run passes through e2e/scripts/preflight.mjs first: if a port is already in use, the run stops before a browser opens and names the process holding it.

Command	Covers
pnpm e2e:smoke	The specs tagged @smoke: boot, boards and sections, cards, links, selection, library, views, sharing, sign-in, collab
pnpm e2e:canvas	Boot, boards, cards, links, selection, input, library, appearance
pnpm e2e:collab	Collab, sharing, account
pnpm e2e:views	Views
pnpm e2e:changed	Playwright’s --only-changed origin/main: the specs touched since that ref
pnpm e2e	Every spec, the milestone run
pnpm e2e:doctor	Reports the processes holding the e2e ports, and stops nothing
pnpm e2e:stop	Stops only those listeners
pnpm e2e:logs	The tail of the captured Vite and relay logs under e2e/.artifacts/

Variable	Default	Meaning
CI	unset	Selects the github reporter and retries a failed test once. Servers are always fresh outside UI mode.
SUITE_WORKERS	3	Playwright browser workers. Lower it to isolate a timing failure; raise it only on a machine with headroom.
SUITE_TRACE	unset	on records a trace for every test; anything else keeps retain-on-failure.

Three workers keep the two-browser specs from starving the other browsers on one relay. Artifacts land under e2e/test-results/<test-slug>/: trace.zip on failure, test-failed-1.png at the moment of failure, and video.webm only when a retry happens. The reporter is list or github, so there is no HTML report unless you pass --reporter=html.

Projects

Project	Runner	Scope	Command
graph	Vitest, node	packages/graph/src/**/*.test.ts	pnpm exec vitest --project graph
importer	Vitest, node	packages/import-obsidian/src/**/*.test.ts: the drop walk, markdown and frontmatter parsing, canvas colour mapping, and the vault as a whole	pnpm exec vitest --project importer
licence	Vitest, node	packages/licence/src/**/*.test.ts	pnpm exec vitest --project licence
server	Vitest, node	apps/collab-server/src/**/*.test.ts (http/, store/, projection/, config/, sync/, ops/)	pnpm exec vitest --project server
control-plane	Vitest, node	apps/control-plane/src/**/*.test.ts	pnpm exec vitest --project control-plane
auth	Vitest, node	services/auth/src/**/*.test.ts (http/, identity/, invitations/, licensing/, organizations/, store/)	pnpm exec vitest --project auth
canvas	Vitest, jsdom	apps/canvas/src/**/*.test.{ts,tsx}, with apps/canvas/src/test/setup.ts	pnpm exec vitest --project canvas
e2e	Playwright, chromium	e2e/tests/**/*.spec.ts, grouped by area (boot/, boards/, cards/, links/, collab/, sharing/, …)	pnpm e2e
billing	node --test	services/billing/src/**/*.test.ts	pnpm --filter @suite/billing test

pnpm test is vitest run and covers every Vitest project in vitest.config.ts; pnpm --filter @suite/billing test runs the billing suite outside it. UI mode is the one e2e command that reuses a running server, so the window can stay open between runs.

One test at a time

# One unit file, or one test in it.
pnpm exec vitest run packages/graph/src/schema.test.ts
pnpm exec vitest run apps/collab-server/src/store/storeUpgrade.test.ts -t "refuses a store a newer build wrote"

# One spec, or one test in it. Extra arguments reach Playwright.
pnpm e2e e2e/tests/collab/collab.spec.ts
pnpm e2e -g "undoes a removal from the board"

A single spec is the right tool while iterating: it boots the same servers as the full run, so a pass in isolation is real evidence. Treat it as weaker evidence when the failure depends on order or on state an earlier spec left behind.

Watching a failing run

pnpm e2e:ui opens a persistent Playwright window on http://127.0.0.1:8181: test tree, live browser pane, timeline, and a DOM snapshot per step. One window at a time, since the port is pinned; a second call fails with EADDRINUSE. Free it with pnpm e2e:ui:stop.

pnpm e2e:headed runs the real suite in a visible browser, one worker (--workers=1), sequential and followable. Use it when the failure depends on execution order, and UI mode when you want to pause and inspect.

pnpm e2e:trace opens the newest trace.zip from e2e/test-results/. Traces are kept on failure by default; SUITE_TRACE=on pnpm e2e records one for every test.

pnpm e2e:logs prints the tail of the captured Vite and relay logs under e2e/.artifacts/. Read it when a run stops producing test results or a browser loses its server. The relay log names the request that failed; the Vite log names the file that triggered a reload.

Reading a failure

An e2e spec that passes alone and fails in the suite usually depends on order or timing, not on a server another spec started. Run pnpm e2e:headed with --workers=1 to keep order and watch the sequence.

A jsdom test that fails on a measurement is the wrong tool: layout, scrolling and getBoundingClientRect are zero there, so the assertion belongs in an e2e spec or a story.

A graph test that fails only in the full run usually shares a seed or reuses a container across cases. A 15 s timeout means real work.

A flaky test is a bug in the test. --retries exists for videos and traces when CI is set; do not use it to turn a red test green locally.

Coverage

pnpm coverage runs the whole Vitest suite with @vitest/coverage-v8 and writes coverage/ (coverage/lcov-report/index.html for the browsable view). coverage.include names four sources: graph, licence, server and canvas, so importer, control-plane and auth run without being measured. Of the measured four only @suite/graph is gated, through one glob key in coverage.thresholds at the root of vitest.config.ts. The others are printed, not gated: their uncovered lines are socket, SQLite and entry-point wiring that e2e already drives, and a threshold would invite assertion-free tests.

Mutation testing

Stryker asks whether a test would notice if an expression were wrong. It is a diagnostic for choosing the next test, never a score to raise.

pnpm mutation:quick --mutate packages/graph/src/views.ts   # the normal way to try one file
pnpm mutation:server                                       # licence, limits, config
pnpm mutation                                               # the full graph run, minutes not seconds

Read reports/mutation/<run>.html. A survivor is a missing behaviour, an equivalent mutant, or tool noise. Missing behaviour: add a test through the public API. Equivalent mutant: a // Stryker disable next-line comment with the reason. Never close one with expect(result).toBeDefined().

Property-based tests

Five fast-check suites exist, and they are where “it must hold for every input” belongs:

Suite	Property
packages/graph/src/sections.property.test.ts	derived section membership under arbitrary placement moves
packages/graph/src/merge.property.test.ts	concurrent edits converge regardless of order
packages/graph/src/search.property.test.ts	query semantics hold across generated workspaces
packages/licence/src/property.test.ts	licence parsing rejects every mutation
apps/collab-server/src/config/limits.property.test.ts	limit resolution order holds for any layer mix

They run inside pnpm test. Add a case to the matching suite rather than a table of examples when the invariant is the thing worth pinning.

Visual baselines

e2e/tests/visual/visual.spec.ts keeps a small curated set of layout snapshots as committed PNGs under e2e/tests/visual/, so a run needs no vendor service. Story surfaces are served from the built Storybook on :6006; the app states run against the normal webServer.

pnpm --filter @suite/e2e e2e:visual            # check the baselines
pnpm --filter @suite/e2e e2e:visual:update     # rewrite them

--update-snapshots is the only supported way to change a baseline, and never the first thing to try. A failure means the pixels moved: read -diff.png and -actual.png in test-results/ and decide whether the movement is the change you intended.