End-to-end checks

These checks use the public API, built commands, real local servers, and short-lived PostgreSQL databases. An end-to-end check follows a task through those parts and checks its result. Some outside services use fixed test replies. The Cloudflare check uses a real model service.

Run the project checks

Run commands from the project root. The build must finish before tests start. Do not build in another shell while tests run: a build replaces dist files that tests and child processes load.

devenv shell -- pnpm build
devenv shell -- pnpm typecheck
devenv shell -- pnpm test:e2e
devenv shell -- pnpm -r test

By default, files run in at most four separate test processes. The count does not exceed the available CPUs. Services and databases stay separate. Tests within one file run in order. Add --maxWorkers=1 to use one process when you debug.

All end-to-end checks live in packages/e2e, under the package name @openbot-build/e2e. The root command pnpm test:e2e runs that package’s type check and tests. To run one task group after a build, use:

devenv shell -- pnpm --filter @openbot-build/e2e exec vitest run test/read-docs

For a small change, run one file or watch it for changes after a build:

devenv shell -- pnpm --filter @openbot-build/e2e exec vitest run test/chat/ChatQueries.test.ts
devenv shell -- pnpm --filter @openbot-build/e2e exec vitest --watch test/chat/ChatQueries.test.ts

exec vitest skips a full package type check on each run. Finish the build first. Stop watch mode before you build again. When the change is done, run pnpm typecheck and pnpm test:e2e.

Folders in packages/e2e/test name user tasks. packages/e2e/scenarios.json records the checks and gaps for each task. It separates normal, edge, and extreme cases. A normal case checks the usual result. An edge case checks an input or state near a rule’s limit. An extreme case checks a large bounded set, repeated work, loss, or recovery. These names do not mean that all possible inputs or failures have been tested.

Package test commands also check test types. pnpm test runs all test files in one Vitest run, but does not build packages or check test types. Package tests outside e2e check smaller units and parts of the product. Tests make and remove their own local databases. They do not use a production database.

API scope

packages/e2e/test/api-access/ApiE2E.test.ts reads the HTTP contract and sends real requests to its protected routes. It checks each protected route without sign-in. For each protected write, it checks a session cookie with both a missing Origin and a wrong Origin. This is an access check. It does not prove that every route completes every valid task. Route and request counts come from the current contract. The JSON report records the exact counts for each run. Public and hook routes are named in a fixed list, so an accidental removal of an authentication rule does not silently reduce coverage.

The end-to-end package groups checks by user task:

Task group	Folder in packages/e2e/test
Sign in and start the server	sign-in, start-server
Read and change organization settings	organizations, admin-settings
Chat, approve tool calls, and recover work	chat, approve-tool-calls, chat-recovery
Use and share bots	bot-work
Connect apps and use a protocol service	connect-apps, protocol-service
Read and write files	files
Connect channels and set triggers	connect-channels, triggers
Run code and connect computers	run-code, connect-computers
Use live sessions and model services	live-chat, use-models
Run commands and read docs	cli, read-docs

The docs checks serve the built site on a local HTTP port. They check three manual languages, the API reference, search files, the icon, every built file, local links, and decoded anchors. API pages have only English language links. External links are not fetched. These are static site checks; they do not run a browser or test interactive search and menus.

The CLI checks start built commands. The computer checks include a real terminal. Protocol checks use independent clients and local servers. Unit and component checks stay in the core, domain, api, client, openbot, cli, computer, protocol, and protocol-test packages.

Real model check

The real model check runs only when OPENBOT_E2E_LIVE=true is set. The default run skips it. It needs a Cloudflare account and API token with Workers AI access. It makes paid service calls when the account’s free allowance is used up. Keep credentials outside tracked files.

The test makes a session with Server.layerFixed, then starts the built production entry to call the real model. Session setup must leave no index work or failures. It checks cookie and API-token calls, streamed replies, saved history, repeated commands, usage, and token revocation. It makes no claim about model quality from these small tasks.

Set these variables in a local shell or secret store. Do not put the key in a command argument or a tracked file.

Variable	Value
OPENBOT_E2E_LIVE	true to run the real service check
OPENBOT_E2E_CLOUDFLARE_ENDPOINT	The account Workers AI OpenAI-compatible base URL
OPENBOT_E2E_CLOUDFLARE_KEY	The API token
OPENBOT_E2E_CLOUDFLARE_MODEL	Optional model ID; the default is @cf/meta/llama-3.3-70b-instruct-fp8-fast
OPENBOT_E2E_EVIDENCE	Optional absolute path for the result JSON

After the build, run:

devenv shell -- pnpm --filter @openbot-build/e2e exec vitest run test/use-models/Cloudflare.test.ts

Without OPENBOT_E2E_LIVE=true, this one test is skipped. A skipped real-service test is not a successful service check.

Paper experiments

devenv shell -- python3 papers/experiments/run.py --seeds 30 --output .cache/e2e/papers.json

The runner uses only Python’s standard library. It runs each of the 15 papers’ small abstract models and worked examples. The JSON result records inputs, seeds, source hashes, checks, observations, and limits. A failed check gives a nonzero exit code. A check is an assertion made during a run, not an independent study or a separate user scenario.

The experiments compare the stated method with a baseline where the paper gives one. They include counterexamples: inputs where a baseline breaks an intended rule. Passing these small models does not prove that the product implements the method, or that a large service has the same result.

Paper	Main experiment
01: Durable execution	Unknown outside results, ownership fences, and replay
02: Authority preservation	Current rights, saved identity, and action admission
03: Layered authorization	Exact rules, model thresholds, and human review
04: Agent coordination	Messages in flight, work counts, and closing a task
05: Evidence-based memory	Cited source support, corrections, and cursor movement
06: Memory information flow	Source readers and access to combined memory
07: Context compaction	Links across a cut and stale writes
08: Tool program replay	Input frames, call order, delivery order, and return checks
09: Secret references	Secret handles, allowed uses, and output forms
10: Tool discovery	Conflicting evidence and withdrawal with failed reads
11: Protocol tests	Reachable states and faults in generated traces
12: Budget scheduling	Unknown cost, late receipts, prices, caps, and shares
13: Live history	Playback receipts, epochs, and committed prefixes
14: Compute fit	Resource needs and effect limits on every trace prefix
15: Runtime composition	Plugin order, guards, hooks, and resource release

Product and paper differences

The papers define methods and studies. Several methods go beyond current product behavior. Keep three results apart: product regression tests, abstract model experiments, and checks that compare the product with a paper’s intended rule.

Run the internal product checks for Papers 06 and 12 after the build:

devenv shell -- node --disable-warning=ExperimentalWarning packages/openbot/test/PaperProductEvidenceMain.ts --strict

These checks call internal services. They compare product rules with the papers; they are not end-to-end user tasks in packages/e2e.

The command writes its evidence as JSON to standard output. With --strict, it exits with a nonzero code when an intended paper rule fails. The checks first confirm that the test data and access rules are in place. Missing data cannot make a check pass.

The Paper 06 check combines sources from Ana and Bob into one memory. An early run showed that each could read the result without access to the other person’s source. The check now requires access to every source. Neither person may read the result alone. A group that can read both sources may read it. Other users and organizations stay blocked. This is the intersection of source readers: only readers shared by all sources may read the result. Old memory with no proof of its sources’ access stays hidden until it is rebuilt.

The Paper 12 check sets a cap of 1,000 microUSD. A microUSD is one millionth of a US dollar. An early run released the first call’s held estimate when its cost became unknown. A second call spent the freed amount. A late receipt for the first call then took the final cost to 2,000 microUSD. The check now requires OpenBot to keep that hold and block the second call. A confirmed receipt replaces the hold with the actual cost. Sending the same receipt again must not add a charge. An old unknown call whose hold was lost must block new calls under the hard budgets it affects until its cost is confirmed. Held unknown costs must also survive a new budget period.

These checks must pass before the repairs are treated as complete. They do not prove a hard bound on the provider’s final charge. Paper 12’s bound assumes that every held amount is a verified upper bound on its call’s cost. OpenBot uses an estimate based on input size and a fixed output allowance. The actual cost can exceed that estimate. See Budgets.

Paper 13’s device playback receipts, Paper 14’s trace-based placement rules, and Paper 15’s stronger registration checks are not established by the existing product tests. Do not describe passing adapter or plugin tests as proof of those methods.

Limits

A full test run covers the named inputs and paths. It cannot cover all possible inputs, network faults, or outside service states. A generated access check is not full task coverage of its endpoint. A small model run is not a production load test.

Real third-party channel delivery, SSO providers, cloud computer providers, object stores, and voice playback need their own service accounts and test resources. Fixed test servers check the local contract for these paths. They do not establish that a live third-party deployment is working.

See Contributing for the development tools and the API reference for the HTTP contract.