workflow

Description Adds to restore pre-#4861 scheduling for individually checked runs on the Node VM. In the repro, an old Postgres run replays a different race winner after the upgrade and passes inconsistent values to a new step without failing. Listing that run restores consistency, but listing the other capture breaks it. Operators must check a copy against its saved step inputs first. Listed runs retain the ordering bug fixed by #4861 until they finish. The affected capture used staged DB locks and one worker; recovery is demonstrated only with the default setting. Full capture details and comparisons are in the repro. How did you test your changes? Compared saved histories on the old runtime, main and this PR in memory; separately checked Postgres restart/resume with restored database copies. Four unit cases cover old/new histories and both replay settings. Core tests and the package build passed locally. Docs Preview Awaiting Vercel deployment approval for the docs preview. PR Checklist - Required to merge [x] 📦 Patch changeset included for . [x] 🔒 DCO sign-off passes (). [x] 📝 Ping once ready and the above checklist is complete.

Runs move to a single-orchestrator model (spec version 9). One orchestrator delivery per run makes every decision, and step invocations only run step bodies and report outcomes. Step and wait events become plain appends that the runtime never relies on the World to refuse. The World fences the orchestrator as the only in-band writer. The spec 8 orchestrator stays, side by side. Each delivery is routed by the run's stored spec version, so a World that declares 8 keeps creating spec 8 runs and needs no fence, and runs created before a World moves to 9 finish on spec 8. Code layout : the shared entrypoint. It parses the message, resolves the run's spec (: , , a new hint on every orchestrator message, else ), and hands the delivery to one orchestrator. : main's orchestrator (delivery body, suspension handler, step executor, QuickJS entrypoint, ownership, constants) and main's tests for it. : the single orchestrator. Shared, used by both: (delivery context and helpers), , , , , , (the end-of-run drain is a callback each orchestrator supplies). makes first-party Worlds mint spec 8 runs; world-testing runs its conformance suite over a spec 8 Local World as well. Review document. A temporary page, , covers every change in this PR. It has the decisions, tradeoffs, known gaps and remaining work, each illustrated with an example workflow. It is served at /docs/changelog/single-orchestrator-review on the docs preview, and is removed before merge. Temporary, revert before merge: points at the matching backend preview. The catalog entry points at a pre-release tarball, the client half of the queue's experimental concurrency key and lease handoff. Release that first, then pin the version. While the catalog pins a URL, workbench staging sets , because pnpm 11 refuses URL subdependencies. pnpm 11 users of this PR's pre-release tarballs need the same setting. Warn-level logs on spec 9 delivery exits in . They were added to diagnose the e2e failures below. Behavior Only the orchestrator decides. The background step handler () runs the body, writes and the outcome out-of-band, and wakes the orchestrator with an unkeyed message. It never replays or continues the run. Execution mode is fixed at creation. carries and . An enqueued step never runs inline, and only a redelivery of the creating message re-enqueues a step that hasn't started. One message per step, retried in place. After , the handler returns , and longer backoffs span several hops. A step message is acked only after a terminal event. Retention is sized to the retry span, capped at 7 days. Attempt and the redelivery check. Any delivery other than a first delivery of a non-retry payload reads the full log first: skip if terminal, wait out , fail when retries are exhausted ( is at most once), otherwise run with the log's attempt count. In-band fence. Every orchestrator write goes through one serialized writer (). The writer starts from the load snapshot's and advances by the World's reported count. On it stops all writes and inline bodies and redelivers instead of acking. Consume after commit. An inline step's outcome or a reaches the VM only after its write returns, with the skipped-slot report merged in position order. The runtime reloads when the report is incomplete. Wakes. Out-of-band writes (resume, cancel, wake-up, external attributes, step outcomes) wake the run with no idempotency key. A sleep's timer is a delayed wake sent by the invocation that created it. Replay. Any event for an entity after its terminal event is inert, in both engines, the shared fixtures and the observability UI. Turbo still fast-paths a run's first delivery: is written without waiting, and the first inline bodies start before their start commits. A redelivery or a coalesced re-run loads the log. Not used by spec 9 runs: lazy and optimistic inline start, batched claims, step ownership leases and backstops, the precondition guard, and open-wait clock skew. They remain on spec 8 runs, and their settings are listed under "Settings for spec 8 runs" in runtime tuning. Resilient step dispatch is removed. Worlds. : v5 event API, the fence fields, the list snapshot, the live feed over WS, and same-message redelivery. Spec 8 runs use per-run orchestrator topics with (on this is the opt-in; whether spec 8 returns to 's shared topic is open). Queue serialization on world-vercel (spec 9). Spec 9 messages go to a keyed flow topic, . It has its own flow trigger with no ( in ), because VQS refuses keyed sends to a topic a consumer reads. Orchestrator messages carry the run id as , so VQS keeps one delivery per run and coalesces the run's other orchestrator messages into it. Step messages are unkeyed and run in parallel. When the ACK finds coalesced messages, the handler re-runs within the same delivery (), at most 32 times. The runtime treats a re-run as a redelivery: no turbo, and it reads the log. A delivery that has to end early (deadline, stream flush, replay budget, an unread write) hands its message off (, ). Otherwise it would send itself a message that is coalesced into its own delivery. Handoffs are left out of , which the delivery cap and the creator rules count. Messages that carry state the log doesn't hold are sent , never keyed: replacement messages, divergence recovery, and 's . Coalesced re-runs share the invocation's inline deadline: world-vercel passes in the handler meta, and the spec 9 delivery measures its deadline from it. Running out of the 32 re-runs hands the message off instead of a counted backoff retry (). and : the fence and snapshot. They keep their step and wait rows. Known gaps is an open decision. Its waits stay completed in-band, behind an unkeyed wake. An out-of-band would need the backend to accept it (today it answers 409 ), and run-ahead to treat every awaited sleep like an awaited hook. Rollout coupling. world-vercel sends spec 9 messages to , which only a deployment built with the new subscribes to. Billing. Vercel Queues bills operations on keyed messages at its premium rate. Storm workflows still overlap. After the queue's backend rollout, head logged 19 supersessions on one turbopack deployment running the event-log stress workflows and none on later ones. Cause under investigation. QuickJS continuation path. Its exit requeues hand off through , which is covered by typecheck only. A consume-after-commit check on its log-view path is not explicit. world-postgres race. Two writers holding the same in-band count at the same instant can both pass the pre-check. Only one event lands, but the loser's entity-row update can stick. Coverage removed: the retained-VM 412 in-process restart (the precondition guard it covered no longer exists); concurrent-orchestrator cold-replay equivalence; force-claim wake republish in the suspension tests. Found on Vercel e2e Concurrency key dropped. The first Vercel run of the keyed topics failed on 2 of 27 lanes. Its step started two to four times (22 of 81 spec 9 runs; earlier heads had 1 of 910). The logs showed one run's orchestrator messages delivered in parallel, with fence refusals at 100 to 140 per deployment. Cause: 's dropped , fixed in the pre-release client. After the fix: 0 to 5 refusals per deployment, and all 27 Vercel Prod lanes pass. Merge from main. #4895, the concurrent hook registrations change, is a spec 8 path. It went to through a per-hook creator option on the shared hook writes. Release Custom Worlds no longer have to change: one that declares spec 8 keeps creating spec 8 runs on the spec 8 orchestrator. Only a World that declares 9 has to provide the single-writer guarantee (per-run orchestrator serialization plus the in-band fence). The Building a World docs still mark the spec 9 section as v6 beta; whether this ships as a 5.x minor is now a release decision rather than a compatibility one. Follow-ups (not in this PR) Idempotent in-band retries.** Deferred after measuring. Production event writes fail without a response about 0.003% of the time once a few always-failing deployments are excluded. On spec 9, such a write costs a stop and a redelivery (about 5s plus a replay) and never produces wrong data. Testing CI on : E2E Required Check passes, 184 of 185 checks. The one failure, No Test Overrides, is expected while the override is set. Local, after the merge from main: core 3657, world 270, world-local 685, world-vercel 1033. New tests: keyed topic routing and keys, handoff and re-run metadata, and in world-vercel; the second trigger in builders, SvelteKit and Nitro; in core, using a coalescing and handoff mode of the append-only test World: a deadline handoff with and without the capability, no turbo on a coalesced re-run, the unkeyed replacement and wake-up messages, and resumes arriving while the orchestrator runs. 🤖 Generated with Claude Code

Stacked on #4946 (which is stacked on #4945). Description The QuickJS engine's baseline heap (the workflow bundle evaluated in a fresh VM) is now prebuilt for every production build, with no opt-in env var. It ships as a file in the flow function instead of inline in the route. A cold instance used to compute it in memory before its first QuickJS run; now it loads the file. This helps even with snapshotting disabled. Measured on the workbench (14 MB heap, 3 MB file), first QuickJS run on a cold : Runtime () New options. takes: : a small inline header (engine id, code digest, heap digest, or an reason). A mismatch never costs a file read. : reads files shipped with the route. The heap is read lazily, once per process, and only by QuickJS runs. A missing or corrupt file falls back to computing the heap in memory, with the same result. Shared mechanism. is generic on purpose: the follow-up that ships the QuickJS WASM binaries as static files instead of base64 will use it too. Better build errors. A bundle whose module scope throws in QuickJS now reports the thrown error (name, message, stack) to the build. Builders One base mechanism. writes next to the route and emits the header and loader into it. Each integration only overrides how the route finds the file (): Not prebuilt. Watch mode, Nest's build at application startup, vitest and world-sim skip it, because the runtime computes the heap lazily there anyway. Build failure policy. A bundle whose module scope throws in QuickJS fails the build when is set at build time, since every QuickJS run would fail. Otherwise the build logs the error and continues, so node-engine apps that, say, call at top level aren't broken. Ineligible bundles (module scope reads the clock or draws randomness) are recorded in the header so instances skip the attempt. Bundle size. The gate is unaffected; the route only gains the header and loader, a few hundred bytes. The file shows up in hono's informational number, which walks . Fix found while testing (pushed to #4946) Rolldown inlined the core version into the engine-id template, which turned followed by a digit into an octal escape. That made the QuickJS entrypoint chunk a , so every QuickJS run on Nitro v3 failed. It is fixed in #4946 (f5d4182d6). How did you test your changes? Real production builds of , , , , , , , , , , (), and . In each, the file is in the expected output location, the route's loader resolves to it, and Next's flow-route lists it. SvelteKit and Astro emit no file and no header. End-to-end on QuickJS: with passes against (Turbopack) and (Nitro v3). The debug log shows . With the file removed, the run falls back to computing the heap in memory and still passes. now asserts the file's location per app, and for Next its presence in the flow route's trace. CI's local build step runs this. : Covers the default build, the file plus a working loader, watch mode, and an ineligible bundle. A module scope that throws fails the build with and doesn't fail it otherwise. : The asset is used and read once per process. An engine mismatch is decided without reading the file. A corrupt file, an unreadable file, and a missing loader all fall back. Ineligible bundles and errors are reported, and the baseline is not applied to other code. : tracing includes are added and merged with the user's own. Full unit suites: the full core unit suite passes except , which fails on too under local Node 26. nitro, nest, sveltekit and vitest pass. and builders' stale fail here and also fail without this change. PR Checklist - Required to merge [x] 📦 was run to create a changelog for this PR [x] 🔒 DCO sign-off passes (run on your commits) [ ] 📝 Ping in a comment once the PR is ready, and the above checklist is complete