next-evals-oss

Summary Stable release discovery currently fails after the source branch advances because it checks which branches still end at the release SHA. Classify stable releases and backports from the release commit's parent ancestry instead, which restores discovery for releases such as . Also let the Eve intake distinguish full GitHub Actions matrices from maintainer-run cohorts. Cohort reports identify the measured agents and keep saturation claims scoped to that cohort, allowing useful initial measurements while the automatic runner's GitHub OIDC policy is awaiting approval. Verification Manually resolved as a backport at

This records a preliminary pass@1 cohort against the exact package and release commit. It measures 31 tasks with Claude Opus 5.5, GPT 6 Sol, Gemini 4 Argon, and GPT 6 Luna in Base and AGENTS.md variants. Base passed 104/124 (83.9%); AGENTS.md passed 107/124 (86.3%). Infrastructure attempts are excluded from those scores, and the non-scoring failure confirmations are included for review. Kimi K3 and Grok 4.7 were excluded after run-level provider or runner failures; records why they have no score. The full sanitized transcripts, evaluator output, aggregate scores, and confirmation traces are stored under . This tree is separate from the publishable leaderboard results and does not change . Validation: deterministic export check and credential-value scan passed.

Why Muse Spark Open is available through AI Gateway and is useful to measure as an open model, but it should not become part of the expensive recurring Tier 1 release matrix. This is Muse Spark Open, not Muse Wonder. Wonder is a separate model and remains tracked in #135 until its Gateway provider is available to the team. Approach Add paired Muse Spark Open experiments through AI Gateway and OpenCode. Run the complete pass@4 matrix with and without the repository . Add current AI Gateway list pricing. Export the committed results into the data-driven nextjs.org/evals leaderboard. Publish the model as Tier 2 so this remains a one-off measurement rather than an every-release expense. Results Base: 26/31 (84%) With AGENTS.md: 28/31 (90%) AGENTS.md delta: +2 evals / +6 percentage points Average duration: 325.90s Average list cost: $0.235 per eval Retained-result total at public list pricing: $21.67 AGENTS.md newly passes and , with no regressions. Both variants continue to miss , , and . Scores below are pass@4 across all 31 published evals; unavailable results count as not passed. Tiering Muse Spark Open is Tier 2: its dated result remains visible on the public leaderboard, but it is not a current model that this project intends to rerun for every Next.js release. It is therefore absent from both and the recurring release matrix. Validation (9/9) (36/36) 31 summaries for each experiment with no outstanding status work Retained-result integrity, infrastructure-error, and active-credential scans Relevant context Merging this PR publishes the row directly from this repository's ; no change is needed. Muse Wonder is intentionally tracked separately in #135 because its provider is not currently enabled for the team.