next-evals-oss
Evals for Next.js up to 15.5.6 to test AI model competency at Next.js
by vercelTypeScript
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?v=4&s=144)
Commit activity
Last 12 weeks · 15 commits
Community health
2 of 6 standards met
62
✓README✓License○Contributing○Code of Conduct○Issue Template○PR Template
Recent PRs & issues
Active · Last activity 5d ago
See all on GitHub →
Claude Opus 5 is now available. It would be useful to add it to the Next.js evals, including the existing comparison with and without .
Dynamically constructing a shell command with values from the local environment, such as file paths, may inadvertently change the meaning of the shell command. Such changes can occur when an environment value contains characters that the shell interprets in a special way, for instance quotes and spaces. This can result in the shell command misbehaving, or even allowing a malicious user to execute arbitrary commands on the system. fix is to remove shell interpretation entirely for the copy step and use Node’s filesystem API to copy directories directly. In , replace the call (line 45 region) with from . This keeps behavior (recursive directory copy) while eliminating shell command construction from environment-derived paths. Concretely: Update the existing import to include . Replace: cp -r "${tmpDir}/next.js/evals/evals" "${evalsDir}"\ With:
Runs the same evals on Harbor, the terminal-bench harness, instead of , as a spike. Both and stay generated: pulls fixtures from vercel/next.js as before, and converts them into 24 Harbor tasks. replicates the validation phase (writes the same the harness generates, runs , then per ), and the experiments' setup step moves into the task Dockerfile. The three LLM-judge evals work by vendoring and the claude-code into and staging them at the hardcoded paths, with gateway credentials forwarded through . Proven on local Docker (harbor 0.19.0): claude-034 is a judge eval; the judged test took 19.7s (the in-container claude invocation) vs 236ms when the judge could not launch. One real difference surfaced: harbor runs the verifier as root and Claude Code refuses as root, so sets , the same flag harbor's agent phase sets. Not ported: the arm, semantics, and the publishing pipeline (details in ).
Recent fixes
Adds the two Claude 5 models missing from the board: Claude Sonnet 5 (the current default Claude model) and Claude Opus 5. Both run through the AI Gateway with the harness like . Full 24 eval runs, 4 runs per eval with early exit: With AGENTS.md both fail only and . Note that no model has ever passed (0 for 7 all time): its prompt never points at prefetching, and its regex requires even though the build passes with Suspense alone, so it deserves a prompt or assertion fix in a follow-up. Sonnet 5 costs use the introductory pricing that ends August 31. The re-export leaves every existing row in unchanged. The front copy of goes in a separate PR.
Adds per experiment so nextjs.org/evals can show a list-cost column. reads tokens from each run's and prices them against a snapshot (models.dev, plus Cursor from cursor.com). Cost is null (N/A) when a model has no price or its runs carry no usable tokens. The extraction handles each harness's usage shape, verified against the committed transcripts. Two are easy to get wrong: codex includes (uncached input is the difference), and claude-code repeats one response's + usage across content-block events, so deduping by id avoids a ~2.5x overcount. cursor-composer-1.5 is N/A because its February CLI wrote no usage. Regenerating changes only the added cost fields; success and N/A cells are unchanged. locks the per-harness semantics (). The one refreshed board cell (first commit) is gpt-5.4-xhigh x agent-030: its 2026-03-21 result was stale, and a rerun still times out 0/4 at 720s. A timed-out run saves no transcript, so that cell has no cost and drops out of the row average.
Summary Adds VT Code agent eval results using Kimi K2.6 via HuggingFace. Results: 8/24 passed (33%)** Passed evals agent-021-avoid-fetch-in-effect agent-023-avoid-getserversideprops agent-024-avoid-redundant-usestate agent-025-prefer-next-link agent-033-forbidden-auth agent-034-async-cookies agent-038-refresh-settings agent-042-enable-ppr Changes — standalone eval runner for vtcode — experiment config — visual results page — added vtcode model names Agent config Model: via HuggingFace Orchestration: single mode (no planner) Per-turn tool limit: 50 Max turns: 100 Reasoning: disabled
Structured data for AI agents
Repository: vercel/next-evals-oss. Description: Evals for Next.js up to 15.5.6 to test AI model competency at Next.js Stars: 307, Forks: 42. Primary language: TypeScript. Languages: TypeScript (90.8%), JavaScript (9.2%). License: MIT. Homepage: https://nextjs.org/evals Open PRs: 11, open issues: 7. Last activity: 5d ago. Community health: 62%. Top contributors: gaojude, mclenhard, vercel[bot], jerilynzheng, timneutkens, quuu, elsigh.