evalite

It feels like momentum on the project has stalled - no commits in last ~3 months, v1 still on a branch, no engagement with recently filed issues or open PRs. It would be useful if @mattpocock could provide an update of whether there is any plan to continue pushing this forward and resuming to develop it actively. I was initially excited by this as it seemed to hit a sweet spot of a lightweight, easy to use eval tests framework, and started using it in my codebase. but it's hard to invest more in building a suite atop it with such an uncertain future. This is not a complaint - I totally understand that priorities change. Matt is producing awesome content for the community and perhaps feels his time is better spent on other things. If that's the case it would just be good to make this explicit.

Summary Evalite's watch-mode UI is unreachable from any device that isn't the same host running the server — through Tailscale, cloudflared, ngrok, an SSH tunnel to a different machine, a VPS + reverse proxy, etc. Two independent bugs combine to cause it. Reproduction 1. Run on host A (e.g. a VPS, remote dev machine, cloud IDE) 2. Expose port 3006 to host B (Tailscale, , cloudflared, ngrok — any forwarding mechanism) 3. Open the UI URL from host B in a browser 4. HTML loads. The UI shows "Something went wrong" / "Load failed" Root causes Bug 1 — bundle hardcodes builds its API and WebSocket URLs from a hardcoded string, not from . Two occurrences: On any client that isn't the same machine as the server, resolves to the client itself → every fetch and WebSocket connection fails → "Load failed". Fix: Works for both local () and remote (). Bug 2 — server hardcoded to loopback bind, no config option around line 357: Fastify defaults to when is unset. No env var, no config option. Users can't bind to a specific tailnet IP, or for use behind a firewall, etc. Fix (minimal, backwards-compatible): Default stays loopback — no regression for existing users. Anyone needing to expose the UI sets (or better, add to the config schema). Workaround (what I'm running now) with a patch that applies both fixes. Details + full patch file: Saldoapp PR #315. Works fine, but requires the minified bundle diff (~200 KB) to be tracked in the consumer repo — unavoidable while the assets ship pre-built. Impact Any team using evalite from: A dev VPS with an iPad/phone iterating over Tailscale (my case) A cloud IDE (GitHub Codespaces, Gitpod) → local browser A remote pair via A shared team dashboard behind a reverse proxy …hits both bugs and has to either give up remote access or maintain a fork/patch. First bug (the bundle) is the harder one — no runtime workaround since the assets are pre-built and shipped. Environment evalite (latest) Node 20 Verified on Linux (Debian) VPS ↔ iPad Safari over Tailscale Happy to send a PR The bundle change is a 2-line source edit; the server change is 1 line + a config schema field. Can do both if it'd help. Also happy to just leave this as an issue and let you fix it however fits best.

Summary forward the parsed option to in run and serve modes add regression tests for both command paths add a patch changeset for Root cause and impact The CLI parser captured , but the run and serve command wrappers omitted it when constructing the options passed to . As a result, the reporter continued printing the detailed task-results table even when was used. This is especially noisy for large eval suites that use the UI to inspect results. Validation 102 Evalite source tests passed typecheck passed ESLint passed Prettier passed Fixes #393