Last 12 weeks · 17 commits
3 of 6 standards met
Fixes the twelve findings in #3. All of them reproduced; the notes below are only for the places where the fix needed a decision, or where a boundary moved. Closes #3 Changed safety boundaries Negative-test proof got stricter, and slower to satisfy. used to accept any non-zero exit from the reverted run. Reverting deletes new implementation files, so a test that only imports the new module fails to load whether or not it asserts anything — an assertion-free test was certified as proving the fix. It now classifies the failure and reports only when the test actually ran and failed. The cost is real and worth agreeing to before merging: for a new module, the information isn't there. An import error and an assertion failure are the same observation from outside. That shape now reports , the verifier records , and the PR lands at with a human read. Bug fixes to existing code still get a genuine proof. A test command that fails silently is also , which is a deliberate nudge toward real runners over shell predicates. section 8 documents this. Every factory PR is now explicitly a draft. and the implement skill disagreed about what a non-draft factory PR meant. README, ADVICE.md, and the charter already say every run ends at a draft, so that's the reading I kept; the four conditions that used to gate draftness now set instead. If conditional drafts were the intent, this is the commit to reverse. sweeps at the end. A required gate the DETECT block never reached emitted neither nor and left the verdict GREEN. and on a Python repo both hit it, and both are configs 's own comments invite. Anyone who has already tuned may find a gate they thought was green is actually MISCONFIGURED — which is the point. The push guard matches refspec destinations. and both got through before. is now protected too; the path check looks for a write verb or a redirect target rather than any command that mentions a protected path, so still runs. Not fully fixed Back-pressure is read at run start and the counted label is applied at run end. The count now includes , which closes the window two overlapping runs used to slip through, but it isn't compare-and-swap. Claims have one in the remote ref; back-pressure doesn't, and I didn't invent one. Verification passes. New coverage, each written against the failing behavior first: — a required gate the stack never reaches must be MISCONFIGURED, and the stock config must still reach GREEN — load failure and unclassified failure both report ; the classifier is pinned to sample output from pytest, jest, vitest, go and cargo; co-located files stay out of the revert; the default path runs under bash 3.2 — every spelling of a push at a protected branch blocks, while claiming and releasing a claim branch still works — a repo where skipped must not report healthy
I ran an adversarial review of this template at (structural pass plus a defect hunt, every finding below re-verified against the tree before writing this). The design essay holds up well — the five design rules, the autonomy-tracks-consequence inversion, ADVICE.md's escalation ladder. These findings are all in the deterministic layer, and the five big ones share a property worth naming: they are silent on the documented happy path, with no attacker and no concurrency, and each one removes exactly the guarantee the green signal is supposed to carry. 1. The PR-verify routine never fires as documented has the implement routine open a draft PR unconditionally; has the verify routine trigger on with "Is draft is false". Follow both as written and stage 3 never runs on any factory PR — and it's worse than a filter mismatch, because GitHub emits , not , when a draft is later promoted, so the routine misses promotion too. Downstream effect: nothing in the repo machine-parses the line. With the cold verify stage dead, every check on a factory PR runs inside the implementer's own session, and the verbatim-quoted verdict line degrades to a string the writer pasted into its own PR body. The writer≠grader architecture quietly becomes writer-grades-writer. (Also: makes draft conditional on four triggers, contradicting — the two documents disagree about what a non-draft factory PR means.) 2. accepts any non-zero exit as proof — the entire signal is . Reverting the non-test hunks deletes the implementation, so for the most common change shape (new module + test that imports it), the post-revert failure is an import/collection error, not an assertion failure. A test containing gets — an assertion-free test certified as proving the fix, by the check calls "the check that catches the most real defects." Fixing this properly means classifying the failure (assertion vs. load/collection) from runner output or per-runner exit codes (pytest usefully distinguishes 1 from 4/5; jest/vitest need output parsing), and failing closed to something like when classification isn't possible. 3. is never cleared, so back-pressure jams permanently keeps the label until a human merges or closes — but nothing removes it, and the PR body template () links the issue without a keyword, so merging doesn't close the issue either. Merge N ≥ STOP_IF items over a couple of weeks and every subsequent implement run stops at the throttle check forever, over an empty review queue; a week later the monitor files an issue blaming the human review queue for work that already shipped. 4. Claim branches are never released; any post-claim abort makes the issue permanently unclaimable On failure after claiming, returns the issue to a live label but nothing deletes . A fresh branch can never fast-forward the surviving ref, so every subsequent run selects the same item (highest confidence first), loses the push race against a ghost, treats it as "already claimed," and stops — burning the entire daily run. It's invisible to monitoring: staleness checks only watch , and an item stuck at looks like a healthy queued item. Related: triage's sweep () doesn't exclude / issues from re-triage — with the instant trigger installed, triage can strip from an item mid-implementation, advertising it as claimable while its claim ref is live, which manufactures the same deadlock. 5. reports GREEN for a required gate the stack detection never reaches The MISCONFIGURED path only exists inside (). A required gate that the DETECT block never emits a or for leaves . Two configs that follow 's own comments hit it: (mutation is gated to , so at it's neither run nor skipped), and on a Python repo (the python arm has no line). The stock config avoids it on all four detected stacks, so passes; it bites the first time someone tunes gates.conf as invited. A closing sweep — any name in the required list that appears in neither PASSED/FAILED/SKIPPED → MISCONFIGURED — would fail this closed. Smaller, verified : (force push to main) and both pass the regex — the alternative only matches before or , and a source-branch push never reaches the protected-name check. The hook guards as a file but not as a directory, so into trips nothing — the proof mechanism is the one load-bearing script with no mechanical write protection. itself is also absent from the settings.json Edit deny list. doesn't check that / were actually installed — on a mature repo where install.sh's skip-if-exists left them out, doctor prints PASS on a factory with no shell-level merge guard. expands on a possibly-empty array before its own emptiness guard at line 72 — under on bash 3.2 (macOS ) every invocation without dies with before printing a line. The default test-file patterns () miss pytest's co-located layout: classifies as non-test, so the new test is bundled into the revert and deleted; the focused run then exits 4/5, which (per #2) still prints PROVEN. comments carry no authorship check, and the "issue text is untrusted" rule appears in the triage skill and CONTRACT.md but in neither nor — the two consumers of the fields. On a public repo, any commenter can post a second handoff (the duplicate-handoff rule then drains the item to ) or a downgrade that both the implementer and verifier honor. Back-pressure is read at run start () but the counted label is applied at run end (line 151), so two overlapping runs both pass a limit of 3 at depth 2 and land the queue at 4. What checked out clean Worth saying since I went looking: is injection-safe ( + , no in , minimal permissions), the claim push race is a genuine compare-and-swap and correct for the case covers (the defects are in release/re-entry, not the primitive), and 's trap/restore ordering, clean-tree refusal, and path handling are all right. Happy to split any of these into separate issues if that's easier to track. 1, 3, and 5 look like small unambiguous fixes; 2 is the deep one.
Summary explain how far teams can get with stock Claude Code or Codex primitives give a concrete issue-to-draft-PR starter workflow clarify when Factory earns its queue, handoff, verification, and back-pressure machinery make verification cost and tuning an explicit engineering budget link the advice from the README onboarding path Documentation basis Product claims were checked against current official Claude Code and OpenAI documentation for goals, loops or scheduled tasks, routines, project instructions, skills, and hooks. Validation
Repository: addyosmani/factory. Description: A reference software factory for Claude Code and Codex Stars: 199, Forks: 15. Primary language: Shell. Languages: Shell (100%). License: MIT. Topics: agentic-engineering, claude-code, codex, software-factory. Open PRs: 1, open issues: 1. Last activity: 1mo ago. Community health: 57%. Top contributors: addyosmani.