misterspec

Features · Specs 019, 037

Dogfooding & Evaluation

Before trusting the Context Engine for real work, this feature tested it against misterspec's own real development history — and established the project's own governing rule for ranking changes:

Tune ranking only from observed failures, never from intuition alone.

The methodology

misterspec's own repository doesn't use its own ai/ artifact model to develop itself — it uses this same Spec-Kit-style specs/ convention you're reading right now. So a small, real fixture project was built encoding this project's own actual feature dependency graph (Specs 006 and 011–018, real depends_on relationships, real condensed summaries) as genuine misterspec artifacts, giving every retrieval check a real, verifiable right answer to test against — not synthetic or invented data.

The findings

  • Zero omissions — every one of the 9 evaluated Specs had 100% of its own real, historical dependencies correctly retrieved and tiered.
  • At the default budget (6000 tokens), nothing was ever trimmed against this small fixture — so a deliberately tighter budget (--budget 700, then --budget 300) was run against the most-connected Spec to actually exercise budgeting.
  • Under a realistic, tighter budget, every real dependency survived in full, and lower-priority content was trimmed first — exactly as Ranking and Budgeting is designed to behave.
  • A live Claude Code session confirmed /mister-plan really does call internal context before other exploration, and that the returned pack was sufficient on its own.

The outcome

No ranking or budgeting change was made. Every apparent over-inclusion observed at the generous default budget was fully explained by the fixture being smaller than that default — not by any flaw in the ranking algorithm — and the budget-constrained runs directly confirmed the design already works as intended. The evaluation's own full methodology, fixture, and findings are recorded in specs/019-dogfooding-evaluation/report.md in the repository.

From a one-off exercise to a reusable harness (Spec 037)

The manual exercise above proved the methodology, but had no repetitions, no cost accounting, and nothing reusable for the next retrieval or ranking proposal — every future change would otherwise have to invent its own ad hoc measurement from scratch. Spec 037 turned it into a standing, reusable evaluation harness: misterspec internal eval-retrieval runs a versioned set of deterministic retrieval cases (no LLM session required — safe to run in CI on every change), reporting a case-by-case pass/fail and the exact rank position of every required item. misterspec internal eval-compare then diffs a candidate run against a named, recorded baseline, flagging when more than one configuration dimension changed at once and surfacing a per-case regression even when an unrelated aggregate metric improves in the same comparison.

Live agent task-execution runs stay a documented, hand-recorded protocol — the binary deliberately never launches or drives a live coding-agent session itself (Constitution Principle IV); it only ever consumes the resulting record. Any future retrieval, ranking, or budgeting proposal (like 036's own BM25 change, or 038's still-experimental --prefer-section) is expected to record a baseline and compare against it through this harness before ever being promoted to default behavior — see the full commands in the Command Reference.