misterspec

Features · Specs 011–017, 035–036, 038

The Context Engine

misterspec's own "Second Brain" — a local-first, deterministic retrieval layer that finds and assembles the smallest sufficient project context for a given operation, so a coding agent doesn't have to re-explore the whole repository by hand every time. It never replaces the agent's own judgment about what the returned context means — only what to hand it in the first place.

011 Wikilinks 012 References 013 Chunking 014 SQLite Index 015 Collector 016 Ranking 017 CLI 035 Budget Accuracy 036 BM25 Search 038 Provenance

What it is: Parsing and structural validation for [[SPEC-014]] / [[SPEC-014|alias]] syntax inside an artifact's own body.

Why it exists: Formal frontmatter fields (depends_on, parent) already capture structural relationships — wikilinks add an explicit, navigable semantic layer for "this is relevant to that" connections a formal field was never meant to carry.

Example — a wikilink inside a Spec's own body:

## Related Knowledge

See [[KNOW-003|Authentication Model]] for the constraints this Spec assumes.

What it is: Query an artifact's own outgoing formal + semantic relationships (references), and every artifact that points back to it (backlinks).

Why it exists: Turns 011's own parsed wikilinks, plus the formal fields foundation already tracks, into an actually queryable graph — the exact building block the Collector (015) traverses.

Example:

misterspec internal references SPEC-014
{"ok":true,"references":{"formal":[{"relation":"depends_on","target":"SPEC-011"}],
"semantic":[{"relation":"wikilink","target":"KNOW-003"}]}}

013 · Document Model and Chunking

What it is: A body/section representation (Document/Section/Chunk) with line-range provenance, plus a lightweight token estimator.

Why it exists: Retrieval needs to work on pieces of an artifact (a heading and its own content), not the whole file — and every chunk needs to know exactly where it came from, for both ranking and human review.

Example — one chunk's own provenance:

type Chunk struct {
    ArtifactID string
    Heading    string  // "## Requirements"
    Content    string
    StartLine  int
    EndLine    int
}

014 · Disposable SQLite Index

What it is: A local, fully-rebuildable FTS5 full-text index over documents, chunks, and links — stored at .misterspec/cache/context.db, never authoritative, never committed.

Why it exists: Fast lookup and free-text search need some index, but Constitution Principle III forbids any secondary authoritative state — deleting this file at any time must cost nothing but a re-sync.

Example — deleting the cache costs nothing:

rm -rf .misterspec/cache
misterspec internal context SPEC-014   # transparently rebuilds, same result

015 · Context Collector and Retrieval

What it is: Collect — gathers a deduplicated, tier-labeled candidate set: mandatory content (Constitution + target), formal structural relationships, direct wikilinks/backlinks, free-text matches, and one bounded second-hop graph expansion.

Why it exists: This is where 011-014 actually come together into one answer to "what's relevant to this target?" — explicit project structure is always consulted before probabilistic text similarity, never the reverse.

Example — one candidate's own tier and reason:

{"path":"ai/.../SPEC-011/spec.md","tier":"structural","reasons":["depends_on"]}

016 · Ranking and Budgeting

What it is: Rank assigns every candidate a deterministic score used only to order candidates already sharing the same Tier — Tier always dominates Score, an absolute, never-crossed invariant. ApplyBudget then fits the ranked list into a token budget, tier by tier, always preserving mandatory content in full. A separate, harder ceiling (035 · Budget and Estimator Accuracy) is what actually decides whether mandatory content counts as "exceeded" — the budget here only ever shapes optional content.

Why it exists: A strong but coincidental textual match must never outrank a real dependency, no matter how well it happens to match the request's own words — and a budget has to actually mean something, not just sort everything by score and cut wherever the numbers run out.

Example — a well-fitting budget, nothing trimmed:

{"budget":6000,"budget_exceeded":false,"diagnostics":{"reduction_percent":0}}

017 · Internal Context Command

What it is: misterspec internal context — the one CLI entry point orchestrating 014's index readiness, 015's collection, and 016's ranking/budgeting into one stable JSON response.

Why it exists: This is the actual, callable surface every Context-Pack-aware Skill (see Multi-Agent Skill Integration) uses — see it fully documented, flags and all, in the Command Reference.

Example — a deliberately tight budget, showing mandatory content always wins (note: budget_exceeded here compares against the hard limit, not --budget itself — see 035):

misterspec internal context SPEC-017 --intent planning --budget 300 --hard-limit 300
{"ok":true,"context":{"schema_version":2,"budget":300,"budget_exceeded":true,"overage":7,
"items":[/* Constitution + target only — everything else trimmed */],
"diagnostics":{"reduction_percent":80.0,"estimator":"default","hard_limit":300}}}

033 · Context Pack Output Contract

What it is: A new --mode flag on internal context — manifest (default, metadata only, byte-identical to before), package (every selected item's own full content, a file-absolute location, and a content fingerprint, delivered directly), or markdown (an explicit name for the existing --render shape). Every response also names its own schema_version explicitly, and no mode duplicates the same selected content in two representations at once.

Why it exists: Before this, the default response carried only metadata (path, heading, score, tokens) — never the actual text — so every Context-Pack-aware Skill had to re-read the source files it was just told about, paying the mechanical cost 017's own single entry point was supposed to eliminate. --mode package closes that gap: an agent gets everything it needs to act, in one call, with each item independently locatable and fingerprinted so a later divergence from the source is always detectable.

Example — full content, delivered directly:

misterspec internal context SPEC-014 --mode package
{"ok":true,"context":{"schema_version":2,"target":"SPEC-014",
"items":[{"path":"ai/.../SPEC-014/spec.md","heading":"R1 — First","tier":"mandatory",
"content":"### R1 — First\n\nThe system MUST ...\n",
"location":{"path":"ai/.../SPEC-014/spec.md","start_line":14,"end_line":18},
"fingerprint":"sha256:3a7bd3e2360a3d..."}]}}

An existing caller that never passes --mode sees no change at all — package is strictly additive, and the two modes are never mixed in the same response (full detail in the Command Reference).

035 · Budget and Estimator Accuracy

What it is: Makes 016's own budget promise actually true. The Estimator that produces every token count is now identified by name in the response (diagnostics.estimator). The single budget number is split into two independent limits: a soft budget (the target for optional content) and a separate, always-finite hard limit (the ceiling above which mandatory content is reported as exceeded) — a small --budget alone can no longer misleadingly flag mandatory content as "exceeded." ApplyBudget also stopped giving up on a whole priority tier the instant one candidate didn't fit: it now keeps trying smaller, later candidates of the same tier (diagnostics.exclusions records what was skipped and why), and an oversized single section can be included as a coherent, un-truncated prefix instead of excluded outright.

Why it exists: The Estimator interface existed in the code since 013 but was never actually used — nothing said which method produced a number. Separately, one budget serving two different jobs (optional-content target and mandatory-content ceiling) meant a small, deliberately narrow --budget for a quick lookup could wrongly read as a critical overage. See it fully documented in the Command Reference.

Example — decoupled limits, no false alarm:

misterspec internal context SPEC-017 --budget 500 --hard-limit 50000
{"ok":true,"context":{"schema_version":2,"budget_exceeded":false,"overage":0,
"diagnostics":{"estimator":"default","hard_limit":50000,"exclusions":[]}}}

036 · Robust Text Search and Measurable Ranking

What it is: Free-text queries (--query/--task) are now tokenized and escaped before reaching SQLite's FTS5 engine by default — ordinary punctuation can no longer throw a query-syntax error. An explicit --query-mode advanced opts into raw FTS5 grammar (phrase, prefix, boolean, NEAR) for callers who actually want it. Separately, same-tier free-text ordering now uses the real bm25() relevance value SQLite already computes, instead of a recomputed term-occurrence heuristic that was silently discarding it. A new diagnostics.ranking_version field identifies the scoring formula/weight set behind a response's own ordering — bumped only after comparative evaluation evidence supports a change, never silently under the same value.

Why it exists: A query containing a colon, quotes, or parentheses (exactly the kind of text a real requirement or error message contains) could previously fail outright instead of returning results — the tier-first ranking invariant (016) was never at risk, but the tie-break signal within a tier was weaker than the data SQLite already had.

Example — punctuation that used to fail now just works:

misterspec internal context SPEC-014 --query 'retry-policy: "at most once" (idempotent)'
{"ok":true,"context":{"schema_version":3, ...}}

What it is: Every wikilink-based reference now carries its own provenance through to the Context Pack: which artifact wrote it, which section it falls under, and its own file-absolute line — not just "some artifact references this one." internal references/internal backlinks report source_path/source_section/source_line on every entry (empty/zero for a formal relation, which has no line-level wikilink origin); internal context exposes the same detail per item behind a new opt-in --provenance flag. A still-experimental, off-by-default --prefer-section flag can additionally score a reference written in a Requirements-bearing section above one written elsewhere — gated behind the same evidence-based promotion process as 036's own ranking changes (see Dogfooding & Evaluation) before it can ever become default ordering.

Why it exists: Retrieval already included referenced content — it just couldn't explain why precisely enough to trust or act on. Recording the exact occurrence turns "SPEC-014 references this, somewhere" into "SPEC-014 references this from its own Requirements section, line 42" — a maintainer or agent can judge relevance without opening the source file.

Example — an item's own exact origin:

misterspec internal context KNOW-003 --provenance
{"ok":true,"context":{"schema_version":4,
"items":[{"path":"ai/.../SPEC-014/spec.md","tier":"semantic","reasons":["backlink"],
"provenance":[{"source_path":"ai/.../SPEC-014/spec.md","source_section":"Requirements","source_line":42}]}]}}

The underlying local index (014) widened its own links table to store this — a version bump the index's own existing disposable-rebuild mechanism already handles transparently; no manual migration, ever (Constitution Principle III).

040 · Stable Section Anchors

What it is: A wikilink can now target one specific section of an artifact — [[KNOW-003#retry-policy|Retry Policy]] — via an explicit, stable anchor declared on the heading itself (## Retry Policy {#retry-policy}), independent of the heading's own title text. Resolving such a reference retrieves just that section (plus the minimal ancestor heading context needed to keep it meaningful) instead of the whole target artifact; the item carries heading_path/anchor in the response.

Why it exists: Before this, pointing at "the retry policy inside KNOW-003" meant pulling in the entire Knowledge artifact just to reach one paragraph — expensive, and fragile the moment the artifact grows unrelated sections. A stable, explicitly-declared anchor survives a heading rename (the old §062 wikilink-foundation resolution stays based on the artifact ID; the anchor is a second, independent coordinate), and a broken/duplicate/unknown anchor is a validation Finding, never a silent fallback to the whole document.

Example — an anchor-qualified reference resolves to just that section:

misterspec internal context KNOW-001
{"ok":true,"context":{"schema_version":5,
"items":[{"path":"ai/knowledge/KNOW-003-x.md","heading":"Client Retry Policy",
"heading_path":["Networking Policies"],"anchor":"retry-policy", ...}]}}

internal validate gains two new Codes for this: unknown_anchor (a wikilink names an anchor the target artifact doesn't declare) and duplicate_anchor (two sections in the same artifact declare the same explicit anchor) — see the Command Reference.

043 · Incremental Context Pack Reuse

What it is: A new --base <pack_id> flag on internal context --mode package. Every --mode package response now names its own deterministic pack_id (a hash of the request's own resolved configuration plus every selected item's own content); passing a previously returned pack_id back in as --base returns only what changed since it — an ordered reuse-or-replace diff, never the unchanged content resent — or, when that base can't be confirmed, the full package again, explicitly marked recovered: true.

Why it exists: A caller that already fetched a Context Pack once — starting the next Task in the same Spec, or resuming after its own context got compacted — paid the full cost again every time, even when nothing (or almost nothing) had changed. --base lets it ask for just the difference instead, with one non-negotiable safety rule: a claimed base is a lookup key, never an assertion — an unknown or config-mismatched pack_id always falls back to a full, clearly-marked response rather than a silently incomplete one. The underlying pack cache is explicitly disposable (a new table in 014's own local index) — losing it only ever costs one call's worth of reuse, never correctness.

Example — repeating the same request, nothing changed:

misterspec internal context SPEC-014 --mode package --base sha256:6614b8dd...
{"ok":true,"context":{"schema_version":6,"pack_id":"sha256:6614b8dd...",
"diff":{"base_pack_id":"sha256:6614b8dd...","entries":[{"kind":"reuse","base_index":0}],"removed":[]},
"reuse":{"items_reused":1,"items_sent":0,"tokens_saved_estimate":10}}}

This local reuse mechanism is entirely independent of any LLM provider's own prompt caching — reuse reports a locally measured item/token count, never a claimed dollar or provider-side cost saving. Full flag/response detail in the Command Reference.