Running the model locally was the easy part

The first decision was also the one that constrained every decision after it. The model is not the authority on its own output.

A deterministic compiler takes what the model produces and checks it against the source text the person actually wrote. Was this span really there? Does the claimed condition appear in the source, or is it the extractor's paraphrase, and is it labelled as one? Does the delivered form still carry the thing the source said to avoid? If any of that cannot be established mechanically, the output does not become executable knowledge. It abstains.

Written as a rule: the count of unverifiable outputs delivered as executable is zero, and that number is not allowed to move. Not for a better recall score, not for a capability that would otherwise be impossible, not for a demo.

Holding a zero like that is expensive. It means the system says nothing more often than a looser one would, and every experiment below was measured with it in place. It is also the only reason the rest of the design is defensible, because it converts every model question from "is the model good enough" into "can this particular output be verified", which is a question code can answer.

A set of extraction failures persisted that looked like exactly what a small model would get wrong. The obvious hypothesis was capacity: a 4B model is a 4B model, and some of this needs more.

That hypothesis is cheap to assert and cheap to believe, so it was tested rather than assumed. An 8B was brought in as a diagnostic control, explicitly never a production candidate, and run on the same frozen evaluation.

It did not move. The 8B scored 0.469 macro F1 where the 4B baseline scored 0.463, with the same zero on unverifiable output, and it took about two and a half times as long per call. The extra parameters bought nothing the evaluation could detect.

What did move was the shape of the question. Decomposing the task on the same 4B, asking several narrow questions instead of one broad one and re-deriving the compiler input from the same output, reached 0.635 on the same evaluation without adding a model call. The limit was not how much the model knew. It was how much was being asked of it in a single pass.

This is the finding the whole system rests on, and it is why the production model stayed small. A smaller model asked narrower questions, with a compiler checking the answers, outperformed a larger model asked a broad one.

The instinct with a small model is to train it on your own contracts. That instinct is correct often enough to be worth testing properly, so the evaluation was frozen first: a fixed set of items, a fixed scoring script, and a reference checkpoint with authority. If a new checkpoint regressed the reference, it did not ship. The eval had veto power over the people running it, which is the only arrangement under which an eval is worth having.

One round put three extraction contracts onto a single adapter. Every checkpoint regressed the frozen reference, and the new capability the training was meant to buy came out worse than what the untrained base model already produced on the same contract. Nothing shipped from that round. It cost a night of compute and returned a firm answer to a question that would otherwise have stayed an opinion: the contracts do not fit together at that size.

A narrower round asked whether one primitive deserved a specialist of its own. No checkpoint cleared the acceptance gate, and the one with a genuine product argument bought its precision by proposing about a third as much, which is a worse product even though it is a better precision number. For that primitive the base model, with the adapter deliberately omitted, stayed the production answer.

The system that ships is therefore a mix, and the mix was chosen by measurement rather than by symmetry: some primitives run against a trained adapter, one runs better without one.

Small models produce malformed output. The standard answer is schema-constrained decoding: constrain generation so the shape is guaranteed, and the parsing problem disappears.

It worked exactly as advertised. Output needing salvage went from 11 of 47 to 0 of 47 on the frozen evaluation, and from 10 of 20 to 0 on real memories. The shape problem was completely solved.

It also moved the numbers that matter. Recall fell from 0.95 to 0.86, precision from 0.62 to 0.49, and thirteen proposals disappeared. Forced into a shape, the model stopped saying true things it had been saying. The malformed output had been a symptom of the model reaching for something; removing the symptom removed the reach.

So it was rejected and the apparatus was reverted out of the production package. This is the kind of result that is easy to keep quiet, because a clean 0 of 47 looks like a win in isolation. It is worth stating plainly instead: constrained decoding is a real technique with a real price, and on this contract the price was higher than the benefit.

The second half of the work was delivery. Once knowledge is typed and structured, an agent does not need the paragraph any more, it needs the compiled part that applies to the task in front of it. That got built, and it is dramatic on the axis it targets: a narrative memory that took about 3,000 tokens as prose delivers in roughly 104 to 234 tokens compiled, a 13 to 30 times smaller knowledge payload.

The interesting question is what that does to an agent's actual token usage, and it is worth answering with a measurement rather than an inference. So a turn was decomposed: a bare coding agent run in an empty directory, to isolate its own framing, and then everything else attributed against it.

About 16,000 tokens of the fixed framing is the coding agent's own system prompt. About 7,800 more is its built-in tool schemas. Together that is roughly 83 percent of the frame, it belongs to the agent, and nothing Pathrule does can reduce it. Pathrule's share was about 3,000 tokens of injected prose, around 11 percent, plus a few hundred for path-scoped rules and the protocol.

Compiled delivery cuts that 3,000 to about 230. It is a large cut of the part that is ours, and it is a small part. Which is why this site does not claim that Pathrule reduces your token bill: on the measurement we ran, a 13 times smaller injection did not move the agent's fresh token count, and the decomposition above says exactly why it would not.

Knowing that with numbers is more useful than a percentage would have been. It says where the remaining headroom is not, which is the more expensive thing to learn late.

The effect that did hold is the one the system was actually for. Twelve tasks, twenty-four runs, one agent, one pinned model. Each task ran twice from a byte-identical copy of the same repository with the same prompt, and each hid a project convention that could not be derived from the files. The scoring script was written and frozen before any run.

Without Pathrule: twelve of twelve functionally correct, three of twelve compliant with the project's own conventions, eight violations. With Pathrule: twelve of twelve functionally correct, twelve of twelve compliant, zero violations, at the same tool-call total.

The agent was never the weak part, and the result should not be read as though it were. It completed the work either way. What changed was whether it completed it the way the project had already decided, and the difference between those two is a review cycle, repeated.

Three of the twelve did not separate the conditions, because in those the convention was recoverable from the repository itself and both runs complied. Those are reported alongside the rest. They mark where the effect stops, which is a boundary worth publishing rather than trimming.

A quantized 4B model, about 2.5 GB, delivered as a versioned bundle resolved through a signed manifest. It is activated only after its bytes match the manifest checksum, and activation is the last atomic step, so a failed or corrupt download never touches the version that currently works.

Pathrule Studio and Pathrule CLI share one install under your Pathrule home directory and drive the same resident server, so there is never a second copy or a second version. The model loads on demand and unloads after an idle window; a cold load of a 4B takes tens of seconds and after that it answers immediately.

When it is absent, or the Python runtime behind it is, the system reports which one and falls back to delivering the source knowledge. It does not quietly deliver something degraded and call it compiled. The failure modes are separate on purpose, so the interface can say "Python could not be set up" rather than "something went wrong".

Three things generalise, and none of them are about model selection.

Decide first what you will never ship, then measure everything with that constraint switched on. A quality bar introduced after the fact is negotiable, because by then there is a number it would cost you. A bar that predates the first training run is a design input, and it changes which architectures you even consider.

Give the evaluation authority over the people running it. Freeze the items and the scorer before the run, and let a regression against the reference stop a release without a discussion. Most of the value of the experiments above is not in any single verdict; it is that the verdicts were binding.

Ask the small model narrow questions and let deterministic code carry the rest. Every time the system looked like it needed more model, what it needed was a smaller question and a stricter check on the answer. The compiler is doing more work than the weights are, and that is the intended distribution of labour rather than a compromise on the way to a bigger model.