PathruleHow we measure
PathruleBench, the benchmark behind Pathrule's numbers: the same coding agent, the same team knowledge, delivered through CLAUDE.md files or through Pathrule, measured on cost, input tokens and what reached later sessions, with the conditions where the gap closes.
On this page
Pathrule makes one primary measured claim, and it is about what the same agent has to read, and what still reaches it later, when a team's knowledge is delivered through Pathrule instead of through instruction files.
At 2,000 items of team knowledge, the same coding agent read 2.7 times fewer input tokens and cost about 55% less a chain with Pathrule than with CLAUDE.md files, in Python, Go and TypeScript, and passed more of the memory probes.
This page is the whole picture: the benchmark behind that sentence, its numbers, the conditions where the two channels tie, and the bounds on all of it.
PathruleBench
PathruleBench gives both channels the same thing: the same items of team knowledge, the same agent (Claude Code with claude-sonnet-5-5), the same prompts and the same hidden checks. Only the channel differs.
- CLAUDE.md files. The knowledge compiled into the instruction files Claude Code reads, written well and kept scoped. It is the common "just put it in CLAUDE.md" baseline, done properly rather than badly.
- Pathrule. The open-source CLI in local mode, built from the source of 16 October 2026: a short compiled region in the root file, the rest delivered per prompt by the hook. No Cloud, no embeddings, no model.
- No memory. The floor: the agent with no knowledge files at all, so the cost of the knowledge itself can be read off.
A chain is eight sessions of four turns on a small Python, Go or TypeScript toolkit. Across the chain, 26 memory probes check what reached later sessions: a habit stated once, a lesson a failing command taught, a decision that changed, seeded team knowledge. Every probe is checked by an executable test or a structural check. No model judges anything, and 19 further checks confirm that each session's own task was done in every arm.
The knowledge base scales from 500 to 1,000 to 2,000 items, because the question is not whether a small file works. It is what happens as the knowledge grows.
Cost and tokens a chain
| Condition | Cost, CLAUDE.md to Pathrule | Saving | Input tokens, CLAUDE.md to Pathrule | Fewer tokens |
|---|---|---|---|---|
| Python, 500 items | $2.06 to $1.70 | 17.5% | 5.5 M to 4.2 M | 1.3× |
| Python, 1,000 items | $2.62 to $1.71 | 34.8% | 7.4 M to 4.2 M | 1.8× |
| Python, 2,000 items | $3.89 to $1.76 | 54.7% | 11.8 M to 4.3 M | 2.7× |
| Go, 500 items | $2.13 to $1.91 | 10.0% | 5.4 M to 4.8 M | 1.1× |
| Go, 1,000 items | $2.75 to $1.91 | 30.5% | 7.3 M to 4.7 M | 1.6× |
| Go, 2,000 items | $4.21 to $1.89 | 55.2% | 12.5 M to 4.7 M | 2.7× |
| TypeScript, 500 items | $2.08 to $1.68 | 19.0% | 5.4 M to 4.0 M | 1.4× |
| TypeScript, 1,000 items | $2.68 to $1.86 | 30.4% | 7.3 M to 4.8 M | 1.5× |
| TypeScript, 2,000 items | $4.08 to $1.86 | 54.5% | 12.2 M to 4.5 M | 2.7× |
Cost is Claude Code's own accounting. Input tokens are everything the model read over the chain: fresh input, cache writes and cache reads, summed from the published session rows. With no memory a chain reads 2.8 to 3.3 million tokens. At 2,000 items, CLAUDE.md adds 8.4 to 9.4 million to that, because its files are read whole on every step, and Pathrule adds 1.0 to 1.6 million. Most of the difference is cache reads, which are billed far below fresh input, which is why cost falls less steeply than tokens.
Memory: what reached later sessions
| Condition | No memory | CLAUDE.md | Pathrule | Pathrule minus CLAUDE.md, 95% CI |
|---|---|---|---|---|
| Python, 500 items | 4.7 | 22.3 | 25.0 | +2.6 [+0.8, +4.9] |
| Python, 1,000 items | 4.3 | 20.8 | 25.0 | +4.2 [+1.6, +7.3] |
| Python, 2,000 items | 4.0 | 22.3 | 25.3 | +3.1 [+0.3, +6.0] |
| Go, 500 items | 4.3 | 23.7 | 26.0 | +2.3 [+0.3, +4.9] |
| Go, 1,000 items | 4.7 | 24.7 | 25.3 | +0.8 [+0.0, +1.6] |
| Go, 2,000 items | 5.3 | 25.0 | 26.0 | +1.0 [+0.0, +2.6] |
| TypeScript, 500 items | 3.7 | 24.3 | 24.7 | +0.3 [-1.3, +2.1] |
| TypeScript, 1,000 items | 3.7 | 22.0 | 25.0 | +3.1 [+1.3, +4.9] |
| TypeScript, 2,000 items | 3.7 | 22.3 | 24.0 | +1.6 [-0.8, +3.9] |
Memory probes passed out of 26, as the mean over three chains an arm, five for CLAUDE.md at Python 1,000. The 19 checks of what each session's own prompt asked pass in every arm and every chain, so neither channel cost the agent the task in front of it.
How to read it
Cost stays flat as knowledge grows. An instruction file is read whole on every step, so its cost grows with the knowledge base: from 500 to 2,000 items a chain went from $2.06 to $3.89 in Python, $2.13 to $4.21 in Go and $2.08 to $4.08 in TypeScript. Pathrule's root file stays around 6k characters and the rest arrives per prompt, so the same chains went from $1.70 to $1.76, $1.91 to $1.89 and $1.68 to $1.86.
The person's own words carry. A habit stated once reached later sessions 6.3 to 7 times out of 7 with Pathrule and 4.3 to 6.7 with CLAUDE.md, and the CLAUDE.md figure falls as the knowledge base grows: in TypeScript, 6.7 at 500 items and 4.3 at 1,000. A lesson a failing command taught was kept in 8 of 9 Pathrule chains in Python and 4 of 9 in TypeScript, and in none of the 20 CLAUDE.md chains in those two languages; both channels keep it in Go.
A changed decision is applied. In every Go condition Pathrule applied the newer decision 3 of 3 times, CLAUDE.md 1.7 to 2.7.
Where they tie. Seeded team knowledge, decisions stated in prompts (9.6 to 10 of 10 in both), knowing when to ask instead of deciding, and the whole TypeScript 500-item condition, where the memory difference is +0.3 with an interval of [-1.3, +2.1]. The gap opens from 1,000 items.
Where both miss. In Python and TypeScript about one chain in three writes a new error type with the domain the team moved away from, in both arms. The miss is an omission, a default left in place, not a wrong value.
Limits
- Three chains an arm, five for CLAUDE.md at Python 1,000. Intervals are bootstrap over the 26 probes; at TypeScript 500 and 2,000 items they include zero.
- The published conditions start at 500 items. With a small knowledge base a CLAUDE.md file is cheap to read and hard to beat, and no condition there is published.
- The Pathrule arm is the open-source CLI in local mode, built from the source of 16 October 2026: no Cloud, no embeddings, no model.
- Claude Code 2.1.301 and 2.1.302 with
claude-sonnet-5-5, run on 16 and 17 October 2026. Different agents read files and call tools at different rates. - Chat recall over long conversations, which LongMemEval and similar suites measure, is not what this measures.
What the numbers do not mean
- They are not guarantees. They are what we measured on the setups described above. Your results depend on your tasks, your codebase, how much of your team's knowledge is captured and the agent you run.
- They are size-dependent. The saving is a function of how much knowledge the instruction file would otherwise carry on every step. A team with fifty notes will not see it.
- They describe what the agent read and what reached it, not the size or quality of the code produced.
What we will and will not publish
Fixtures, scenarios, hidden checks, the harness and every chain, with each turn's token usage, are at github.com/pathrule/benchmark under Apache-2.0. We publish the conditions where the channels tie and where both miss, and the Pathrule arm is the same open-source CLI anyone can install.
We do not publish how Pathrule Cloud's ranking decides which knowledge to surface. If you want to understand the behaviour rather than the internals, how retrieval finds the right knowledge describes what you can observe, and safety and fallback describes what happens when Pathrule is not confident.