# Pathrule Pattern: Local LLM Inference with Ollama and vLLM (1.0.0) # ::pathrule:package:local-llm-ollama-vllm ### [RULE] Pin model weights, tokenizer, format, and serving parameters (path: /models) A model name or mutable tag is not a reproducible release. Different weights, tokenizer files, chat templates, or quantization can change output and memory behavior without application code changing. - Record model repository and immutable revision or checksum, tokenizer and chat template identity, file format, quantization, and license or usage constraints. - Store generation defaults and stop behavior with the application evaluation baseline rather than relying on runtime-wide implicit values. - Verify downloaded artifacts before load and restrict model sources and custom code according to the execution trust policy. - Expose model identity in health and generation metadata so traces and evaluations can attribute output to the actual loaded artifact. See /src/inference for the adjacent decision or procedure that completes this constraint. --- ### [RULE] Admit requests within memory and latency budgets (path: /src/inference) KV cache and active sequence memory grow with context and concurrency. Accepting every request until allocation fails creates correlated latency spikes and process loss. - Validate prompt and requested output limits against the deployed model, runtime configuration, and available memory rather than the model's theoretical maximum alone. - Apply concurrency and queue limits with an explicit overload response and useful retry guidance; do not hide an infinite queue behind an API timeout. - Propagate cancellation into queued and active generation and release scheduler state promptly when clients disconnect. - Separate interactive latency and batch-throughput workloads or assign priorities so long generations cannot starve short user-facing requests. See /ops/inference for the adjacent decision or procedure that completes this constraint. --- ### [MEMORY] Runtime choice follows workload shape (path: /ops/inference) Ollama-style runtimes optimize convenient model management and local use, while vLLM-style serving emphasizes continuous batching and accelerator throughput. Neither choice is universally better. - Choose from request concurrency, latency objective, model count, hardware, batching opportunity, API compatibility, observability, and operator skill. - Keep a compatibility adapter at the application boundary so runtime-specific model names, parameters, and errors do not spread through product code. - Benchmark the exact model and context distribution on target hardware; published tokens-per-second numbers from another shape do not size this service. - Define model load, unload, warmup, health, and replacement behavior for the chosen runtime, including what happens to queued and active requests. See /src/inference for the rule or workflow that puts this decision into practice. --- ### [MEMORY] Quantization trades quality, memory, and speed by task (path: /models) Reducing precision can fit a larger model or more KV cache but may degrade structured output, long-context recall, multilingual behavior, tool use, or domain tasks unevenly. - Compare candidate quantizations with identical model revision, prompts, chat template, sampling, context distribution, and scoring. - Measure accelerator and host memory, load time, prompt processing, generation throughput, latency percentiles, and failure behavior. - Evaluate product tasks and critical slices, not only generic benchmarks, and retain outputs for qualitative review where automated scores are incomplete. - Treat a quantization change as a model release with evaluation, canary, rollback, and attributable identity. See /ops/inference for the rule or workflow that puts this decision into practice. --- ### [MEMORY] API compatibility does not guarantee behavior compatibility (path: /src/inference) An endpoint shaped like a popular provider can accept the same JSON while interpreting templates, stop sequences, tools, log probabilities, usage, or errors differently. - Define the subset of request fields the product supports and reject or translate unsupported values explicitly. - Normalize streaming event order, finish reasons, usage accounting, cancellation, and public errors behind the inference adapter. - Test structured output and tool-call formats with the exact model and runtime because schema adherence is a model behavior, not only an API feature. - Do not compare hosted and local costs or quality until prompt templates, context truncation, and generation settings are aligned. See /models for the rule or workflow that puts this decision into practice. --- ### [SKILL] benchmark-local-llm-service (path: /) --- name: benchmark-local-llm-service description: Benchmark an Ollama, vLLM, or other local inference deployment before production use. --- # Benchmark Local Llm Service Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. 1. Pin model, tokenizer, template, quantization, runtime, driver, hardware, and generation settings and record warm and cold conditions. 2. Replay representative prompt and output lengths, concurrency, streaming, tool, and structured-output workloads with product evaluation cases. 3. Measure load time, host and accelerator memory, prompt throughput, generation throughput, queue time, latency percentiles, errors, and quality scores. 4. Increase concurrency through saturation, then test queue limit, overload response, cancellation, client disconnect, model reload, and process restart. 5. Select the smallest capacity that meets quality and latency headroom, publish limits with model identity, and retain a prior artifact and runtime configuration for rollback. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.