Local LLM Inference with Ollama and vLLM

Pathrule2 Rules • 3 Memories • 1 Skill

Local inference moves model download, format, quantization, GPU and host memory, context length, batching, concurrency, warmup, admission, and model provenance into the application's operational boundary. This pattern constrains model identity and request admission, records runtime and quantization decisions, separates interactive Ollama-style use from throughput-oriented vLLM serving, and supplies a capacity benchmark procedure. It differs from model routing and hosted-provider cost control by focusing on self-operated weights, hardware fit, scheduler behavior, and local runtime lifecycle.

Suggested path map

Pathrule places each piece on the matching path, so your assistant only sees it where it belongs. This is the scoping you get on import; you can adjust it in your workspace.

/ workspace root

benchmark-local-llm-service

models/

Pin model weights, tokenizer, format, and serving parameters

Quantization trades quality, memory, and speed by task

src/

inference/

Admit requests within memory and latency budgets

API compatibility does not guarantee behavior compatibility

ops/

inference/

Runtime choice follows workload shape

Rules

Pin model weights, tokenizer, format, and serving parameters/modelshighstrictResolve every deployment to immutable model artifacts and record context, quantization, template, and trust provenance.

1	A model name or mutable tag is not a reproducible release. Different weights, tokenizer files, chat templates, or quantization can change output and memory behavior without application code changing.
2
3	- Record model repository and immutable revision or checksum, tokenizer and chat template identity, file format, quantization, and license or usage constraints.
4	- Store generation defaults and stop behavior with the application evaluation baseline rather than relying on runtime-wide implicit values.
5	- Verify downloaded artifacts before load and restrict model sources and custom code according to the execution trust policy.
6	- Expose model identity in health and generation metadata so traces and evaluations can attribute output to the actual loaded artifact.
7
8	See /src/inference for the adjacent decision or procedure that completes this constraint.

Admit requests within memory and latency budgets/src/inferencehighstrictBound prompt tokens, output tokens, concurrent sequences, queued work, and cancellation before the runtime exhausts accelerator or host resources.

1	KV cache and active sequence memory grow with context and concurrency. Accepting every request until allocation fails creates correlated latency spikes and process loss.
2
3	- Validate prompt and requested output limits against the deployed model, runtime configuration, and available memory rather than the model's theoretical maximum alone.
4	- Apply concurrency and queue limits with an explicit overload response and useful retry guidance; do not hide an infinite queue behind an API timeout.
5	- Propagate cancellation into queued and active generation and release scheduler state promptly when clients disconnect.
6	- Separate interactive latency and batch-throughput workloads or assign priorities so long generations cannot starve short user-facing requests.
7
8	See /ops/inference for the adjacent decision or procedure that completes this constraint.

Memories

Runtime choice follows workload shape/ops/inferenceUse a developer-friendly local runtime for simple model lifecycle and a batching server when sustained multi-request throughput justifies its operational cost.

1	Ollama-style runtimes optimize convenient model management and local use, while vLLM-style serving emphasizes continuous batching and accelerator throughput. Neither choice is universally better.
2
3	- Choose from request concurrency, latency objective, model count, hardware, batching opportunity, API compatibility, observability, and operator skill.
4	- Keep a compatibility adapter at the application boundary so runtime-specific model names, parameters, and errors do not spread through product code.
5	- Benchmark the exact model and context distribution on target hardware; published tokens-per-second numbers from another shape do not size this service.
6	- Define model load, unload, warmup, health, and replacement behavior for the chosen runtime, including what happens to queued and active requests.
7
8	See /src/inference for the rule or workflow that puts this decision into practice.

Quantization trades quality, memory, and speed by task/modelsSelect precision from measured task quality and hardware behavior rather than file size or headline throughput alone.

1	Reducing precision can fit a larger model or more KV cache but may degrade structured output, long-context recall, multilingual behavior, tool use, or domain tasks unevenly.
2
3	- Compare candidate quantizations with identical model revision, prompts, chat template, sampling, context distribution, and scoring.
4	- Measure accelerator and host memory, load time, prompt processing, generation throughput, latency percentiles, and failure behavior.
5	- Evaluate product tasks and critical slices, not only generic benchmarks, and retain outputs for qualitative review where automated scores are incomplete.
6	- Treat a quantization change as a model release with evaluation, canary, rollback, and attributable identity.
7
8	See /ops/inference for the rule or workflow that puts this decision into practice.

API compatibility does not guarantee behavior compatibility/src/inferenceNormalize requests and responses, but validate unsupported parameters, token accounting, streaming, errors, and tool formats per runtime.

1	An endpoint shaped like a popular provider can accept the same JSON while interpreting templates, stop sequences, tools, log probabilities, usage, or errors differently.
2
3	- Define the subset of request fields the product supports and reject or translate unsupported values explicitly.
4	- Normalize streaming event order, finish reasons, usage accounting, cancellation, and public errors behind the inference adapter.
5	- Test structured output and tool-call formats with the exact model and runtime because schema adherence is a model behavior, not only an API feature.
6	- Do not compare hosted and local costs or quality until prompt templates, context truncation, and generation settings are aligned.
7
8	See /models for the rule or workflow that puts this decision into practice.

Skills

benchmark-local-llm-service/rootMeasure model quality, memory, throughput, latency, batching, overload, cancellation, and recovery on target hardware.

1	---
2	name: benchmark-local-llm-service
3	description: Benchmark an Ollama, vLLM, or other local inference deployment before production use.
4	---
5
6	# Benchmark Local Llm Service
7
8	Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result.
9
10	1. Pin model, tokenizer, template, quantization, runtime, driver, hardware, and generation settings and record warm and cold conditions.
11	2. Replay representative prompt and output lengths, concurrency, streaming, tool, and structured-output workloads with product evaluation cases.
12	3. Measure load time, host and accelerator memory, prompt throughput, generation throughput, queue time, latency percentiles, errors, and quality scores.
13	4. Increase concurrency through saturation, then test queue limit, overload response, cancellation, client disconnect, model reload, and process restart.
14	5. Select the smallest capacity that meets quality and latency headroom, publish limits with model identity, and retain a prior artifact and runtime configuration for rollback.
15
16	## Exit criteria
17
18	The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.

Why this pattern

AI agents often load a model by a mutable tag, advertise a context the hardware cannot sustain, accept unbounded concurrent generations, or compare quantizations with different prompts and sampling.

Built for AI platform teams serving open-weight models on developer machines, workstations, or GPU infrastructure.

Keeps your assistant from:

Serving different weights under the same application release
Exhausting accelerator memory through context and concurrency
Queueing requests beyond useful latency and cancellation
Selecting a faster quantization without evaluating task-quality loss

View as markdown

License: Apache-2.0
Version: 1.0.0
Updated: 2026-08-25