# Pathrule Pattern: LLM Fine-Tuning and Dataset Curation (1.0.0) # ::pathrule:package:llm-fine-tuning ### [RULE] Admit training examples through provenance and privacy review (path: /datasets) Training can memorize and redistribute patterns from data, and model weights are harder to delete or inspect than an ordinary record. A convenient internal log or scraped source is not automatically authorized training material. - Assign every source a provenance record and documented legal or product basis for the intended model and distribution. - Remove secrets, credentials, private keys, personal identifiers, unnecessary user content, and restricted material before formatting examples. - Honor deletion, retention, geographic, customer, and licensing constraints through dataset versions and derived artifacts, not an informal one-time export. - Quarantine uncertain sources and fail the dataset build when an example lacks the required provenance or policy fields. See /training for the adjacent decision or procedure that completes this constraint. --- ### [RULE] Keep evaluation data outside training and selection leakage (path: /evals) A model can appear improved by memorizing evaluation examples or near duplicates. Leakage also occurs when developers repeatedly tune prompts and checkpoint choice against the final test set. - Split by source, entity, time, conversation, or other leakage boundary before creating transformed examples. - Run exact and semantic duplicate checks across train, validation, test, and benchmark sources and review high-similarity clusters. - Use validation data for training and checkpoint decisions and reserve a sealed test set for final release comparison. - Track who accessed hidden references and regenerate compromised test slices rather than accepting an inflated metric. See /datasets for the adjacent decision or procedure that completes this constraint. --- ### [MEMORY] The training objective matches the production interaction (path: /training) A dataset of plausible text is not enough. If production uses chat templates, tool calls, structured output, or selective assistant loss, the training representation must preserve those semantics. - Pin the tokenizer and chat template from the base model and render training examples through the same canonical formatter used for evaluation. - Define which tokens contribute to loss and verify user instructions, tool outputs, padding, and control tokens are masked or weighted as intended. - Preserve multi-turn context only where it represents production behavior and remove accidental system or hidden annotation leakage. - Validate serialized examples by decoding samples and exercising the served model interface, not only by checking dataset columns. See /datasets for the rule or workflow that puts this decision into practice. --- ### [MEMORY] An adapter is coupled to one base-model identity (path: /training) Parameter-efficient adapters rely on the exact structure and weights they modify. Loading one on a similarly named but different base revision can fail or silently change behavior. - Pin immutable base-model and tokenizer revisions and store their checksums or repository identities with the run. - Record adapter method, target modules, rank, scaling, dropout, precision, quantization, and library versions. - Distinguish adapter-only, merged, and quantized deployment artifacts and test the exact one production will load. - Expose combined model identity in serving and retain the prior compatible base and adapter pair for rollback. See /evals for the rule or workflow that puts this decision into practice. --- ### [MEMORY] Checkpoint selection is a multi-objective release decision (path: /evals) Training loss measures fit to the training objective. It can improve while generalization, calibration, safety, long-context behavior, or product constraints regress. - Evaluate each candidate on fixed held-out tasks and critical slices with identical prompts, decoding, tools, and runtime. - Compare the base model and prior release, not only neighboring checkpoints from the same run. - Include refusal, privacy, memorization, structured-output, tool, and adversarial cases appropriate to the data and intended use. - Review qualitative failures and operational cost before selecting a checkpoint, and document accepted regressions explicitly. See /training for the rule or workflow that puts this decision into practice. --- ### [SKILL] curate-fine-tuning-dataset (path: /) --- name: curate-fine-tuning-dataset description: Create or revise a dataset for language-model fine-tuning. --- # Curate Fine Tuning Dataset Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. 1. Define the behavior objective, target distribution, exclusion policy, source rights, privacy constraints, and acceptance metrics before collecting examples. 2. Ingest through provenance records, remove prohibited data, normalize schema, and retain source-to-example lineage without exposing restricted content broadly. 3. Deduplicate within and across sources, resolve conflicting examples, and inspect length, language, topic, label, tool, and quality distributions. 4. Split by the correct leakage boundary, seal the test set, and render examples through the pinned tokenizer and production-compatible template. 5. Version the manifest, code, source identities, transformations, statistics, approvals, and hashes so the dataset can be reproduced or withdrawn. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks. --- ### [SKILL] release-fine-tuned-model (path: /) --- name: release-fine-tuned-model description: Release a fine-tuned model or adapter into an inference service. --- # Release Fine Tuned Model Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. 1. Resolve immutable base, dataset, code, recipe, random seeds, hardware, libraries, tokenizer, template, and output-artifact configuration. 2. Train with checkpoint, metric, resource, instability, and interruption evidence and reproduce a small run before spending the full budget. 3. Evaluate candidates against base and prior release on held-out quality, slices, safety, privacy, format, tools, latency, memory, and serving errors. 4. Package the exact adapter or merged artifact with base identity and load it in the production runtime for compatibility and capacity tests. 5. Canary bounded traffic with attributable model identity, monitor regressions and feedback, and roll back both artifact and serving configuration on acceptance failure. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.