# Pathrule Pattern: Pandas and DataFrame Pipelines (1.0.0) # ::pathrule:package:pandas-dataframes ### [RULE] Assert DataFrame schema at every external boundary (path: /src/schemas) Pandas inference changes with source values and missing data. A column that appears numeric in one file can become object data in another, and a timestamp can lose timezone meaning without raising an error. - Define the expected column set and fail on missing required columns and unexpected sensitive columns before transformation begins. - Parse identifiers as strings when leading zeros or mixed formats are valid, and parse numeric measures with explicit coercion and rejection reporting. - Normalize timestamps to an explicit timezone policy and distinguish date-only values from instants; do not compare naive and aware data implicitly. - Assert uniqueness and nullability for keys used in joins, deduplication, and output identity so later row-count changes have an attributable cause. See /src/pipelines for the adjacent decision or procedure that completes this constraint. --- ### [RULE] Validate merge cardinality and row-count intent (path: /src/pipelines) A merge can complete successfully while multiplying rows and corrupting totals. The dangerous case is an unexpected many-to-many relation hidden by duplicate keys on both inputs. - Check key nulls and duplicates on both sides before the merge, and use cardinality validation that matches the intended relationship. - Record input row counts, output row count, matched keys, left-only keys, and right-only keys as pipeline evidence. - Choose suffixes and projected columns deliberately so similarly named fields cannot be confused after the merge. - When many-to-many is intentional, estimate the expansion and aggregate at the correct grain before joining rather than accepting an unbounded Cartesian multiplication. See /tests/data for the adjacent decision or procedure that completes this constraint. --- ### [MEMORY] Assignment is explicit about the target frame (path: /src/pipelines) Selection can return an object that shares data or behaves as a copy depending on shape and operation. Production code should not rely on that ambiguity or suppress warnings without clarifying ownership. - Use `.loc` with row and column selectors for an in-place update to the intended DataFrame. - Call `.copy()` when a derived frame must own independent data, especially before mutation or returning it across a function boundary. - Prefer pure transformation functions that return a new named frame when the extra memory is acceptable and clarity outweighs mutation savings. - Do not disable chained-assignment diagnostics globally. Treat them as evidence that the target ownership is unclear and rewrite the selection or copy boundary. See /src/schemas for the rule or workflow that puts this decision into practice. --- ### [MEMORY] Missing values are domain states, not one universal null (path: /src/schemas) Pandas offers several missing representations across dtypes, but the business meaning must be decided before a convenient `fillna` changes unknown data into a real measurement or category. - Define allowed missing states per column and preserve nullable dtypes where absence is legitimate. - Do not fill measures with zero unless zero is the domain value for missing, and do not forward-fill across entity or session boundaries. - Keep invalid parse outcomes separate from genuinely absent input so data-quality failures can be reported and repaired. - Test aggregations with missing groups and values explicitly because defaults around dropping null groups or skipping null values can alter totals. See /src/pipelines for the rule or workflow that puts this decision into practice. --- ### [MEMORY] Vectorization is a correctness tool before a speed tool (path: /src/pipelines) Vectorized operations make dtype, alignment, and missing-value behavior visible at the column level. Row-wise callbacks hide repeated coercion and can return inconsistent types across records while paying Python call overhead. - Use arithmetic, comparisons, string accessors, datetime accessors, mapping, grouping, and window operations before reaching for row-wise `apply`. - When combining Series, check whether index alignment is intended; reset or align explicitly rather than accepting silent label-based gaps. - If a Python callback is unavoidable, define its input and output schema, measure it on representative volume, and isolate it from the rest of the pipeline. - Profile memory as well as runtime because chained vectorized expressions can create large temporary arrays even when they avoid Python loops. See /src/schemas for the rule or workflow that puts this decision into practice. --- ### [SKILL] validate-pandas-pipeline (path: /) --- name: validate-pandas-pipeline description: Validate a Pandas transformation after changing input, joins, dtypes, grouping, or output format. --- # Validate Pandas Pipeline Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. - [ ] Run the job against empty, minimal, representative, malformed, high-cardinality, duplicated-key, and missing-value fixtures. - [ ] Assert schema and row-count transitions after each join or aggregation and preserve unmatched-key diagnostics. - [ ] Shuffle input order and rerun; outputs that should be order-independent must remain identical after stable sorting by the declared key. - [ ] Measure peak memory and runtime at expected volume, then exercise chunking or column projection for sources that exceed the process budget. - [ ] Compare a canonical output sample and aggregate control totals against the approved baseline, explaining every intentional difference. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.