# Pathrule Pattern: Apache Spark Data Processing (1.0.0) # ::pathrule:package:apache-spark ### [RULE] Keep production data distributed (path: /jobs) The driver coordinates work and has a finite heap. A call that materializes all rows there can pass on development input and terminate the application when production volume arrives. - Use distributed writes and aggregations for full results, and apply an explicit limit before collecting diagnostic samples. - Do not convert a Spark DataFrame to a local DataFrame unless a proven row and byte bound fits the driver with headroom. - Broadcast only when the relation's serialized size and expected growth are measured against executor memory; remove forced hints when the assumption is no longer durable. - Keep large lookup data in a distributed relation and let the physical plan choose or enforce a measured join strategy instead of building a driver-side map. See /schemas for the adjacent decision or procedure that completes this constraint. --- ### [RULE] Make writes retry-safe and partition-bounded (path: /jobs) Spark can retry tasks and whole jobs, while object stores and external sinks have different atomicity. A side effect performed per partition without an idempotency contract can be repeated after failure. - Define the logical output key from source interval, partition values, and transformation version so rerunning the same work replaces or returns the same result. - Stage output away from the published location and expose it only after completeness checks and commit metadata succeed. - Avoid uncoordinated external API or database writes inside row functions; use a sink with idempotent keys, batching, retry limits, and reconciliation. - Control output partition count from downstream file-size and parallel-read needs; do not coalesce everything to one file or emit one tiny file per task by default. See /tests/data for the adjacent decision or procedure that completes this constraint. --- ### [MEMORY] Partition count follows data volume and the next expensive boundary (path: /jobs) Partitioning is a workload decision. Too few partitions underuse the cluster and create oversized tasks; too many spend time scheduling and create tiny files. A good count can change after a filter or shuffle. - Inspect input split sizes and post-filter volume before the first wide transformation rather than assuming source file count describes useful work. - Repartition on the key required by an upcoming join or aggregation only when reducing skew or avoiding repeated shuffles justifies the movement. - Use coalescing for a measured reduction after data shrinks, and verify it does not collapse a skewed partition into a new bottleneck. - Record task duration, input bytes, shuffle bytes, spill, and output file distribution so future volume growth has a baseline. See /schemas for the rule or workflow that puts this decision into practice. --- ### [MEMORY] Skew is a data property with an explicit mitigation (path: /jobs) One partition holding a dominant key can keep an entire stage running after every other executor is idle. Adding executors does not split that key unless the data or operation changes. - Measure key-frequency distribution and task-duration spread on representative data; averages conceal the tail that controls stage completion. - Filter or process exceptional sentinel keys separately when they represent unknown or global data with different business semantics. - Pre-aggregate before a join when many rows can collapse at the same grain, reducing both shuffle volume and hot-key multiplication. - Use deterministic salting only with a plan to recombine results correctly, and document which keys require it so the workaround does not become invisible schema behavior. See /tests/data for the rule or workflow that puts this decision into practice. --- ### [SKILL] review-spark-physical-plan (path: /) --- name: review-spark-physical-plan description: Review an Apache Spark job before increasing its data volume or cluster schedule. --- # Review Spark Physical Plan Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. - [ ] Capture the logical and physical plan with representative parameters and identify every exchange, sort, broadcast, Python boundary, and repeated scan. - [ ] Compare estimated and observed relation sizes where available, then confirm join strategy assumptions under the largest expected dimension and hottest key. - [ ] Review stage task distributions for duration, input, shuffle, spill, garbage collection, and failed attempts rather than only total job time. - [ ] Inspect output file count and size distribution and prove the chosen partition layout matches downstream reads and object-store behavior. - [ ] Kill or retry the job during write and rerun the same interval; verify no duplicate side effects or partial published output remains. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.