Apache Spark Data Processing

Pathrule2 Rules • 2 Memories • 1 Skill

Spark jobs fail at scale when small development inputs hide shuffles, skewed keys, tiny files, driver collection, unstable schemas, or non-idempotent writes that retries duplicate. This pattern constrains driver-bound operations and output identity, records partition and join strategy decisions, and provides a repeatable physical-plan and skew review for production datasets. It differs from Pandas by focusing on distributed execution, stage boundaries, partition movement, executor memory, and retry-safe commits rather than local DataFrame mutation semantics.

Suggested path map

Pathrule places each piece on the matching path, so your assistant only sees it where it belongs. This is the scoping you get on import; you can adjust it in your workspace.

/ workspace root

review-spark-physical-plan

jobs/

Keep production data distributed

Make writes retry-safe and partition-bounded

Partition count follows data volume and the next expensive boundary

Skew is a data property with an explicit mitigation

Rules

Keep production data distributed/jobshighstrictNever collect or convert an unbounded dataset on the driver; inspect samples and aggregates through bounded operations.

1	The driver coordinates work and has a finite heap. A call that materializes all rows there can pass on development input and terminate the application when production volume arrives.
2
3	- Use distributed writes and aggregations for full results, and apply an explicit limit before collecting diagnostic samples.
4	- Do not convert a Spark DataFrame to a local DataFrame unless a proven row and byte bound fits the driver with headroom.
5	- Broadcast only when the relation's serialized size and expected growth are measured against executor memory; remove forced hints when the assumption is no longer durable.
6	- Keep large lookup data in a distributed relation and let the physical plan choose or enforce a measured join strategy instead of building a driver-side map.
7
8	See /schemas for the adjacent decision or procedure that completes this constraint.

Make writes retry-safe and partition-bounded/jobshighstrictWrite under stable run identity, stage into isolated locations, and commit complete outputs without duplicate or partial visibility.

1	Spark can retry tasks and whole jobs, while object stores and external sinks have different atomicity. A side effect performed per partition without an idempotency contract can be repeated after failure.
2
3	- Define the logical output key from source interval, partition values, and transformation version so rerunning the same work replaces or returns the same result.
4	- Stage output away from the published location and expose it only after completeness checks and commit metadata succeed.
5	- Avoid uncoordinated external API or database writes inside row functions; use a sink with idempotent keys, batching, retry limits, and reconciliation.
6	- Control output partition count from downstream file-size and parallel-read needs; do not coalesce everything to one file or emit one tiny file per task by default.
7
8	See /tests/data for the adjacent decision or procedure that completes this constraint.

Memories

Partition count follows data volume and the next expensive boundary/jobsSize partitions from bytes, skew, cluster parallelism, and downstream file goals instead of copying one static number.

1	Partitioning is a workload decision. Too few partitions underuse the cluster and create oversized tasks; too many spend time scheduling and create tiny files. A good count can change after a filter or shuffle.
2
3	- Inspect input split sizes and post-filter volume before the first wide transformation rather than assuming source file count describes useful work.
4	- Repartition on the key required by an upcoming join or aggregation only when reducing skew or avoiding repeated shuffles justifies the movement.
5	- Use coalescing for a measured reduction after data shrinks, and verify it does not collapse a skewed partition into a new bottleneck.
6	- Record task duration, input bytes, shuffle bytes, spill, and output file distribution so future volume growth has a baseline.
7
8	See /schemas for the rule or workflow that puts this decision into practice.

Skew is a data property with an explicit mitigation/jobsIdentify hot keys and choose filtering, pre-aggregation, salting, or split processing based on their business meaning.

1	One partition holding a dominant key can keep an entire stage running after every other executor is idle. Adding executors does not split that key unless the data or operation changes.
2
3	- Measure key-frequency distribution and task-duration spread on representative data; averages conceal the tail that controls stage completion.
4	- Filter or process exceptional sentinel keys separately when they represent unknown or global data with different business semantics.
5	- Pre-aggregate before a join when many rows can collapse at the same grain, reducing both shuffle volume and hot-key multiplication.
6	- Use deterministic salting only with a plan to recombine results correctly, and document which keys require it so the workaround does not become invisible schema behavior.
7
8	See /tests/data for the rule or workflow that puts this decision into practice.

Skills

review-spark-physical-plan/rootInspect a Spark plan and stage evidence for scans, shuffles, joins, skew, spills, partitions, and retry-safe output.

1	---
2	name: review-spark-physical-plan
3	description: Review an Apache Spark job before increasing its data volume or cluster schedule.
4	---
5
6	# Review Spark Physical Plan
7
8	Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result.
9
10	- [ ] Capture the logical and physical plan with representative parameters and identify every exchange, sort, broadcast, Python boundary, and repeated scan.
11	- [ ] Compare estimated and observed relation sizes where available, then confirm join strategy assumptions under the largest expected dimension and hottest key.
12	- [ ] Review stage task distributions for duration, input, shuffle, spill, garbage collection, and failed attempts rather than only total job time.
13	- [ ] Inspect output file count and size distribution and prove the chosen partition layout matches downstream reads and object-store behavior.
14	- [ ] Kill or retry the job during write and rerun the same interval; verify no duplicate side effects or partial published output remains.
15
16	## Exit criteria
17
18	The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.

Why this pattern

AI agents often collect data to the driver, repartition blindly, broadcast an input that no longer fits, or write append-only output from a retried job without stable run identity.

Built for Data engineering teams operating batch or streaming Spark workloads on shared clusters.

Keeps your assistant from:

Crashing the driver by collecting production-scale data
Creating a single hot partition from a skewed key
Producing thousands of tiny output files
Duplicating output when a stage or job retries

View as markdown

License: Apache-2.0
Version: 1.0.0
Updated: 2026-08-25