The Data Foundry
Built by Data with Pranjal
All Practice
Choose a guided lab or practice track
SQL Lab
Interview SQL with real data and validation
Python Lab
Data engineering Python practice
PySpark Lab
Spark code review and production fixes
Airflow Lab
DAG incidents, retries, sensors, and backfills
AWS Lab
Service choices, architecture, security, and cost
System Design Studio
Architecture trade-offs and interview framing
The Data Foundry is improving every week. If something feels unclear, broken, or missing, tell us and we will use it to improve the platform.
Practice Spark production fixes: inspect the data, repair the code, run a concept check, then compare with the model answer.
Labs
70
Attempted
0
Completed
Free
10
Choose a question first
Open any PySpark question from the library. Once you choose one, the question navigator stays on the left so moving to the next lab is quick.
The job appends the full day each time it runs. In production, retries and backfills must be idempotent so the same input does not create duplicate warehouse rows.
Row-wise Python UDFs move data through Python workers and block many Spark optimizations. This cleanup can be done with native Spark SQL functions.
A hot customer_id value is concentrating too many rows into one shuffle partition. The fix depends on table sizes and whether the dimension can be broadcast.
Too many small files increase metadata overhead and scheduling cost. The job needs better write sizing and a compaction strategy.
Caching is useful only when an expensive DataFrame is reused. Caching one-time DataFrames consumes memory and causes eviction/spill overhead.
collect and toPandas pull distributed data into driver memory. Production fixes should keep aggregation and filtering distributed.
dropDuplicates is not deterministic for choosing the latest record. You need an explicit ordering rule.
DataFrame union aligns columns by position, not by name. Different column order can silently corrupt data.
Invalid/default keys can funnel many rows into the same processing path and hide a data quality issue. Treat invalid keys separately instead of forcing them through the normal join.
Exploding multiple arrays in the same row can multiply row counts. Some nested structures should become child tables rather than one wide fact.
Taking to_date directly on a UTC timestamp assigns late-night local events to the wrong business day.
Schema inference can be unstable across files and dates. Production ingestion should define contracts and capture unexpected fields.
Broadcast joins are powerful only when the broadcast side is small enough. Broadcasting a large fact can crash executors.
High-cardinality partition columns create too many physical partitions and metadata overhead.
Late-arriving facts require rewriting an affected window, not only the current processing date.
Watermarks bound state size but also define how late data can arrive before being dropped from stateful aggregation.
Exact distinct counts are expensive at high cardinality. Some reporting use cases can use approximate distinct with an agreed error tolerance.
Hard-coded status filters are brittle. Production pipelines need normalized status mapping and unknown-value monitoring.
File readers need explicit corrupt-record handling so malformed input is counted, quarantined, and replayable.
coalesce(1) forces output through one task. Production exports should stay distributed unless the file is truly tiny.
The Endless Final Stage tests Data skew in shuffle-heavy stages. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Shuffle Storm After a Big Join tests Exploding shuffle volume. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
One Core Busy, Cluster Idle tests Too few partitions and poor parallelism. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Thousands of Tiny Tasks tests Too many partitions and scheduler overhead. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
The groupByKey Memory Trap tests Using groupByKey on large keys. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Window Functions on Unbounded Partitions tests Expensive window operations at scale. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
The Python UDF Tax tests Serialization overhead from Python UDFs. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Broadcast Join Backfires tests Unsafe broadcast of the build side. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Static Shuffle Settings on a Moving Workload tests Poor performance from outdated partition settings. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
The Recompute Loop tests Long lineage and repeated recomputation. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Executor OutOfMemoryError on Wide Rows tests Wide rows and oversized shuffles. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Driver Out of Memory tests Collecting too much data to the driver. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Cache Ate the Cluster tests Over-caching and missing unpersist. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Spill Everywhere tests Shuffle spill caused by heavy sort or aggregation. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Broadcast Timeout tests Broadcast exchange cannot complete in time. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Serialization Pain tests Expensive object serialization. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Dynamic Allocation Thrash tests Executors repeatedly added and removed. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Preempted or Lost Executors Kill the Stage tests Executor loss and fragile retries. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
GC Hell tests Excessive garbage collection. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Pandas UDF or Arrow Batch Blowups tests Oversized Arrow batches. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Reading Millions of Small Files tests Small-file scan overhead. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Writing a Forest of Tiny Files tests Too many output files. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Partitioned by User ID, Dying by User ID tests High-cardinality partitioning. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Some Tasks Read Gigabytes, Others Read Almost Nothing tests Uneven file sizes. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
CSV and Gzip at Terabyte Scale tests Unsplittable compression and text parsing. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Schema Inference Surprises tests Inferred types change across files. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Listing the Lake Takes Longer Than Reading It tests Metadata listing bottleneck. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Overwrite Wiped More Than Intended tests Unsafe overwrite semantics. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Table Gets Slower Every Month tests Missing maintenance and table optimization. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
MERGE Jobs Get Slower Every Day tests Unbounded merge scan. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
The Duplicate Explosion After Rerun tests Non-idempotent reruns. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Late Data Changes Yesterday tests Late-arriving data. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Incremental Load Missed a Slice of Data tests Unsafe watermark logic. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Timezone Drift Broke the Metric tests UTC and business-date mismatch. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Silent Casts Turned Data Into Nulls tests Unsafe type casting. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Join Produced Fewer Rows Than Expected tests Accidental inner join behavior. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
dropDuplicates Kept the Wrong Row tests Non-deterministic deduplication. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Schema Drift Broke Downstream Selects tests Unexpected source columns or missing fields. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Surrogate Keys Changed After Backfill tests Unstable key generation. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Joined Table Has Two Status Columns tests Ambiguous columns after joins. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
The Stream Cannot Catch Up tests Input rate exceeds processing capacity. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
State Store Keeps Growing Forever tests Unbounded streaming state. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Valid Events Are Being Dropped as Late tests Watermark too aggressive. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Streaming Join State Explosion tests Unbounded join state. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
One Kafka Partition Burns While Others Coast tests Kafka partition skew. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Checkpoint Trouble After a Restart tests Checkpoint incompatibility. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Exactly Once Until the Sink Retries tests Non-idempotent sink writes. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
foreachBatch Became the Bottleneck tests Slow sink-side upserts inside foreachBatch. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Restart Read the Wrong Offsets tests Accidental replay or skip after deployment. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Debugging Blind in Production tests Weak logs, metrics, and data-quality alarms. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.