The Data Foundry
Built by Data with Pranjal
All Practice
Choose a guided lab or practice track
SQL Lab
Interview SQL with real data and validation
Python Lab
Data engineering Python practice
PySpark Lab
Spark code review and production fixes
Airflow Lab
DAG incidents, retries, sensors, and backfills
AWS Lab
Service choices, architecture, security, and cost
System Design Studio
Architecture trade-offs and interview framing
The Data Foundry is improving every week. If something feels unclear, broken, or missing, tell us and we will use it to improve the platform.
Practice Spark production fixes: inspect the data, repair the code, run a concept check, then compare with the model answer.
Labs
70
In progress
0
Done
Free
10
Choose a question first
Open any PySpark question from the library. Once you choose one, the question navigator stays on the left so moving to the next lab is quick.
The job appends the full day each time it runs. In production, retries and backfills must be idempotent so the same input does not create duplicate warehouse rows.
Row-wise Python UDFs move data through Python workers and block many Spark optimizations. This cleanup can be done with native Spark SQL functions.
A hot customer_id value is concentrating too many rows into one shuffle partition. The fix depends on table sizes and whether the dimension can be broadcast.
Too many small files increase metadata overhead and scheduling cost. The job needs better write sizing and a compaction strategy.
Caching is useful only when an expensive DataFrame is reused. Caching one-time DataFrames consumes memory and causes eviction/spill overhead.
The Endless Final Stage tests Data skew in shuffle-heavy stages. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Shuffle Storm After a Big Join tests Exploding shuffle volume. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
One Core Busy, Cluster Idle tests Too few partitions and poor parallelism. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
Thousands of Tiny Tasks tests Too many partitions and scheduler overhead. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.
The groupByKey Memory Trap tests Using groupByKey on large keys. The telemetry table contains multiple signals. Rows with risk_score at or above action_threshold require action. The broken code uses a strict comparison and misses boundary alerts.