The Data Foundry
Built by Data with Pranjal
All Practice
Choose a guided lab or practice track
SQL Lab
Interview SQL with real data and validation
Python Lab
Data engineering Python practice
PySpark Lab
Spark code review and production fixes
Airflow Lab
DAG incidents, retries, sensors, and backfills
AWS Lab
Service choices, architecture, security, and cost
System Design Studio
Architecture trade-offs and interview framing
The Data Foundry is improving every week. If something feels unclear, broken, or missing, tell us and we will use it to improve the platform.
Practice Spark production fixes: inspect the data, repair the code, run a concept check, then compare with the model answer.
Labs
20
Attempted
0
Completed
Free
5
Choose a question first
Open any PySpark question from the library. Once you choose one, the question navigator stays on the left so moving to the next lab is quick.
The job appends the full day each time it runs. In production, retries and backfills must be idempotent so the same input does not create duplicate warehouse rows.
Row-wise Python UDFs move data through Python workers and block many Spark optimizations. This cleanup can be done with native Spark SQL functions.
A hot customer_id value is concentrating too many rows into one shuffle partition. The fix depends on table sizes and whether the dimension can be broadcast.
Too many small files increase metadata overhead and scheduling cost. The job needs better write sizing and a compaction strategy.
Caching is useful only when an expensive DataFrame is reused. Caching one-time DataFrames consumes memory and causes eviction/spill overhead.
collect and toPandas pull distributed data into driver memory. Production fixes should keep aggregation and filtering distributed.
dropDuplicates is not deterministic for choosing the latest record. You need an explicit ordering rule.
DataFrame union aligns columns by position, not by name. Different column order can silently corrupt data.
Invalid/default keys can funnel many rows into the same processing path and hide a data quality issue. Treat invalid keys separately instead of forcing them through the normal join.
Exploding multiple arrays in the same row can multiply row counts. Some nested structures should become child tables rather than one wide fact.
Taking to_date directly on a UTC timestamp assigns late-night local events to the wrong business day.
Schema inference can be unstable across files and dates. Production ingestion should define contracts and capture unexpected fields.
Broadcast joins are powerful only when the broadcast side is small enough. Broadcasting a large fact can crash executors.
High-cardinality partition columns create too many physical partitions and metadata overhead.
Late-arriving facts require rewriting an affected window, not only the current processing date.
Watermarks bound state size but also define how late data can arrive before being dropped from stateful aggregation.
Exact distinct counts are expensive at high cardinality. Some reporting use cases can use approximate distinct with an agreed error tolerance.
Hard-coded status filters are brittle. Production pipelines need normalized status mapping and unknown-value monitoring.
File readers need explicit corrupt-record handling so malformed input is counted, quarantined, and replayable.
coalesce(1) forces output through one task. Production exports should stay distributed unless the file is truly tiny.