Data QualityBeginnerLog / Error AnalysisFree

The Unit Tests That All Pass

You are the data engineer on call for this production path. Transformation unit tests pass, but the production pipeline fails because source volume and distribution differ. What would you investigate, how would you fix it safely, and how would you prove the issue is resolved?

Scenario context

The incident centers on test pyramid for data systems. The current implementation or operating process does not make that contract explicit, so the team needs a diagnosis supported by evidence rather than a tool or configuration guess.

Business requirement

Identify the most likely failure mechanism, propose a reversible production-safe fix, and explain validation, trade-offs, monitoring, and recovery.

Schema

Data Reliability evidence is shown below. Treat it as a production review artifact rather than a toy exercise.

Broken logic / code

pipeline_status=SUCCESS
freshness=PASS
row_count=PASS
business_reconciliation=FAIL
lineage_owner=UNKNOWN
runbook=NOT_FOUND

Logs / error

[Production review] Scenario 193: The Unit Tests That All Pass
Observed symptom: Transformation unit tests pass, but the production pipeline fails because source volume and distribution differ. What would you investigate, how would you fix it safely, and how would you prove the issue is resolved?
Core contract at risk: test pyramid for data systems.
Evidence to collect: Track test pass rate, false-positive rate, time to detect, time to recover, freshness, completeness, anomaly magnitude, and unresolved ownership. Add periodic review of thresholds, fixtures, lineage, and runbooks. Run reconciliation before and after backfills, migrations, and corrections.

Actual output

Transformation unit tests pass, but the production pipeline fails because source volume and distribution differ. What would you investigate, how would you fix it safely, and how would you prove the issue is resolved?

Expected output / expected logic

A strong response should define the contract, rank likely causes, propose a safe fix, and prove correctness with monitoring and reconciliation.

Your attempt

Write your root-cause analysis

Think before revealing the answer. A partial but honest attempt is better practice than reading the model solution first.

Saved

Interview-style explanation

Now explain your solution as if you are in an interview: symptom, root cause, fix, edge cases, trade-offs, monitoring, and prevention.