Premium lockedPySparkIntermediate
Too Many Small Files from Hourly Writes
Each hourly job writes many tiny files into the same date partition. Metadata overhead dominates scan time.
Practice type
Log / Error Analysis
Estimated time
17 min
Skills
PySpark, Small Files, Compaction
Create an account to continue
Enter your name and email. We will send a secure OTP before you choose a plan and pay with Razorpay.