AWS S3
.json
.csv
raw log
Databricks Cluster
Python 3.11 / PySpark
Raw Ingest & Parse
PySpark Transform
Schema Validate
Structured Delta Table
id
timestamp
value
101
12:00:01
42.50
102
12:00:02
18.90
103
12:00:03
99.10
CLEAN