Data Engineer (Databricks)
Onit, Inc., Pune, Maharashtra
What you will do
- Pipeline Development: Design, build, and maintain scalable batch and incremental pipelines in Databricks (PySpark, Spark SQL, Delta Lake) that ingest data from multiple source systems into a medallion (bronze/silver/gold) architecture.
- Data Integration & Modelling: Implement the canonical data model defined with the Data Architect, so figures reconcile consistently across every product surface.
- Entity Resolution: Build production pipelines that resolve duplicate and conflicting records across systems that were never designed to reconcile with one another.
- Data Quality Engineering: Profile source data, implement automated data quality checks and expectations, and build monitoring and reporting so data quality stays continuously visible against defined targets.
- Orchestration & Reliability: Orchestrate and monitor jobs with Databricks Workflows, optimise Spark performance and cost, and troubleshoot pipeline failures.
- Governance & Privacy: