Lead Software Engineer, Data Engineering
ValGenesis, Chennai
What you will do
- Architect, Design, develop, and maintain data ingestion, transformation, and orchestration pipelines (batch and real-time).
- Build and optimize data Lakehouse architectures using Azure Synapse, Delta Lake, or similar frameworks.
- Integrate and manage structured and unstructured data sources (SQL/NoSQL, files, documents, IoT streams).
- Develop and operationalize ETL/ELT pipelines using Azure Data Factory, Databricks, or Apache Spark.
- Collaborate with Data Scientists to prepare and serve ML-ready datasets for model training and inference.
- Implement data quality, lineage, and governance frameworks across pipelines and storage layers.
- Work with BI tools (Power BI, Superset, Tableau) to enable self-service analytics for business teams.
- Deploy and maintain data APIs and ML models in production using Azure ML, Kubernetes, and CI/CD pipelines.
- Ensure scalability, performance, and observability of data workflows through effective monitoring and automation.