Associate Data Platform Engineer
impact.com, Cape Town
What you will do
- Pipeline development: Build and extend batch and streaming pipeline components under guidance. Write clean, tested Scala and Python code. Add Spark transformation steps, build Airflow DAGs, and implement Kafka consumers with senior support.
- Platform configuration: Configure platform components such as BigQuery tables, partitioning strategies, and Cloud Storage buckets with guidance.
- Observability & data quality: Build monitoring dashboards and alerting for job success, latency, and pipeline health. Implement data quality checks to catch anomalies, schema violations, and missing data.
- Operations & incident response: Shadow the on-call rotation, help investigate root causes, debug using logs, metrics, and traces, and maintain runbooks and operational documentation.
- Security, governance & cost awareness: Follow least-privilege access, secrets management, and PII handling by default. Consider the cost and performance implications of your changes with guidance.