Data Platform Engineer
impact.com, Cape Town
What you will do
- End-to-end domain ownership: Own platform components and a pipeline domain end-to-end, including design, implementation, reliability, performance, and cost. Design and build scalable ETL/ELT and streaming pipelines using Spark/Dataproc, Kafka, and Pub/Sub. Integrate diverse sources into BigQuery and other stores. Build modular, fault-tolerant components with graceful schema evolution.
- Infrastructure & performance tuning: Configure and maintain core platform infrastructure as code, including Dataproc, Kafka, Airflow/Astronomer, BigQuery, and Cloud Storage. Tune performance and cost through query optimisation, partitioning, and resource configuration. Tune streaming configurations for throughput and reliability.
- Reliability & incident response: Own and track SLOs for freshness, success, and latency. Implement monitoring, alerting, and data quality checks. Participate in on-call as a Tier 2 responder, troubleshooting and resolving incidents independently.
- Security, governance & self-service: Implement security and governance by design, including access control, secrets, encryption, audit logging, lineage, and retention. Build self-service capabilities and paved roads for analytics engineers, analysts, and data scientists.