Engineer III - Data Engineer
Anaplan, Gurugram +1
What you will do
- Data Pipeline & Orchestration: Design, build, and run batch and scheduled data pipelines (Airflow, Dagster, Prefect) and lead orchestrator upgrades.
- Data Ingestion & Scraping: Own and extend data collectors and engines, including web/API ingestion, large-scale web scraping (Playwright, Selenium, Scrapy), parsing, normalization, and file merging.
- Distributed Processing: Build scalable distributed processing architectures (Spark on EMR, Databricks, Dataproc) and analytical query layers (Athena, Snowflake, BigQuery), including cross-cloud ingestion pipelines.
- Caching & Entity Matching: Improve entity matching and caching services (Redis, MemoryDB, DynamoDB) to optimize accuracy, latency, and operational cost.
- Data Access & APIs: Develop and maintain high-performance Python dataaccess layers and APIs for downstream ML and product teams.
- Data Quality & Observability: Own data quality initiatives, including validation frameworks, retention policies, backfills, monitoring, and OpenTelemetry-based observability.