- Design scalable extract-transform-load (ETL) and extract-load-transform (ELT)
pipelines on AWS using Databricks, PySpark, Spark, Delta Lake, AWS Glue, SQL, and lakehouse patterns for batch and streaming workloads.
- Build ingestion, transformation, enrichment, calculation, allocation,
reporting, and publishing pipelines that produce maintainable, well-modeled data products.
- Develop production-grade Python, PySpark, and SQL with disciplined
engineering practices including testing, peer review, performance tuning, resiliency, and secure coding.
- Operate orchestration and scheduling workflows using Airflow (or equivalent),
including dependency management, retries, backfills, service level agreement monitoring, and operational runbooks.
- Implement continuous integration and continuous delivery practices with
automated quality gates (linting, unit and integration tests, security scans, data-quality checks, schema validation, and release controls).
- Build and maintain infrastructure as code using Terraform and/or AWS
CloudFormation, and create architecture, design, lineage, and operational artifacts that meet scalability, performance, cost, resiliency, governance, and security requirements.
- Lead adoption of AI-assisted engineering practices across the ETL lifecycle
(Copilot-assisted development, Claude-enabled refactoring and troubleshooting, and reusable AI skills and task-oriented agents) and integrate approved developer tools with Jira, source control, documentation, observability, incident triage, and release readiness—ensuring controls for human review, secrets protection, and safeguards for personally identifiable information and payment card information.