Data Platform Engineer

Treeswift Inc, New York Office

What you need

Design, build, and maintain data pipelines at scale. We run Apache Airflow 3 on Astronomer with pipelines that process terabytes of real-world physical data across many file types—imagery, audio, point clouds, and more. You will develop and evolve DAGs that orchestrate complex, multi-step workflows: dozens of tasks, fan out/in in pipelines, Python and Kubernetes operators split across generalized and specialized node pools, and dynamic DAG generation. You will work closely with our in-house ML team (feature pipelines and model deployment live in these DAGs) and coordinate with our hardware team on ingestion and formats. Scope is a mix of pipeline development and platform ownership and we are happy to adjust the scope and balance of responsibilities based on your interests and strengths.
Help us scale and harden our data platform. We have one dedicated data engineer today; you will be the second. The broader engineering team is highly collaborative and you will work with members of the full-stack and machine learning teams. We are looking for someone to improve DAG design and execution, resource and cost tuning, reliability and observability, and contribute to how we run Airflow and Kubernetes in the cloud. If you enjoy writing pipelines and improving the platform that runs them, this role has room for both.

What you need

Design, build, and maintain data pipelines at scale. We run Apache Airflow 3 on Astronomer with pipelines that process terabytes of real-world physical data across many file types—imagery, audio, point clouds, and more. You will develop and evolve DAGs that orchestrate complex, multi-step workflows: dozens of tasks, fan out/in in pipelines, Python and Kubernetes operators split across generalized and specialized node pools, and dynamic DAG generation. You will work closely with our in-house ML team (feature pipelines and model deployment live in these DAGs) and coordinate with our hardware team on ingestion and formats. Scope is a mix of pipeline development and platform ownership and we are happy to adjust the scope and balance of responsibilities based on your interests and strengths.
Help us scale and harden our data platform. We have one dedicated data engineer today; you will be the second. The broader engineering team is highly collaborative and you will work with members of the full-stack and machine learning teams. We are looking for someone to improve DAG design and execution, resource and cost tuning, reliability and observability, and contribute to how we run Airflow and Kubernetes in the cloud. If you enjoy writing pipelines and improving the platform that runs them, this role has room for both.