Lead Data Engineer
Redhorse Corporation, Remote
The role
Redhorse transforms the way government uses data and technology. To support this mission, we are seeking a Senior Data Engineer / Lead to join our team supporting the US National Defense Strategy. In this role, you will play a vital role in translating complex defense requirements into scalable data pipelines and actionable insights, directly contributing to the modernization of critical defense systems.
What you will do
- Technical Leadership & Task Oversight: Provide day-to-day task prioritization and technical direction to team members; lead code reviews, ensure design best practices, and mentor engineers in modular Python development and SQL optimization.
- Requirements Development & Architecture: Work directly with government stakeholders to elicit, refine, and translate complex mission requirements into robust technical specifications, interface designs, and scalable data models.
- Pipeline & Platform Development: Architect, develop, and optimize high-throughput ETL/ELT workflows using Databricks; maintain, refactor, and enhance existing Python and SQL codebases for performance, stability, and security.
- Pipeline Diagnostics & Troubleshooting: Identify pipeline bottlenecks, debug complex data workflows, and implement automated solutions to enhance pipeline reliability and data integrity.
- Cross-Functional Collaboration: Partner closely with government product owners, data scientists, Data Analysts, and technical peers to document architecture, coordinate technical sprints, and communicate engineering trade-offs to technical and non-technical audiences.
- Technical Documentation: Develop new and maintain existing pipeline architecture documentation relating to pipeline code architecture and configurations to ensure high system reliability and seamless team onboarding.
What you need
- Security Clearance: Must be a U.S. Citizen with a current, active U.S. Government Secret clearance. (Applicants without an active Secret clearance cannot be considered).
- Experience Level: Minimum of 8+ years of professional experience in data engineering, building enterprise-scale data platforms and pipelines.
- Technical Guidance: Demonstrated experience providing technical mentorship, performing code reviews, and providing day-to-day technical task oversight to other engineers.
- Databricks: Proven, hands-on experience designing and optimizing data transformation workflows in Databricks.