Jobs Ask AI Pay Interviews Companies CompareSign in

Wait…

We read every job from the company's own job board. You apply on the company site.Jobs Job types Internships New graduate jobs Companies Pay Reported pay Check an offer Add your pay Interview questions Add your interview questions Compare with LinkedIn Job sites compared Pay transparency laws API for developers Monthly report Companies we cover Make a portfolio page For companies: find people About Privacy

Lead Data Engineer

Black Canyon Consulting, Bethesda, MD +1

Posted pay

Pay not posted

Location: Bethesda, MD +1
Work type: Not given
Level: Senior
Experience: 6+ years
Skills in the job text: Python
AWS
SQL
Elasticsearch
Kafka
LangChain
Posted: Sep 25, 2026
Last check: 7 h ago
Job type: Data engineering

Apply on the Black Canyon Consulting site Sign in to save

The Black Canyon Consulting site opens in a new tab.

BC

Black Canyon Consulting

About Black Canyon Consulting: funding, hiring and pay

Check my fit

Compare your résumé with this job. See where you match, what is missing and which lines to change.

Sign in to check your fit

The role

Overview

We are seeking a Lead Data Engineer to design and implement the core analytical data platform for a federal permitting modernization initiative.

The Lead Data Engineer will establish a modern lakehouse environment using DuckDB, Apache Iceberg, Amazon S3, and AWS Glue Catalog and will lead the engineering patterns used to ingest, harmonize, and serve data across multiple source systems. The role will also help implement metadata-driven and agentic approaches to data engineering, including source analysis, mapping, pipeline generation, testing, reconciliation, and quality assurance.

This is a hands-on technical leadership role with responsibility for guiding and reviewing the work of other Data Engineers.

This position is remote.

What you will do

Design and implement the core DuckDB / Apache Iceberg data platform on AWS.
Define physical data organization, schema, partitioning, file layout, snapshot, compaction, and optimization strategies for Iceberg tables.
Integrate DuckDB with Amazon S3, Apache Iceberg, and AWS Glue Catalog.
Establish bronze, silver, and gold data-layer conventions and reusable engineering patterns.
Lead and mentor Data Engineers, providing technical direction, code reviews, and engineering guidance.
Build and maintain analytical datasets supporting application services, reporting, analytics, and data science.
Develop reusable patterns for incremental loads, CDC, schema evolution, late-arriving data, backfills, and historical reconstruction.
Work with the Data Architect to implement the common canonical data model and source-to-canonical mappings.
Lead migration and harmonization of datasets from multiple source systems.
Implement reconciliation processes that demonstrate completeness and accuracy between source and target systems.
Establish automated data-quality monitoring, operational observability, and testing.
Implement agentic and AI-enabled data engineering patterns supporting activities such as source analysis, mapping, pipeline generation, troubleshooting, reconciliation, and quality assurance.
Support integration of structured data with associated document and geospatial datasets.
Lead development of certified data products, analytical datasets, and business metrics.
Optimize DuckDB queries and Iceberg datasets for performance and efficiency.
Develop engineering documentation, operational procedures, and technical handoff materials.

What you need

Educational Requirements

Bachelor's degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related field.

Requirements

6+ years of relevant data engineering or data platform experience, including experience leading or mentoring other engineers.
Strong hands-on experience with DuckDB and Apache Iceberg.
Advanced SQL and strong Python development experience.
Experience building cloud-based analytical platforms using object storage and columnar formats such as Parquet.
Strong understanding of partitioning, schema evolution, snapshots, compaction, incremental processing, and analytical query optimization.
Experience implementing bronze/silver/gold or comparable layered data architectures and transforming heterogeneous enterprise datasets.
Experience with AWS data services, particularly Amazon S3 and AWS Glue.
Experience designing reusable, metadata-driven ingestion and transformation frameworks.
Experience implementing automated data quality, reconciliation, monitoring, and operational controls.
Experience with agentic or AI-enabled data engineering solutions, including the use of agents or AI-assisted workflows for activities such as data mapping, pipeline generation, validation, testing, or troubleshooting.
Hands-on experience with agentic frameworks such as LangChain, LangGraph, or similar frameworks for building tool-using or multi-step AI workflows.
Strong troubleshooting, performance tuning, and technical leadership skills.

Nice to have

Experience with Kafka or Amazon MSK and CDC architectures.
Experience with GeoParquet, PostGIS, or geospatial datasets.
Experience with OpenSearch, document stores, or vector search technologies.
Experience integrating enterprise SaaS platforms, APIs, databases, and file-based sources.
Experience working in the U.S. Federal Government space.

Experience with permitting, regulatory, land management, environmental, energy, or natural-resource datasets.

Pay and benefits

We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement.

If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!

Read the job posting on the Black Canyon Consulting site

Similar jobs at other companies

CH
Lead Data Engineer
Cohere HealthHyderabad, Telangana, India +1
- Senior
- Data engineering
Pay not posted
Posted Jun 16, 2026
Principal Data Engineer
Black Duck Software, Inc.Belfast, UK +1
- Principal
- Data engineering
Pay not posted
Posted Apr 7, 2026
Lead Data Engineer
Redhorse CorporationRemote
- Senior
- Data engineering
$170K–$190K / yr
Posted 20 days ago
Lead Data Engineer
ElasticUnited States
- Senior
- Data engineering
$113K–$215K / yr
Posted Oct 6, 2026
Principal Data Engineer
Constant ContactRemote, US or Remote, Ontario, Canada +2
- Principal
- Remote
- Data engineering
$138K–$172K / yr
Posted yesterday
Lead Data Engineer
OddballRemote
- Senior
- Data engineering
$110K–$145K / yr
Posted 5 days ago

Source: the Black Canyon Consulting Greenhouse job board. We last checked this job 7 h ago. We first saw it 5 days ago. A job on a job board does not always mean that the team interviews people now. We found this job board in a public list. We did not yet compare it with the careers page of the company.

Interview questions

No one has reported an interview at Black Canyon Consulting yet. Did you interview there? Add the questions.

BC

Black Canyon Consulting

Open tech jobs: 54
With posted pay: 24%
Median posted base: $115K–$160K

About Black Canyon Consulting: funding, hiring and pay