Data Processing Engineer - I/O
DataPelago, Remote
The role
Data Processing Engineer – I/O Mountain View, CA / Hyderabad, IN / Remote
As a Data Processing Engineer – I/O, you will be a key individual contributor in advancing data read and write capabilities of DataPelago’s data processing engine. You will enhance functional breadth, performance, scale, and reliability of the DataPelago engine in reading and writing large scale data of various data types from diverse data sources and data sinks. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers.
What you will do
- Architect: Influence the architecture of how our data processing engine interfaces with data
sources and sinks, catalogs, data formats.
- Design: Lead design of functional and performance enhancements to adapters/connectors,
data representations, data filtering, caching and more in our data processing engine.
- Core Development: Individually design, implement, test, optimize, and maintain components of the data processing engine.
- Innovation and Differentiation: Analyze technology roadmap of existing and emerging data
formats and libraries, open table formats, catalog services, and more (e.g., Apache Arrow,
Apache Parquet, Apache Iceberg) and identify opportunities for our engine to enhance technology and product leadership.
- Collaboration: Partner effectively with engineering and product management in defining and
realizing the data I/O roadmap of our product..
- Continuous Improvement: Foster best practices in design and code reviews, testing, CI/CD,
and issue resolution to maintain highest product quality, security, efficiency, & productivity.
What you need
- Bachelor's degree in Computer Science or a related field with 7+ years of relevant experience OR a Master's degree in Computer Science or a related field with 5+ years of relevant
experience.
- 3+ years of deep technical experience in developing and optimizing data read and write interfaces for large-scale data processing, particularly related to Apache Parquet, Apache
ORC, Apache Iceberg, Apache Spark, and similar technologies.
- Demonstrated experience in instrumenting, analyzing, and optimizing the performance of