Software Engineer, Ray Data

Ray aims to provide a universal API for building distributed applications (e.g. a machine learning pipeline of feature engineering, model training, and evaluation). Data is usually a core element connecting these different stages, and therefore plays a critical role in Ray’s usability, performance, and stability. We are looking for strong engineers to build, optimize, and scale Ray’s Datasets https://docs.ray.io/en/latest/data/dataset.html library and data processing capabilities in general.

The Ray Data team currently develops and maintains the Ray Datasets https://docs.ray.io/en/latest/data/dataset.html library, which is already powering critical production use cases (e.g. large scale data compaction at Amazon https://www.anyscale.com/events/2022/01/20/ray-meetup-is-back-in-the-new-year-2022, and ML pipeline at Alibaba ). Ray Datasets is a Python library built on top of Apache Arrow and Ray Core (Ray’s C++ backend), and the Ray Data team interacts closely with Ray Core components including the scheduler and the memory & I/O subsystems.