Technical Skills Required
Programming Languages: Proficiency in Python, SQL, and PySpark.
Data Warehousing: Experience with Snowflake, NOSQL and Neo4j.
Data Pipelines: Proficiency with Apache Airflow.
Cloud Platforms: Familiarity with AWS (S3, RDS, Lambda, AWS batch, SageMaker processing Job, CloudFormation, etc.) or GCP (Vertex AI RAG, Data pipeline, Bigquery, GKE)
Operating Systems: Experience with Linux.
Batch/Realtime Pipelines: Experience in building and deploying various pipelines.
Version Control: Experience with GitHub.
Development Tools: Proficiency with VS Code.
Engineering Practices: Skills in testing, deployment automation, DevOps/SysOps.
Communication: Strong presentation and communication skills.
Collaboration: Experience working with onshore/offshore teams.
GenAI Engineer Specific Skills
- Industry Experience: 8+ years of experience in data engineering, platform engineering, or related fields, with deep expertise in designing and building distributed data systems and large-scale data warehouses.
- Data Platforms: Proven track record of architecting data platforms capable of processing petabytes of data and supporting real-time and batch ingestion processes.
- Data Pipelines: Strong experience in building robust data pipelines for document ingestion, indexing, and retrieval to support scalable RAG solutions. Proficiency in information retrieval systems and vector search technologies (e.g., FAISS, Pinecone, Elasticsearch, Milvus).
- Graph Algorithms: Experience with graphs/graph algorithms, LLMs, optimization algorithms, relational databases, and diverse data formats.
- Data Infrastructure: Proficient in infrastructure and architecture for optimal extraction, transformation, and loading of data from various data sources.
- Data Curation: Hands-on experience in curating and collecting data from a variety of traditional and non-traditional sources.
- Ontologies: Experience in building ontologies in the knowledge retrieval space, schema-level constructs (including higher-level classes, punning, property inheritance), and Open Cypher.
- Integration: Experience in integrating external databases, APIs, and knowledge graphs into RAG systems to improve contextualization and response generation.
- Experimentation: Conduct experiments to evaluate the effectiveness of RAG workflows, analyze results, and iterate to achieve optimal performance.