Data Engineer with Kafka - Pune

Capco, India - Pune

What you will do

Design, develop, and maintain scalable, reliable, and high-performance data pipelines for batch and real-time data processing.
Implement and optimize Kafka-based event streaming solutions, ensuring reliable data ingestion, processing, and integration.
Develop production-grade data processing applications using Apache Spark and Scala/PySpark.
Optimize Spark workloads through effective partitioning, caching, skew handling, join optimization, and performance tuning.

What you need

Required Technical Skills

Apache Kafka / Event Streaming
Strong understanding of Kafka architecture, event streaming concepts, and implementation.
Hands-on experience developing and supporting Kafka-based data ingestion and streaming pipelines.
Knowledge of Kafka producers, consumers, topics, partitions, consumer groups, and offset management.
Understanding of fault tolerance, message delivery guarantees, and streaming performance optimization.
Apache Spark
Strong hands-on experience with Apache Spark for large-scale data processing.
Expertise in Spark performance tuning, including joins, partitioning, caching, data skew handling, and troubleshooting.
Experience developing and optimizing production-grade Spark applications.
Scala / PySpark
Strong proficiency in at least one of Scala or PySpark, with working knowledge of the other.
Ability to write efficient, maintainable, and production-ready code.
SQL
Advanced SQL skills, including complex joins, window functions, query optimization, and data modeling.
Experience working with large datasets and optimizing SQL performance.
Data Orchestration – NiFi / Airflow / ADF
Strong experience in at least one orchestration tool, with working knowledge of the others.
Ability to independently design, develop, monitor, and troubleshoot data pipelines.
Cloud Storage – S3 / ADLS / Data Lake
Experience working with cloud-based data lakes and storage solutions.
Understanding of file formats, partitioning strategies, storage design, data lifecycle management, and performance considerations.
Python
Good Python scripting and automation skills.
Experience building utilities, ETL components, and automated testing frameworks.
Production Support & Troubleshooting
Experience handling production incidents, monitoring data pipelines, and performing root cause analysis.
Ability to implement resiliency improvements and ensure pipeline stability.

Qualifications

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.
Minimum 5+ years of relevant experience in data engineering, big data technologies, and data pipeline development.
Hands-on experience with Kafka-based streaming solutions and distributed data processing frameworks.

Data Engineer with Kafka - Pune

What you will do

Data Engineer with Kafka - Pune

What you will do

What you need

Required Technical Skills

Qualifications

Nice to have

Make an impact

#beyourselfatwork

Career advancement

Additional Expectations

About Capco

Similar jobs at other companies

Senior Data Engineer AWS

Kafka Engineer

Data Engineer

Data Engineer (in person)

Staff Engineer ( Full Stack )

Data Engineering Team Lead

Capco