Senior Data Engineer

Overview

Our client is looking to recruit a Senior Data Engineer.

The ideal candidate will have a passion for working in white space, creating impact from the ground up, and transforming data driven outcomes into fully scalable and optimized solutions for always-on production environments. You will have extensive experience in data engineering, particularly within the cloud ecosystem (we are on AWS), and will be proficient in Python, SQL, and Linux. You will be responsible for designing, developing, and maintaining our data infrastructure, ensuring data quality, and supporting data-driven decision-making across the organization.

• Design, develop, and maintain scalable data pipelines and ETL processes using open source and AWS services.
• Orchestrate data workflows and pipelines.
• Optimize and manage databases and data warehouses.
• Implement data transformation workflows.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Ensure data quality, integrity, and security across all data platforms.
• Develop and maintain documentation for data engineering processes and systems.
• Monitor and troubleshoot data pipelines and workflows to ensure reliability and performance.
• Implement and manage CI/CD pipelines to automate the deployment and testing of data engineering solutions.
• Mentor and provide guidance to other team members

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering or a related role.
• Strong proficiency in Python and SQL.
• Experience with NoSQL databases, Amazon Redshift and DynamoDB
• Hands on experience with cloud-based architectures and software stacks
• Deep understanding of large always-on enterprise system architecture
• Proficiency in Linux server environments.
• Experience with containerization technologies such as Docker.
• Hands-on experience with data transformation, such as DBT.
• Experience with Prefect or similar orchestration tools.
• Strong understanding of ETL/ELT processes, data warehousing concepts and best practices.
• Experience with CI/CD pipelines and version control systems like Git.
• Excellent problem-solving skills and attention to detail.
• Strong experience modelling machine behavioural data
• Understanding of data governance and security
• Understanding of infrastructure-as-code
• Experience with monitoring and alerting systems
• Strong understanding of big data processing techniques
• Proficient with machine learning techniques
• Experience with a wide variety of statistical languages and data mining tools