Sr. Data Engineer-Hadoop,Hive,Spark,Trino, Star Rocks, Clickhouse,Data engineering-3+yrs AI exposure

Sr. Data Engineers are responsible for collaborating with stakeholders, including product teams, business partners, and cybersecurity, to identify and clarify requirements for data-driven features and products. They independently understand the data ecosystem, security, privacy, and retention requirements necessary to support business and product features. Sr. Data Engineers create and document technical designs for components, identify systems, architectures, tools, and storage needs, and present technical solutions to peers.

They stay current with market trends and industry dynamics, applying this knowledge to drive innovation and ensure the safety and security of product components.

All roles require digital fluency, including the ability to work with emerging technologies such as Generative AI tools (e.g. ChatGPT, Microsoft Copilot) to support everyday work.

• Collaborate proactively with stakeholders to clarify requirements and identify key product stakeholders.
• Translate functional requirements into architecture designs for one or more components, leveraging existing architecture design patterns, and communicate these to Data Engineers.
• Implement extensible, maintainable, and reusable code using appropriate coding patterns, guidelines, and best practices; participate in code reviews to ensure standards are followed.
• Provide end-to-end ownership of data products or data pipelines, including mitigation of technical debt and documentation of errors or unexpected bugs.
• Identify and conduct required testing for moderately complex use cases, including unit testing, coding standards, and security scans.
• Build tools and standard automation processes to assist in transforming, managing, accessing, deploying, and monitoring data processes in batches and in real time.
• Write queries to extract and compile raw data across end-to-end pipelines, and implement orchestration techniques to automate data extraction logic.
• Ensure adherence to data management principles, governance, and tools to maintain data quality across multiple features.
• Develop and update troubleshooting guidance and procedures for reviewing, addressing, and fixing advanced problems flagged by testing.
• Manage software upgrades and server patches for security remediation where applicable.

Visa requires at least 3 days in office, expectations of these days will be confirmed by your Hiring Manager.

• 3+ years of relevant work experience and a Bachelor's degree, OR 5+ years of relevant work experience.
• Experience in solving data problems using data technologies
• Experience with Hadoop, Hive, HBase, Spark, Trino, or comparable distributed data systems.
• Experience in designing and implementing data pipelines and large-scale data processing jobs.
• Experience in writing, modifying, and reviewing high-quality, testable, and efficient code.
• Data pipelines, ETL/ELT, data-quality frameworks, Java or Python, SQL, Git, and production troubleshooting.
• Experience in developing unit, integration, and end-to-end tests with a focus on automation.
• Experience in building and maintaining CI/CD pipelines and deploying applications using containerization and orchestration tools.
• Experience in monitoring applications in production and troubleshooting issues using observability tools.
• Experience in integrating AI or ML services into applications, where applicable.
• Experience in collecting and analyzing metrics to guide optimizations and improvements.
• Experience in partnering with stakeholders to clarify requirements and ensure deliverables meet business needs.
• Experience in applying secure coding practices and complying with regulatory standards.

• 3 or more years of work experience with a bachelor’s degree or more than 2 years of work experience with an Advanced Degree (e.g. Masters, MBA, JD, MD).
• Experience in solving data problems using data technologies
• Experience with Hadoop, Hive, HBase, Spark, Trino, or comparable distributed data systems.
• Experience in contributing in Open-source components.
• Use/Build AI agents, and agentic patterns to improve development, testing, operations, troubleshooting, automation, onboarding, and documentation.
• Data pipelines, ETL/ELT, data-quality frameworks, Java or Python, SQL, Git, and production troubleshooting.
• Experience in enterprise-class systems engineered for high availability, low latency, scalability, and supportability. Architecture depth across storage/compute separation, query and table formats, streaming, metadata, governance, and interoperability.
• Experience in building and pushing code into production.
• Experience in implementing and supporting real-time systems.

Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be paid.

At Visa, you'll have the opportunity to create impact at scale — tackling meaningful challenges, growing your skills and seeing your contributions impact lives around the world.

Join Visa and do work that matters – to you, to your community, and to the world. Progress starts with you.

Job Description