Senior Data Engineer (Real-Time Data Processing & Streaming)

Job description

We are looking for an experienced Senior Data Engineer to join Endava and contribute to the design, development, and maintenance of scalable, high-performance data processing solutions. In this role, you will focus on real-time data processing, streaming technologies, and event-driven architectures , working with complex data pipelines that ingest, transform, enrich, and distribute data from connected physical devices to downstream applications and storage systems. You will collaborate closely with Tech Leads, software engineers, and cross-functional teams to build reliable, efficient, and maintainable data processing solutions in a cloud-native environment. Key Responsibilities: Design, develop, and maintain scalable real-time and near-real-time data processing pipelines. Implement streaming data processing solutions using technologies such as Apache Kafka, Apache Flink, and Apache Spark. Develop and maintain Java-based services and integration components for processing data received from physical devices. Implement complex data transformations, mappings, validations, and enrichments to support business and technical requirements. Build and maintain event-driven integrations using Kafka, Kafka Streams, Kafka Connect, and messaging protocols such as AMQP. Ensure reliable data delivery to downstream consumers, applications, and storage systems. Implement and maintain MongoDB collections, data structures, and related application-level integrations, following architectural guidelines. Develop, deploy, and maintain cloud-native services primarily on AWS, leveraging Kubernetes and managed cloud services. Contribute to the modernization and migration of legacy data processing components toward Kafka-based streaming architectures. Collaborate with technical leads and engineering teams to translate architectural designs into robust, production-ready implementations. Troubleshoot performance, reliability, and data consistency issues across distributed data processing systems. Contribute to code quality, automated testing, CI/CD practices, and continuous improvement of engineering processes. Required Skills and Experience: Strong professional experience in Data Engineering, with a focus on streaming data processing and distributed systems . Advanced programming skills in Java (11+) , including experience with the Spring framework. Hands-on experience with Apache Kafka , including Kafka Streams, Kafka Connect, and the development of Kafka-based integrations. Practical experience with streaming and distributed processing frameworks such as Apache Flink, Apache Spark, or Spark Streaming . Solid understanding of event-driven architectures , asynchronous communication patterns, and messaging systems, including AMQP. Experience designing and implementing data transformation, enrichment, and integration pipelines. Experience developing and integrating microservices and distributed applications. Hands-on experience with AWS cloud services , particularly S3, SQS, and EKS. Experience deploying and operating containerized applications using Docker and Kubernetes . Working knowledge of MongoDB , including collections, document structures, and Java-based data access. Familiarity with Redis and its use in distributed applications. Experience with Git-based development workflows and CI/CD pipelines, preferably using GitLab and GitLab CI/CD . Strong troubleshooting and problem-solving skills, particularly in distributed, high-throughput data processing environments. Ability to work collaboratively with technical leads and cross-functional engineering teams, while taking ownership of implementation and delivery. Nice to Have: Experience with Apache Pulsar or Apache Storm, particularly in legacy system maintenance or modernization initiatives. Familiarity with Microsoft Azure services and hybrid or multi-cloud environments. Experience processing telemetry, IoT, or other event data generated by physical devices. Knowledge of SMTP-based integrations. Experience migrating legacy messaging and stream processing solutions to modern Kafka-based architectures. Understanding of performance optimization, fault tolerance, monitoring, and observability in distributed streaming systems. Discover some of the global benefits that empower our people to become the best version of themselves: Finance: Competitive salary package, share plan, company performance bonuses, value-based recognition awards, referral bonus; Career Development: Career coaching, global career opportunities, non-linear career paths, internal development programmes for management and technical leadership; Learning Opportunities: Complex projects, rotations, internal tech communities, training, certifications, coaching, online learning platforms subscriptions, pass-it-on sessions, workshops, conferences; Work-Life Balance: Hybrid work and flexible working hours, employee assistance programme; Health: Global internal wellbeing programme, access to wellbeing apps; Community: Global internal tech communities, hobby clubs and interest groups, inclusion and diversity programmes, events and celebrations. At Endava, we’re committed to creating an open, inclusive, and respectful environment where everyone feels safe, valued, and empowered to be their best. We welcome applications from people of all backgrounds, experiences, and perspectives—because we know that inclusive teams help us deliver smarter, more innovative solutions for our customers. Hiring decisions are based on merit, skills, qualifications, and potential. If you need adjustments or support during the recruitment process, please let us know. At Endava, we’re committed to creating an open, inclusive, and respectful environment where everyone feels safe, valued, and empowered to be their best. We welcome applications from people of all backgrounds, experiences, and perspectives—because we know that inclusive teams help us deliver smarter, more innovative solutions for our customers. Hiring decisions are based on merit, skills, qualifications, and potential. If you need adjustments or support during the recruitment process, please let us know. Technology is our how. And people are our why. For over two decades, we have been harnessing technology to drive meaningful change. By combining world-class engineering, industry expertise and a people-centric mindset, we consult and partner with leading brands from various industries to create dynamic platforms and intelligent digital experiences that drive innovation and transform businesses. From prototype to real-world impact - be part of a global shift by doing work that matters.