AI Data Engineer

Job description

Role Overview: We are looking for an exceptional AI/GenAI Data Engineer to join our AI Platform team — one of the most technically ambitious initiatives at LSEG. This is not a role to maintain legacy pipelines. You will design and build the systems that make AI possible at HR-market scale. You will operate at the intersection of data engineering, machine learning infrastructure, and large language model applications. Your work will directly impact products used by the largest financial institutions in the world. Key responsibilities: Design and operate high-throughput, low-latency data pipelines Build and maintain the feature stores, vector databases, and data layers that power production ML models and LLM-based applications across LSEG's product suite Architect SQL and NoSQL data systems to serve analytical and operational workloads with strict SLAs, and build real-time event streaming pipelines using Apache Kafka Implement Redis-based caching and pub/sub systems to support sub-millisecond data access for trading and analytics applications Build and deploy machine learning pipelines — from data preprocessing and feature engineering through model training, evaluation, and serving at scale Integrate and fine-tune large language models for different tasks Design and own CI/CD pipelines for data and ML systems, covering automated testing, model validation, and deployment automation Partner with Applied Research, Product Engineering, and Platform teams to translate AI research into reliable, production-grade systems Must Have Skills: LLM & Generative AI: Hands-on experience building LLM-powered applications using APIs from OpenAI, Anthropic, or similar providers. Experience with RAG (Retrieval-Augmented Generation) pipelines, embedding models, vector stores, and frameworks such as LangChain or LlamaIndex. Familiarity with prompt engineering, LLM evaluation techniques, and fine-tuning is strongly preferred. MCP (Model Context Protocol): Understanding of MCP primitives for tool invocation, context injection, and secure LLM-to-system interaction. Machine Learning: Solid foundations in supervised and unsupervised machine learning using Scikit-learn, XGBoost, and PyTorch. You should have end-to-end ownership experience — from feature engineering and model training through evaluation, deployment, and production monitoring. Familiarity with MLflow or similar experiment tracking tools and an understanding of model drift detection are expected. Programming — Python & Node.js: Strong Python is essential. You should be comfortable with async programming, writing production-grade APIs with FastAPI, data processing with Pandas and PySpark, and building clean, testable, well-documented code as a default. Node.js with TypeScript experience for event-driven backend services is a plus. SQL — ClickHouse, PostgreSQL & Snowflake: Deep hands-on experience with at least two of ClickHouse, PostgreSQL, and Snowflake. You should be confident with complex query writing and optimisation, indexing strategies, partitioning, window functions, and understanding the trade-offs between OLTP and OLAP workloads. Experience with ClickHouse for high-performance analytical queries on time-series financial data is particularly valued. NoSQL — Elasticsearch & Redis: Production experience with Elasticsearch for full-text search, aggregations, and log analytics — including index lifecycle management, mapping optimisation, and cluster configuration. Redis experience covering caching patterns, pub/sub messaging, Redis Streams, and session management in high-throughput environments. Streaming — Apache Kafka: Hands-on experience with Apache Kafka for building real-time event streaming pipelines. You should be comfortable with producer/consumer patterns, topic partitioning, consumer group management, and Kafka Connect for data integration. Experience with Kafka Streams or processing financial market events (order flow, price ticks, corporate actions) in a low-latency production environment is a strong advantage. CI/CD: Solid understanding of CI/CD principles with hands-on experience using GitHub Actions or Jenkins. You should be comfortable with Docker, Kubernetes, and writing Infrastructure as Code using Terraform or Helm. The expectation is that you treat deployment pipelines as production code — automated testing, rollback strategies, and environment parity are non-negotiable. Cloud — AWS: Primary cloud experience on AWS. You should have working knowledge of core data and compute services — S3, RDS, Redshift, Lambda, ECS/EKS, and SageMaker. Experience with cost optimisation and multi-region architecture design is a plus. Qualifications: 7+ years of hands-on data engineering or ML engineering experience in production environments Strong system design instincts — you consider failure modes, backpressure, and graceful degradation as first-class concerns You default to observability — metrics, tracing, and alerts are built in before systems go live, not retrofitted after incidents You write the runbook. On-call ownership is something you embrace, not avoid Able to communicate technical trade-offs clearly to product managers, data scientists, and senior stakeholders Experience mentoring junior engineers and conducting meaningful technical code reviews Experience with HR data (time-series, tick data, reference data) is a significant advantage Nice to have: (Review Certifications) Cloud (AWS/Azure/GCP Data Engineer) Databricks or Spark-based certification ML / AI certification We're proud to have been recognised as a Great Place to Work® in India ‘25. Career Stage: Senior Associate London Stock Exchange Group (LSEG) Information: Join us and be part of a team that values innovation, quality, and continuous improvement. If you're ready to take your career to the next level and make a significant impact, we'd love to hear from you. LSEG is a leading global financial markets infrastructure and data provider. Our purpose is driving financial stability, empowering economies and enabling customers to create sustainable growth. Our purpose is the foundation on which our culture is built. Our values of Integrity, Partnership , Excellence and Change underpin our purpose and set the standard for everything we do, every day. They go to the heart of who we are and guide our decision making and everyday actions. Working with us means that you will be part of a dynamic organisation of 25,000 people across 65 countries. However, we will value your individuality and enable you to bring your true self to work so you can help enrich our diverse workforce. We are proud to be an equal opportunities employer. This means that we do not discriminate on the basis of anyone’s race, religion, colour, national origin, gender, sexual orientation, gender identity, gender expression, age, marital status, veteran status, pregnancy or disability, or any other basis protected under applicable law. Conforming with applicable law, we can reasonably accommodate applicants' and employees' religious practices and beliefs, as well as mental health or physical disability needs. You will be part of a collaborative and creative culture where we encourage new ideas. We are committed to sustainability across our global business and we are proud to partner with our customers to help them meet their sustainability objectives. Our charity, the LSEG Foundation provides charitable grants to community groups that help people access economic opportunities and build a secure future with financial independence. Colleagues can get involved through fundraising and volunteering. LSEG offers a range of tailored benefits and support, including healthcare, retirement planning, paid volunteering days and wellbeing initiatives. Please take a moment to read this privacy notice carefully, as it describes what personal information London Stock Exchange Group (LSEG) (we) may hold about you, what it’s used for, and how it’s obtained, your rights and how to contact us as a data subject . If you are submitting as a Recruitment Agency Partner, it is essential and your responsibility to ensure that candidates applying to LSEG are aware of this privacy notice.