Senior Software Development Engineer - Big Data

Job description

**This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Software Development Engineer – Big Data based in the United States.** We are seeking a highly experienced Senior Software Development Engineer to design and build scalable data platforms that power analytics, machine learning, and AI-driven products. In this role, you will develop reliable, secure, and high-performance data pipelines and services that transform complex datasets into valuable business insights. You will also contribute to the development of a shared AI context layer and agent platform, enabling teams to build intelligent applications and deliver enhanced customer experiences. Working at the intersection of software engineering, distributed systems, and artificial intelligence, you will collaborate with cross-functional teams to turn business and technical requirements into production-ready solutions. You will help establish engineering best practices, improve platform reliability, and mentor other engineers. This fully remote opportunity offers the chance to solve complex data challenges at scale while shaping the future of AI-powered technology in a collaborative and innovative environment. ### Accountabilities - Design and develop scalable data infrastructure: Architect, implement, and maintain secure, observable, and reliable data pipelines and services that collect, process, and store large, diverse datasets for analytics, machine learning, and product applications. - Optimize data processing and storage: Develop efficient data models, storage strategies, and processing frameworks to support high-volume, real-time, and batch workloads while balancing performance, reliability, and infrastructure costs. - Lead technical excellence: Drive technical design reviews, code reviews, and engineering best practices, providing guidance and mentorship to engineers on data quality, system resilience, and operational excellence. - Collaborate across teams: Partner with product management, analytics, data science, and platform engineering teams to translate business objectives and modeling requirements into robust, production-grade data solutions and APIs. - Build AI and agentic platform capabilities: Develop and maintain shared context-layer services, agent tools, authentication mechanisms, and APIs that enable teams across the organization to build and deploy AI-powered applications at scale. - Ensure reliability and data quality: Implement automated testing, monitoring, alerting, and incident response processes to improve service availability, data integrity, and overall platform performance. - Drive technology innovation: Evaluate and introduce modern big data technologies, development tools, and architectural patterns to improve developer productivity, scalability, and end-to-end data lifecycle management. - Establish engineering standards: Document data flows, system interfaces, and operational runbooks while contributing to shared frameworks, reusable components, and best practices for big data engineering. - Support continuous improvement: Identify opportunities to strengthen platform architecture, simplify operations, and improve the efficiency and maintainability of distributed data systems. ## Requirements - Education: Bachelor's degree in Computer Science, Computer Engineering, or a related technical field, or an equivalent combination of education and professional experience. - Professional experience: At least 8 years of software engineering experience, with a strong focus on building, deploying, and operating data-intensive applications or distributed systems. - Big data engineering expertise: Demonstrated experience designing, implementing, and supporting production-grade data pipelines and platforms using modern big data, streaming, and/or <