Sr. Data Platform Engineer

Job description

**This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Sr. Data Platform Engineer based in the United States.** We are seeking an experienced Data Platform Engineer to build and scale robust data infrastructure that powers advanced analytics, AI-driven intelligence, and critical business decisions. In this senior role, you will design reliable batch and streaming pipelines, process diverse structured and unstructured datasets, and develop maintainable software systems that support growing data demands. You will have meaningful ownership over technical decisions, architecture, and engineering practices while contributing to the evolution of the broader data platform. The role combines strong Python development, modern data engineering, distributed processing, and applied machine learning techniques. You will also leverage AI-assisted development tools to accelerate coding, refactoring, prototyping, and documentation. This is an opportunity to make a significant technical impact in a high-growth technology environment while helping shape the future of enterprise intelligence. ### Accountabilities - Build and optimize data pipelines: Design, implement, and maintain scalable batch and streaming pipelines that efficiently process large volumes of diverse data. - Develop data processing systems: Build extensible data infrastructure that supports the addition, maintenance, and scaling of pipelines as business and product needs evolve. - Process and model complex datasets: Extract, analyze, transform, and model structured and unstructured data to generate actionable insights and support intelligent applications. - Ensure production reliability: Develop and maintain data pipelines with strong reliability, scalability, fault tolerance, and observability, ensuring consistent performance in production environments. - Apply strong software engineering practices: Design testable, maintainable, and reusable software components while following sound development lifecycle practices and engineering standards. - Contribute to data strategy: Help shape the data platform's technical direction, architecture, and long-term vision, making informed decisions about infrastructure and implementation approaches. - Apply machine learning techniques: Use methods such as anomaly detection, clustering, regression, classification, and summarization to extract meaningful value from complex datasets. - Leverage AI-assisted development: Use tools such as Cursor and other AI-powered coding assistants to accelerate development, scaffold components and APIs, refactor legacy code, improve documentation, and prototype solutions. - Improve engineering tooling and practices: Stay informed about emerging data engineering technologies and incorporate relevant tools, frameworks, and best practices into the platform. - Collaborate and take ownership: Work closely with product and engineering colleagues, communicate technical ideas clearly, take initiative in solving complex problems, and contribute to continuous improvement across teams. ## Requirements - Professional experience: At least 4 years of experience in Data Engineering and at least 4 years of hands-on Python development. - Production data engineering: Demonstrated experience building, maintaining, and operating production-grade data pipelines with a focus on reliability, scalability, fault tolerance, and observability. - Core data engineering knowledge: Solid understanding of ETL/ELT processes, batch and streaming architectures, data warehouses, data lakes, and distributed data processing. - Python and data processing: Strong Python programming skills and familiarity with libraries and frameworks such as Pandas and PySpark. - Software engineering fundamentals: Strong knowledge of the software development lifecycle (SDLC), testing, maintainability, and other established software engineering practices. - Infrastructure as code: Familiarity with infrastructure-as-code principles and tools for managing and deploying infrastructure. - Technical stack familiarity: Experience with some of the following technologies is expected: - Languages and data tools: Python, PySpark, Pandas, SQL, and PostgreSQL. - Data orchestration and search: Airflow and Elasticsearch. - Cloud and data platforms: AWS services such as S3, Batch, Athena, RDS, DynamoDB, Glue, ECS, and Amazon Neptune; Databricks. - Development and infrastructure tools: Docker, CircleCI, GitHub, and Terraform. - AI-assisted coding expertise: Familiarity with AI-powered coding tools such as Cursor, GitHub Copilot, or Codex, with practical experience using AI-enabled development environments. - Effective AI development workflows: Demonstrated ability to use AI coding assistants to scaffold components and APIs, refactor existing code efficiently, reduce context switching, accelerate documentation, and prototype with rapid feedback. - Analytical problem-solving: Strong critical thinking and problem-solving abilities, with the capacity to investigate complex technical challenges and develop practical solutions. - Communication and collaboration: Ability to explain technical concepts clearly, exchange ideas effectively, and collaborate with colleagues across teams. - Ownership and initiative: A self-directed approach to work, a willingness to take responsibility for technical outcomes, and the curiosity to explore new technologies and continuously improve. - Learning mindset: Genuine interest in emerging data engineering practices, AI-assisted development, and evolving technologies that can improve platform capabilities. ## Benefits - Competitive salary: Starting at $170,000 USD per year, with the potential for a higher offer based on expertise, seniority, location, cost of living, taxes, and market experience. - Equity participation: Equity in a well-funded, fast-growing technology company. - Flexible time off: Unlimited vacation time, allowing employees to take time off as needed. - Healthcare coverage: 99% of health, dental, and vision insurance premiums covered for employees and their dependents. - Retirement savings: Access to a 401(k) plan through Empower, with investment options. - Remote work: Fully remote position within the United States. - Growth and technical ownership: The opportunity to influence technical direction, work with modern data and AI technologies, and contribute to the development of a scalable data platform. - Inclusive workplace: A commitment to equal opportunity, diversity, equity, and belonging, with encouragement for candidates to apply even if they do not meet every listed qualification.