Data Engineer

Job description

WHO ARE WE? MISSION Altimate AI, founded in 2022 in San Francisco, is revolutionizing enterprise data operations through the power of AI. Our mission is to alleviate the burden on overworked and understaffed enterprise data teams by providing innovative AI-driven solutions that automate and accelerate a wide range of data tasks. Our flagship product, DataPilot, offers advanced data automation capabilities, while our new DataMates technology brings the concept of agentic AI to data operations, acting as virtual teammates for data professionals. Our solutions seamlessly fit into existing tools like VSCode https://marketplace.visualstudio.com/items?itemName=innoverio.vscode-dbt-power-user, Git, and Slack, performing tasks ranging from data documentation to performance optimization. By leveraging a proprietary framework that combines multiple language models and a custom-built knowledge graph, we enable contextually aware AI agents that integrate seamlessly into existing workflows. Our solutions, including ambient AI for continuous monitoring and optimization, are designed to meet the growing demands of data operations, business intelligence, and analytics in an era of ever-increasing data volumes. Used by thousands of users across the world, and backed by prominent investors , we're positioned at the forefront of the AI-powered data engineering revolution. You can read more about us in a recently published venture beat article https://venturebeat.com/ai/how-agentic-ai-could-improve-enterprise-data-operations/. TEAM As a team, we are Silicon Valley veterans who previously created category-defining data and AI products loved by thousands of companies worldwide. We have experienced the journey of a small startup to IPO very closely. We have started on a similar journey again and are backed by prominent advisors and VC firms with multi-billion dollar portfolios. WHAT ARE WE LOOKING FOR? We're in search of a Data Engineer who brings a passion for innovation and problem-solving, not limited to traditional data engineering. This role is for those with deep expertise in building large scale data pipelines, SQL optimization, and cloud infrastructure, excited to design and build the backbone of our AI-powered data solutions. The ideal candidate will play a crucial role in enhancing our AI capabilities, developing high-performance data systems, and integrating advanced AI using the product into thousands of data teams' daily workflows. Qualifications: - 3-5 years of experience in data engineering, with a focus on building scalable data pipelines and systems - Strong proficiency in Python and SQL - Extensive experience with SQL query profiling, optimization, and performance tuning, preferably with Snowflake - Deep understanding of SQL Abstract Syntax Tree (AST) and experience working with SQL parsers (e.g., sqlglot) for generating column-level lineage and dynamic ETLs - Experience in building data pipelines using Airflow or dbt - [Optional] Solid understanding of cloud platforms, particularly AWS - [Optional] Familiarity with Kubernetes (K8s) for containerized deployments WHY YOU SHOULD JOIN ALTIMATE AI Projects You'll Work On: - Building PB Scale Data Pipelines: Building highly performant large scale data infrastructure that can scale to 100K+ jobs and handle PB scale data per day. - Cloud-Native Data Infrastructure: Design and implement robust, scalable data infrastructure on AWS, utilizing Kubernetes and Airflow for efficient resource management and deployment. - Intelligent SQL Ecosystem: Design and develop a comprehensive SQL intelligence system encompassing query optimization, dynamic pipeline generation, and data lineage tracking. Leverage your expertise in SQL query profiling, AST analysis, and parsing to create a sophisticated engine focused on query performance improvements, building adaptive data pipelines, and implementing granular column-level lineage. Impact: - Innovation at the Forefront: Push the boundaries of data engineering by combining traditional techniques with cutting-edge AI technologies. - High Visibility: Directly affect the productivity and capabilities of global data teams, as your contributions will be crucial to the daily operations of thousands of users spread across 100s of countries. - Open Source Contribution: As part of our commitment to the developer community, you will contribute to our open-source initiatives https://github.com/AltimateAI/vscode-dbt-power-user, gaining recognition in the tech community. - Career Growth: This role is a launchpad into the rapidly advancing field of AI-powered data engineering, offering exposure to state-of-the-art technologies and generative AI applications. HOW TO APPLY If this sounds like the perfect role for you, we can't wait to meet you! Apply now and let's build something amazing together.