Senior Software Engineer - Fleet Engineering

Job description

Lambda, The Superintelligence Cloud, is a leader in AI cloud infrastructure serving tens of thousands of customers. Our customers range from AI researchers to enterprises and hyperscalers. Lambda's mission is to make compute as ubiquitous as electricity and give everyone the power of superintelligence. One person, one GPU. If you'd like to build the world's best AI cloud, join us. *Note: This position requires presence in our San Francisco, San Jose, or Bellevue office location 4 days per week; Lambda’s designated work from home day is currently Tuesday. Fleet Engineering at Lambda is responsible for logically deploying, provisioning, and maintaining our rapidly growing fleet, spanning GPU, CPU, and storage hosts along with the InfiniBand and networking fabric that ties them together. As a Senior Software Engineer in Fleet Engineering, you will build and improve the systems that take our clusters from hardware receiving through provisioning, configuration, and day-to-day operation in production. You will work on automation and tooling that keep the fleet consistent, healthy, and easy to operate as it grows. The systems you build directly determine how quickly and reliably new hardware turns into usable customer capacity. What You’ll Do Fleet Engineering spans several teams. Depending on the team you join, your day-to-day will involve some combination of the following: - Develop and Maintain Production Systems: Design, implement, and improve the software that powers fleet lifecycle management, machine configuration, and cluster state at scale. - Automate Provisioning and Deployment: Build and enhance automation that takes clusters from logical design and racking through OS provisioning, configuration, validation, and customer hand-off. - Support New Hardware and Site Bring-Up: Enable bring-up, validation, and production readiness for new server, accelerator, and network platforms, as well as new datacenter sites. - Improve Machine Lifecycle Workflows: Refine bare metal provisioning, firmware and DPU updates, imaging, and system health monitoring across the fleet. - Keep Fleet State Consistent and Healthy: Build systems that reconcile intended against actual configuration, catch drift before it causes deployment failures, and maintain production SLAs. - Debug Hardware and Firmware Issues: Investigate failures across BIOS, BMC, firmware, DPUs, networking, storage, and boot flows. - Collaborate Across Teams: Work closely with datacenter and deployment operations, networking, architecture, security, and product engineering teams to build scalable, maintainable solutions. You - Have 5+ years of engineering experience - Are fluent in Python, Go, or similar, and comfortable with APIs, distributed systems, and automation pipelines - Work confidently in Linux environments and can debug across the OS, hardware, and networking layers - Are excited about working at the intersection of hardware, software, and physical datacenter builds - Have owned production systems with real SLAs - Can lead technical design on medium-to-large features: take an ambiguous problem, write the doc, drive alignment across teams, and ship - Leave systems, and the teammates around you, better than you found them Nice to Have - Experience in AI or ML infrastructure, or other hyperscale compute environments - Hands-on experience with bare metal provisioning and lifecycle management, including technologies such as PXE, Redfish, IPMI, BMC, DHCP, and DNS - Familiarity with datacenter physical infrastructure, including racks, switches, InfiniBand fabric, and power domains - Experience diagnosing issues involving drivers, firmware, and hardware compatibility across GPU servers - Experience with DPUs or programmable network accelerators - Experience with network source-of-truth systems (NetBox or similar) or DCIM tooling - Experience building Linux distributions or managing OS customization and imaging - Familiarity with Ansible or similar configuration management tooling - Exposure to Kubernetes and container orchestration concepts - Experience incorporating AI-assisted development tools into engineering workflows, including code generation, debugging, test development, and documentation If you don’t meet all of these requirements but believe you may be a good fit, please still apply and provide a cover letter that helps us understand your experience and readiness for this role. Salary Range Information The annual salary range for this position has been set based on market data and other factors. However, a salary higher or lower than this range may be appropriate for a candidate whose qualifications differ meaningfully from those listed in the job description. About Lambda - Founded in 2012, with 500+ employees, and growing fast - Our investors notably include TWG Global, US Innovative Technology Fund (USIT), Andra Capital, SGW, Andrej Karpathy, ARK Invest, Fincadia Advisors, G Squared, In-Q-Tel (IQT), KHK & Partners, NVIDIA, Pegatron, Supermicro, Wistron, Wiwynn, Gradient Ventures, Mercato Partners, SVB, 1517, and Crescent Cove - We have research papers accepted at top machine learning and graphics conferences, including NeurIPS, ICCV, SIGGRAPH, and TOG - Our values are publicly available: https://lambda.ai/careers - We offer generous cash & equity compensation - Health, dental, and vision coverage for you and your dependents - Wellness and commuter stipends for select roles - 401k Plan with 2% company match (USA employees) - Flexible paid time off plan that we all actually use Equal Opportunity Employer Lambda is an Equal Opportunity employer. Applicants are considered without regard to race, color, religion, creed, national origin, age, sex, gender, marital status, sexual orientation and identity, genetic information, veteran status, citizenship, or any other factors prohibited by local, state, or federal law.