Staff Engineer - AI Platform

Job description

**About HighLevel:** HighLevel is an AI-powered business operating system that gives agencies, entrepreneurs and SMBs the infrastructure to build, automate and scale. Today, HighLevel supports SMBs across 150+ countries, fueling community-driven growth rooted in real customer outcomes. To date, businesses operating on HighLevel have generated over $7 billion in ecosystem value, demonstrating the impact of shared infrastructure at scale. By centralizing conversations, automation and intelligence into one system, we help businesses move faster, reduce complexity and execute efficiently. Behind the platform, HighLevel powers more than 4 billion API hits and 2.5 billion message events daily. With 250 terabytes of distributed data, 250+ microservices and over 1 million domain names supported, our architecture is built for performance, resilience and long-term scalability. **Our People** With over 2,000 team members across 10+ countries, HighLevel operates as a global, remote-first organization built for speed and ownership. We value initiative, clarity and execution, creating space for ambitious people to build systems that support millions of businesses worldwide. Here, innovation thrives, ideas are celebrated and people come first, no matter where they call home. **Our Impact** Every month, HighLevel enables more than 1.5 billion messages, 200 million leads and 20 million conversations for the more than 1 million businesses we support. Behind those numbers are real people building independence, expanding opportunity and creating measurable impact. We’re proud to be a part of that. Learn more about us on our [YouTube Channel](https://www.youtube.com/channel/UCXFiV4qDX5ipE-DQcsm1j4g) or [Blog Posts](https://www.gohighlevel.com/blog-on-the-move). ### About the Role: We're looking for a Staff Engineer (AI) to architect and build the agentic systems and AI foundations within our Platform Services team - the team responsible for the shared infrastructure, internal tooling, and core services that other product teams build on. This is a deeply hands-on individual contributor role: you will set technical direction, write and review code, and own the design of systems that let HighLevel engineering teams build, ship, and operate AI-driven capabilities reliably at enterprise scale. ### What You'll Build: - Production-grade agentic systems - multi-step, tool-using agents with orchestration, planning, memory and context management, and human-in-the-loop controls - for platform automation and internal engineering workflows. - A shared AI platform for product teams: reusable agent frameworks, tool and MCP integrations, an LLM gateway (routing, rate limiting, caching, fallbacks), and prompt/model versioning. Evaluation and observability for AI systems - eval harnesses, tracing of agent runs, and metrics for quality, latency, cost, and safety. - AI-driven engineering practices that improve developer velocity - agentic developer workflows, AI-assisted tooling, and automation of operational toil. - Cloud infrastructure for AI workloads across GCP/AWS, including provisioning, scaling, security, and cost governance for inference and token spend. ### What You'll Drive: - **Architecture & Scale:** Own and evolve the architecture of agentic systems and AI platform services, ensuring they are reliable, extensible, and performant under enterprise-scale load. - **AI-First Culture:** Set and drive an AI-first culture within the Platform Services team - making AI tools and agentic workflows the default way the team builds and operates. - **Reliability & Safety:** Build agents that fail safely - with guardrails, scoped permissions, sandboxing, defenses against prompt injection, audit trails, graceful degradation, and clear incident management practices. - **Evaluation & Data-Driven Decisions:** Establish rigorous evaluation practices - offline evals, regression suites, and production monitoring - so agent behavior is measurable and comparable across model and prompt changes. Use that data to prioritize, evaluate trade-offs, and report on agent quality, platform health, and cost. - **Cost & Efficiency:** Drive continuous optimization of inference, token, and infrastructure cost without compromising quality or reliability - with measurable, sustained impact. - **Technical Leadership & Mentorship:** Set technical standards, lead design reviews, mentor engineers, and build a culture of ownership and craftsmanship in AI engineering across the org. - **Cross-Functional Collaboration & Communication:** Partner closely with product engineering teams, SRE/infra, security, and leadership to align AI platform capabilities with organizational needs, and communicate technical direction, risks, and progress clearly across the org. ### What Success Looks Like: - Agentic systems run reliably in production at growing scale, with clear quality, latency, and safety targets that are tracked and met. - Agent changes ship with evals and observability in place, so regressions are caught before customers see them. - Product and platform teams adopt a shared AI platform and see it as a trusted, well-documented foundation to build on. - AI-related costs are optimized and predictable, with clear ownership of spend against usage. - The Platform Services team works AI-first, and engineering teams measurably gain velocity from AI-driven workflows and automation. - Engineers around you grow technically and recognize you as a go-to authority on agentic system design. ### Requirements: - 12+ years of overall software engineering experience, including significant time as a senior or staff-level individual contributor. - At least the last 1+ year spent architecting agentic systems in production - designing multi-step, tool-using agents, orchestration, memory and context management, and guardrails, not just integrating LLM APIs. A champion of AI agent orchestration at scale, running many agents and workflows reliably across teams. - AI-first mindset (must have) - actively using AI tools in your own engineering workflow, with the ability to set and drive an AI-first culture within the Platform Services team. - Expert-level proficiency in either Go, Python or Nodejs. - Strong hands-on orientation - comfortable writing and reviewing code, debugging production issues, and going deep on architecture. Candidates in a purely managerial (non-hands-on) role for the last 1+ year will not be considered a fit. - Strong working knowledge of LLM application fundamentals - tool/function calling, MCP, retrieval and context engineering, prompt and model management, and evaluation methodologies for non-deterministic systems. - Strong observability fundamentals, with deep experience in AI observability - tracing agent runs and monitoring quality, latency, cost, and safety in production - using tools like Prometheus, Grafana, ELK/OpenSearch, or equivalent. - Strong understanding of distributed systems architecture - consistency, availability, fault tolerance, scaling patterns, and trade-offs at scale. - Cost-aware by design - a track record of measurable cost optimization in an enterprise cloud environment, including managing LLM inference and token spend through model routing, caching, and budgets. - Practical experience with GCP and/or AWS service management - provisioning, scaling, IAM, networking, and cost governance. - Prior experience working as an individual contributor on platform teams for at least 3+ years, at enterprise scale. - Proven ability to lead technically without authority - writing design docs, driving cross-team alignment, and mentoring engineers. - A data-driven approach to decision-making, prioritization, and performance reporting. - Excellent verbal and written communication skills, with the ability to influence both technical and non-technical stakeholders. - Good to have: open-source contributions or published work on agentic systems.