Senior DevOps Engineer, AI Platform

Job description

## **Why this role exists** FloQast's AI products have outgrown the infrastructure patterns the rest of the platform runs on. Transform, AI Matching, and AutoBuilder are customer-facing products carrying real accounting workloads, and they behave nothing like a CRUD service. They call foundation models in multiple regions, execute generated code in sandboxes, spend money per token rather than per request, and fail in ways a 500-rate dashboard never catches. Today, DevOps engineers carry this work alongside the wider fleet. We are making it someone's whole job. You will embed with the Transform and Close AI pods and own the AI runtime the way our other embedded DevOps engineers own their business unit's platform. This is a DevOps role with an AI infrastructure specialization, not a research or modeling role. You will not train models or tune prompts for accuracy. You will make the systems that serve them **fast, observable, multi-region, cost-bounded, and auditable**. ## **The products you'll support** **Transform.** FloQast's data transformation and analytics product: a monorepo of containerized services on AWS, including an agentic LLM thread runtime, a natural-language-to-SQL service, and queue-driven workflow workers. You'll own the foundation-model runtime across our US, EU, and AU regions, sandbox isolation for AI-executed code, autoscaling for the worker fleet, and cost attribution for model spend. **AI Matching.** The automated reconciliation matching stack in our Close product line: LLM-backed match scoring, a matching copilot service family, and a regression harness that guards matching quality. You'll own throughput and unit economics at close-cycle peak, the sandbox for AI-generated code, and keeping the eval harness running in CI so a model or prompt change cannot ship blind. **AutoBuilder.** The Transform capability that generates transformation workflows and scripts for users instead of making them hand-build each one. You'll own generation-queue health and backpressure, triage that distinguishes a model failure from an infrastructure failure, scale-to-demand behavior for spiky load, and the latency budget for a user waiting on a generated artifact. ### What You'll Do: **AI runtime infrastructure.** Own our AWS Bedrock and Bedrock AgentCore footprint across US, EU, and AU: model access, provisioned versus on-demand throughput, cross-region inference, quotas and throttles, and region-appropriate model availability. Data residency drives the multi-region design as much as latency does. Define the standard way product teams call a model, retry it, budget it, log it, and trace it, so they stop hand-rolling those primitives. **Sandboxed execution.** Operate the execution environments for AI-generated code, with session lifecycle limits, network controls, and least-privilege IAM. **Infrastructure as code.** Write and review Terraform across a multi-account, multi-region AWS estate. Every AI resource lands as code. Nothing is hand-made in a console. **Observability for AI workloads.** Extend our Grafana platform with the signals AI systems need: token consumption, per-model and per-region latency distributions, throttle and retry rates, tool-call failure taxonomy, sandbox session outcomes, generation success rate, and end-to-end agent traces. Define SLOs against critical user journeys, because an AI SLO that only measures HTTP health measures nothing. **Cost engineering.** Treat AI spend as a first-class cost line: model inference, serving capacity, sandbox compute, and the telemetry volume these systems generate. Tag everything so per-business-unit, and ideally per-tenant, attribution shows up in our FinOps tooling. **CI/CD and delivery.** Build GitHub Actions pipelines for Node/TypeScript and Python services in NX monorepos. Make model and prompt changes releasable like code: versioned, gated on eval suites, feature-flagged, reversible. A prompt change that silently degrades matching accuracy should be caught by a pipeline, not by a customer. **Reliability and on-call.** Join the DevOps on-call rotation, with runbooks you help write for AI-specific failure modes: region-wide model throttling, sandbox exhaustion, silent degradation without errors, poison messages in generation queues. Plan capacity against the accounting calendar; month-end close is the load test. **Security, compliance, and tenancy.** Hold the AI stack to our SOC 2 and ISO 27001/42001 evidence bar: audit logging, encryption, least-privilege access, patching, and asset inventory, for model endpoints and sandboxes too. Enforce tenant isolation on every AI path, including prompts, retrieved context, generated code, and logs. ### What You'll Bring: - 5+ years in DevOps, SRE, platform, or infrastructure engineering, including production systems you were on call for. - Deep, hands-on AWS: ECS/Fargate, Lambda, SQS, S3, IAM, VPC and networking, ALB/NLB. - Terraform at production scale: modules, state management, multi-region, multi-account. - CI/CD and container ownership: GitHub Actions preferred, Docker, image supply chain, scaling policies. - Production AI infrastructure: you have operated at least one LLM-backed or ML-serving workload in production and can speak concretely about tokens, latency, throttling, and cost. Managed foundation-model APIs (Bedrock, Vertex, Azure OpenAI, Anthropic), serving platforms (SageMaker, KServe, Ray Serve, vLLM, Triton, TrueFoundry), agent or tool-execution runtimes, and eval harnesses in CI all count. - Observability practice: Grafana, Prometheus, or OpenTelemetry, distributed tracing, and the judgment to define SLOs that measure user experience rather than server aliveness. - Working fluency in Python or TypeScript/Node.js, at the level of reading a service, debugging it, and sending a real PR, plus the ability to read the other. ## **Nice to have** - Multi-region infrastructure under data-residency constraints (US/EU/AU). The product requires this; prior experience with it moves you to the front of the line. - AI-specific cost and performance work: token accounting, prompt caching, batching, model routing, right-sizing serving capacity. - Sandboxed execution of untrusted or generated code. - Terraform orchestration layers (Atmos or comparable) and monorepo build systems (NX, Turborepo, Bazel). - Progressive delivery with feature flags (Harness or similar). - FinOps tooling (CloudZero or similar) and per-tenant cost attribution. - Data infrastructure exposure: MongoDB, PostgreSQL, Snowflake, EMR/Spark. - Audit evidence you produced for SOC 2 / ISO 27001, or a cloud-cost reduction you delivered and can explain, including what you traded to get it. - Prior work in a regulated or audited SaaS domain: fintech, accounting, healthcare. ## **Not required** - Model training, fine-tuning, or research publications. - An accounting background. We'll teach you the domain, and you'll need enough of it to know what “wrong” looks like. - A PhD or formal ML credential. Production judgment beats coursework here. ### What to Expect: **The AI stack** **Model serving:** AWS Bedrock and Bedrock AgentCore across US, EU, and AU regions; TrueFoundry as the model-serving and deployment path **Sandboxed execution:** AgentCore code-interpreter sessions with session lifecycle limits, network controls, and least-privilege IAM **AI observability:** Grafana AI agent observability: token spend, per-model and per-region latency, throttle and retry rates, tool-call failures, and end-to-end agent traces, tied to journey-based SLOs **Delivery gates:** model and prompt changes versioned, gated on eval and regression suites in CI (GitHub Actions), and rolled out or rolled back with Harness feature flags **Runs on:** multi-region AWS (ECS Fargate and Lambda), defined entirely in Terraform ## **Your first 90 days** **Days 1–30, map and instrument.** Inventory the AI runtime across all three products: regions, model dependencies, IAM posture, IaC coverage, observability gaps. Establish an honest AI cost baseline attributed by business unit. Ship one visible observability win, such as a token-spend-and-throttle dashboard. **Days 31–60, close the highest-risk gap.** Bring the multi-region model runtime fully under Terraform with no drift. Publish draft journey-based SLOs for the three products, with the pods bought in. Land one delivery-safety improvement: the matching eval suite running as a required CI gate. **Days 61–90, make it durable.** Own the model-serving path end to end: production-ready, documented, and with no single point of knowledge. Write down the AI runtime patterns and runbooks so the next team extends them instead of rebuilding. Propose the next quarter of AI infrastructure work with the reliability or cost impact attached. ## **How we'll measure success at 12 months** 1. The multi-region AI runtime is production-grade: deployed and operated across US, EU, and AU, entirely in code, with no drift attributable to AI infrastructure. 2. Journey-based SLOs are live for Transform, AI Matching, and AutoBuilder, with AI-specific signals (tokens, throttles, tool-call failures, generation success) on the dashboards leadership already reads weekly. 3. AI spend is attributed and bounded: per-business-unit attribution, a defensible unit-cost metric, and a delivered reduction on the AI line. 4. Model and prompt changes ship behind gates: eval suites run in CI, changes are flag-controlled and reversible, and silent accuracy regressions are caught before production. 5. Tenant isolation is enforced and evidenced on every AI path, and the AI stack clears its SOC 2 / ISO cycle with evidence ready. 6. The runtime patterns are documented and adopted rather than rebuilt, with no single point of knowledge. ## **Interview process** About four hours of conversation after a 30-minute recruiter screen. We move fast between stages. 1. **Recruiter screen (30 min).** Scope, level, location, compensation. 2. **Hiring manager (45 min).** The AI infrastructure problem in your own words; what you've owned and what broke. 3. **Technical deep dive (75 min).** A production incident or migration you led, interrogated properly, then hands-on: read an unfamiliar Terraform module and service, diagnose a described failure, propose the change. Use your normal tooling, including AI assistants; we care about how you verify. 4. **AI infrastructure design (60 min).** Design the serving, scaling, observability, and cost-control path for an LLM-backed feature under multi-region data-residency constraints. Whiteboard, no coding trivia. 5. **Team panel (60 min).** An engineering lead from Transform or Close AI, a Security or Compliance partner, and a conversation against FloQast's engineering values: Real Artists Ship; Do What Makes the Beer Taste Better; Cross the Water Then Build a Bridge; Look Before You Lock.