Principal Engineer, Observability

Job description

**Firmus Technologies** Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most efficient AI infrastructure by combining cutting-edge technology with a steadfast commitment to sustainability. At Firmus, we are unique in our approach. We design, build, and operate a new class of digital infrastructure – the AI Factory. Through our model-to-grid technology approach, we have pushed the boundaries of multi-generational liquid cooling systems, energy management, AI software orchestration, and construction. For our customers, this approach allows us to make every watt count and deliver low-cost AI tokens globally. **Firmus AI Cloud** Our large-scale GPU cloud platform, Firmus AI Cloud, is purpose-built to deliver energy-efficient AI compute at scale to customers. It empowers developers, enterprises, educational institutions, and government users to train and deploy AI models with unmatched efficiency and cost savings. With an ever-growing suite of services and applications, we are committed to delivering a cloud experience that is market-leading, proprietary, and built to scale. **ROLE SUMMARY** Firmus Technologies is seeking a Principal Engineer, Observability to own how Firmus operations teams and customers see the AI factory as one system, from site power and cooling to the GPU and the workload running on it. You will own the telemetry standard every team emits to, the platform that collects, stores, and queries it, and the applications that operations teams and customers use. Every team should be able to build its own dashboards and alerts without waiting on yours. Every new site adds millions of samples per second, and the platform must keep up as Firmus grows to hundreds of thousands of GPUs in gigawatt-scale AI factories, without losing critical data or slowing queries. **KEY RESPONSIBILITIES** **Architecture and decisions** - Own the architecture of the observability platform across every data centre and cloud region, from collection and transport to storage, query, alerting, and the applications on top. Set a target architecture two to three years ahead and plan the steps from today's system to it. - Design the platform to grow tenfold without a redesign, through sharding, retention tiers, cardinality budgets, and capacity plans. Prove the headroom with load tests before each new site goes live. - Design around the same repeatable unit as the estate. Each site keeps observing itself when its link to the rest of Firmus is down, one global view spans every site, and new sites come on through the site deployment kit with no local fork. - Own the telemetry standard: what each layer emits and how signals are named, labelled, and versioned, enforced by checks in every team's build pipeline. - Lead design reviews for significant changes across the teams, sign off architecture decision records, and settle technical disputes escalated by engineers. Decide at pace with incomplete information, write down the assumptions, and revisit a decision when the evidence changes. **Build and evolve** - Write production code for the hardest parts, including collection at fleet scale, the telemetry pipeline, query performance, and the observability API and applications. - Make observability self-service for every team. Teams onboard a service and build their own dashboards, alerts, and SLOs as code from templates, within the telemetry and alerting standards and their own budgets for cardinality and retention, without filing a ticket. - Set delivery guarantees for each class of telemetry. Critical operational and customer-billable signals are never lost, and any loss elsewhere is measured and recoverable. - Build and own the live topology model (digital twin) of every site, hall, rack, tray, GPU, network link, and cooling loop. Keep it in step with the inventory source of truth and map every signal to it. - Cover all layer customers depend on, from GPU and interconnect health and network fabric events to storage performance per job and Kubernetes control-plane metrics. Run passive and active health checks that signal failing GPUs, links, and nodes for draining before customers notice. - Bring facilities telemetry into the same model as compute. Operations teams see power, cooling, and leak events immediately, together with the workloads they affect. - Serve telemetry to internal consumers, including the data platform and the intelligence platform, through stable, versioned interfaces with agreed freshness and completeness, and make it usable by AI agents for diagnosis, with scoped access and an audit trail. - Deliver telemetry to customers and partner platforms on open standards such as OpenTelemetry, within seconds of collection, with ready-made dashboards, query access, alerts to their own channels, and forwarding to their own tools. Each customer sees only their own tenancy, enforced at every layer. **Reliability, security, and cost** - Set SLOs and error budgets for observability and use the budget to decide when reliability work comes first. - Lead the technical response to major incidents in observability and make sure every post-mortem ends with a fix that stops it happening again. - Set the alerting standard. Every alert is actionable, reaches the right responder, and links to a runbook. Prove detection works by injecting faults. - Make service-level measurement reproducible. Firmus and a customer should get the same availability from the same data. - Design the telemetry stores to upgrade in rolling waves with no gap in data, and keep dependencies and images patched within policy. - Keep Firmus' SOC 2 Type 2 and ISO 27001 controls passing at every new site. - Own cost at the design level, including retention, cardinality, and data volume. **Technical leadership** - Grow engineers through mentorship, design review, and code review. Hiring and performance review sit with the