Senior DevOps Engineer

Job description

Come work at a place where innovation and teamwork come together to support the most exciting missions in the world! Role Overview We are seeking a Senior DevOps Engineer, Observability to design, build, operate, and continuously improve a high-scale observability platform built around ClickHouse, HyperDX, OpenTelemetry, Kubernetes, and modern DevOps automation practices. This role is intended for a senior hands-on engineer who can take end-to-end ownership of observability infrastructure for production environments. The engineer will be responsible for building reliable telemetry pipelines for logs, metrics, and distributed traces, optimizing ClickHouse for large-scale observability workloads, operating HyperDX for troubleshooting and application performance analysis, and partnering with application, platform, and SRE teams to improve production visibility, reliability, and incident response. The ideal candidate is deeply technical, operationally disciplined, automation-oriented, and comfortable working in high-volume, production-critical environments. Senior-level expectation: This role requires ownership beyond task execution, including technical judgment, production accountability, automation-first delivery, mentoring, and clear communication during incidents and escalations. Key Responsibilities • Design, build, and operate scalable observability platforms using ClickHouse, HyperDX, OpenTelemetry, Kubernetes, Prometheus, Grafana, Alertmanager, Fluent Bit, and Filebeat. • Architect and optimize ClickHouse for high-volume observability workloads, including logs, traces, metrics, and telemetry analytics. • Design and manage ClickHouse schemas, partitioning strategies, ordering keys, TTLs, materialized views, retention policies, storage efficiency, and query optimization. • Build, operate, and improve HyperDX for log search, distributed tracing, service analysis, dashboards, telemetry correlation, troubleshooting, and root-cause analysis. • Build and maintain scalable OpenTelemetry Collector pipelines for collecting, processing, enriching, filtering, sampling, and routing telemetry data. • Implement reliable correlation across logs, metrics, and traces to support faster application troubleshooting, service dependency analysis, and incident resolution. • Design resilient telemetry pipelines with batching, queuing, retries, backpressure handling, sampling, rate limiting, and cardinality controls. • Deploy and operate observability infrastructure on Kubernetes, with focus on scalability, high availability, capacity planning, resiliency, and operational safety. • Automate infrastructure deployment, configuration management, platform upgrades, application onboarding, and recurring operational tasks using DevOps best practices. • Build and maintain CI/CD workflows using Jenkins, infrastructure automation using Terraform and Ansible, and service discovery or secrets management integrations using HashiCorp Consul and Vault. • Partner with application engineering, platform engineering, SRE, and operations teams to troubleshoot production issues, improve observability coverage, and reduce mean time to detect and resolve incidents. • Analyze production performance issues across infrastructure, applications, telemetry pipelines, and ClickHouse queries, and drive corrective actions to closure. • Define and improve standards for telemetry instrumentation, log quality, metric hygiene, trace propagation, dashboard design, alert quality, and production readiness. • Participate in incident response, post-incident reviews, capacity planning, operational reviews, and remediation tracking for observability services. • Mentor junior engineers, review designs and automation changes, and raise the overall technical and operational maturity of the team. Required Qualifications • 6+ years of experience in DevOps, SRE, Platform Engineering, Infrastructure Engineering, or Observability Engineering roles. • Strong production experience with ClickHouse, including architecture, administration, schema design, performance tuning, query optimization, troubleshooting, retention management, and operations at scale. • Hands-on experience deploying and operating HyperDX, including integration with ClickHouse and usage for log search, distributed tracing, dashboards, service analysis, and troubleshooting. • Strong experience with OpenTelemetry and OpenTelemetry Collector, including receiver, processor, exporter, sampling, enrichment, batching, and routing configurations. • Strong working knowledge of Fluent Bit, Filebeat, Prometheus, Alertmanager, Grafana, ClickHouse, HyperDX, and OpenTelemetry. • Strong DevOps and automation experience with Jenkins, CI/CD pipelines, Ansible, Terraform, HashiCorp Consul, HashiCorp Vault, and Kubernetes. • Strong understanding of Linux, networking, microservices, REST, gRPC, distributed systems, high-availability architectures, and production operations. • Experience operating and troubleshooting high-volume production systems with focus on reliability, scalability, capacity, and performance. • Ability to debug complex issues across application telemetry, Kubernetes infrastructure, data ingestion pipelines, storage systems, and query performance. • Strong scripting and automation skills, with the ability to reduce manual operations and improve repeatability, reliability, and operational efficiency. • Ability to take ownership of production systems, drive problems to closure, communicate clearly during incidents, and collaborate effectively with engineering and operations teams. Preferred Qualifications • Development experience with Java and Python, including the ability to understand application code, troubleshoot instrumentation issues, and analyze performance behavior. • Experience developing automation, internal tools, APIs, platform services, or self-service onboarding workflows. • Experience with Kafka or other high-throughput messaging and streaming platforms. • Strong understanding of APM, distributed tracing, OpenTelemetry instrumentation, context propagation, service maps, and service dependency analysis. • Experience with cloud platforms such as AWS, Azure, GCP, or OCI. • Experience designing observability solutions for large-scale Kubernetes, microservices, or distributed application environments. • Experience improving alert quality, reducing noise, defining SLOs or SLIs, and supporting production readiness reviews. Senior Engineer Expectations As a Senior DevOps Engineer, this role is expected to go beyond task execution and operate with strong ownership, judgment, technical leadership, and accountability for production outcomes. The candidate should be able to: • Own critical observability services end to end, from design and implementation to production operations, troubleshooting, and continuous improvement. • Make sound technical decisions around scalability, reliability, performance, cost, capacity, security, and operational maintainability. • Proactively identify risks, gaps, and scaling bottlenecks before they become production issues. • Build automation-first solutions rather than relying on manual processes. • Lead technical investigations during complex incidents and drive clear remediation plans. • Influence application and platform teams to improve instrumentation quality, telemetry consistency, and operational readiness. • Write clear technical documentation, operational runbooks, onboarding guides, and incident review notes. • Mentor engineers, review technical designs, and help establish standards for observability engineering. • Communicate effectively with technical and non-technical stakeholders during incidents, escalations, and planning discussions. • Demonstrate strong accountability for production stability, platform reliability, and measurable operational outcomes.