Senior Software Engineer - SRE, Retail and Pharmacy

Job description

We’re building a world of health around every individual — shaping a more connected, convenient and compassionate health experience. At CVS Health®, you’ll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger – helping to simplify health care one person, one family and one community at a time. Position Summary: About the Team Our Site Reliability Engineering team is the execution engine behind the reliability, availability, and performance of distributed store technology powering thousands of retail and pharmacy locations nationwide. We operate across pharmacy platforms, Point of Sale (POS) systems, handheld devices, store servers, dispensing systems, and edge computing infrastructure — spanning hybrid cloud and on-premises environments at massive fleet scale. Our engineering philosophy is grounded in five pillars: Detection , Prevention , Recovery , Learning Loops , and Developer Experience (DevX) . Our operating principle is the reliability covenant : our success is not measured by incident response volume — it is measured by the reliability capability we transfer to the engineering teams we serve. Your success in this role is measured by what the engineering teams in your domain can do independently after working with you , not by how indispensable you become to them. An SSE who has enabled a development team to detect, respond to, and learn from production failures without SRE involvement has delivered the highest-value outcome this role can produce. We track operational toil as an engineering metric . Engineers at this level are expected to identify recurring manual work, eliminate it through automation, document the reduction, and treat toil accumulation as a reliability risk — not as a sign of operational expertise. About the Role As a Senior Software Engineer — SRE , you independently own the reliability posture of an assigned engineering domain. You are not waiting for direction — you are setting it for your domain. You design the alerting strategy, own the SLO health, lead incident command for production issues, facilitate postmortems, tune anomaly detection models, and partner directly with engineering domain owners to shift reliability left into design. You are a technical mentor to SE-level engineers and an escalation resource during active incidents. You have the technical depth to diagnose complex distributed system failures, the data instincts to distinguish genuine anomalies from noise in ML-generated signals, and the organizational skills to drive reliability practice adoption in teams that did not necessarily ask for SRE involvement. Scope: Domain ownership — you operate independently and influence adjacent engineering teams. The Environment You Are Joining This role exists inside an active SRE transformation. The domain-based SRE ownership model you will operate within is in its early stages. Some of the toolchains you will work with are being built in parallel with the operational work. Engineering domain owners are simultaneously learning what SRE can offer them. This is an honest description of the role, not a caveat. The opportunity is to shape a domain's reliability posture from the ground up in a large-scale, consequential technology environment. The engineering decisions you make will affect pharmacy dispensing, prescription fill workflows, and store operations across thousands of locations. If you are energized by the combination of technical depth and organizational building, the scope here is significant. If you are looking for a mature, fully-defined SRE environment where the frameworks and toolchains already exist, this role will feel like a different challenge. Success in this role requires patience alongside technical rigor : you will demonstrate value before you demand process change, build credibility before you expect adoption, and earn trust with engineering domain owners through partnership rather than mandate. The operating environment includes an edge computing fleet deployed directly inside store locations — unattended nodes where deployment blast radius is geographic and fleet-wide, not functional and service-scoped. You will develop a fleet operations mindset: the primary failure mode in this environment is deployment and configuration propagation, not service logic. What You Will Do Detection & Observability Own SLI/SLO health for your assigned domain end-to-end; monitor error budget burn rates and drive proactive burn-down actions before incidents reach end users or pharmacy patients Design multi-signal alerting strategies that go well beyond threshold alerts: burn rate alerting, composite health signals, and anomaly-based detection using time-series models — and validate their output against production ground truth Tune and maintain ML-based anomaly detection models in production: adjust sensitivity thresholds, evaluate false positive rates against alert fatigue metrics, and iterate on model configurations based on incident data Build Critical User Journey (CUJ)-anchored dashboards that surface end-to-end business flow health — not just individual service metrics Drive alert signal-to-noise improvement across the domain; own suppression policies during deployments and maintenance windows to protect on-call quality Identify and instrument unmonitored external dependencies — third-party APIs, downstream services, data providers — using tools like Prometheus Blackbox Exporter , OpenTelemetry , or custom health probes Contribute to the observability platform design: understand the hot/warm/cold tier architecture ( Apache Kafka → ClickHouse → cold storage) and contribute domain-specific data models and SLI schemas Prevention & Reliability Engineering Lead Production Readiness Reviews (PRR) for services in your assigned domain; own the readiness gate sign-off and be accountable for what makes it into production on your watch Design and execute fault injection experiments at service level using tools such as LitmusChaos , Chaos Toolkit , or Gremlin ; validate blast radius assumptions before rollouts reach production Partner with engineering domain owners on reliability requirements during architecture design and sprint planning — reliability is designed in, not bolted on Own dependency risk mapping for your domain: catalog third-party API failure modes, shared infrastructure failure paths, and chain-wide blast radius scenarios Apply a fleet operations mindset to change management: for any configuration or deployment change touching the edge fleet, assess deployment blast radius by node cohort, review the rollback procedure, and contribute to the go/no-go decision on high-risk changes Understand progressive rollout strategies — canary cohorts, staged fleet expansion, blast radius budgets — and apply them in high-risk deployment reviews Incident Response & Recovery Serve as the primary on-call Technical Incident Commander (IC) for domain incidents; drive structured bridge calls from detection to resolution using established incident command frameworks Author and maintain P0/P1 runbooks with validated, step-by-step remediation procedures; own a quarterly review and dry-run testing cadence to ensure runbooks are accurate when they are actually needed Lead post-incident reviews using structured root cause analysis: causal chain documentation, origin layer classification, and contributing factor identification — not just a timeline of what happened Serve as the real-time escalation point and technical decision support for SE engineers during active incidents Pursue Technical Incident Commander (TIC) certification; qualify as a cross-domain IC candidate available to lead major incidents beyond your assigned domain Learning Loops & Continuous Improvement Learning Loops at this level is systems engineering for organizational memory — not postmortem administration. Facilitate domain-level postmortems with rigor and structure: timestamped timelines, contributing factor taxonomy (origin layer + failure pattern classification), systemic findings, and action items with owners, due dates, and measurable definitions of done Measure learning velocity : track not just whether action items close, but whether the incident class frequency decreases after a fix is applied. Report the ratio of incident classes that recurred versus incident classes that were resolved systemically. This is your primary learning loop quality metric Identify recurring incident patterns across your domain and drive architectural or process changes that eliminate the root cause — not suppress the symptom Close the prevention loop : translate postmortem findings directly into PRR requirement updates, new monitoring coverage, runbook improvements, or development-team practices that prevent recurrence. A postmortem that doesn't change something upstream hasn't finished Mentor SE-level engineers on SRE craft — not just on what to do, but on why the learning loop exists: connect the postmortem to the PRR, the PRR to the design review, the design review to the runbook, and the runbook back to the next incident Developer Experience & Automation (DevX) Eliminate toil systematically : identify and automate manual operational work in your domain using Python , Go , or Bash . Track toil volume eliminated per quarter — time recovered, manual touchpoints removed, and error modes eliminated through automation Build self-service observability tooling that enables development teams to own their own service health dashboards and SLO status pages — reducing SRE as a dependency for basic operational visibility Track and report DORA metrics — Deployment Frequency, Lead Time for Change, MTTR, and Change Failure Rate — for your assigned domain on a quarterly basis; use the data to target reliability investments where they will have the most impact Partner with development teams to shift reliability practices left: reliability checklists in design reviews, SLI instrumentation in definition-of-done, runbook templates that developers can author and own Contribute to the team's shared SRE tooling library; write reusable, well-documented automation modules for common operational patterns Organizational Influence This is a core function of the SSE role — not a soft skill addendum. Drive reliability practice adoption without mandate : you will work with engineering teams that have operated independently for years. Adoption of SLO culture, PRR gates, and chaos engineering practices depends entirely on your ability to build credibility through demonstrated value, not through authority you don't have Establish working relationships with engineering domain owners; understand their delivery pressures, their quality concerns, and what they need from SRE to say yes to new reliability requirements Identify the smallest, highest-value reliability practice that a skeptical engineering team will adopt first — land that win, document the outcome, and use it to earn the trust needed for the next one When reliability practices are not being adopted, diagnose the real reason: unclear value, unclear ownership, too much friction, or wrong timing — and adapt the approach rather than escalating to mandate Required Qualifications 5+ years of experience in SRE, DevOps, platform engineering, or related production-systems roles 3+ years operating cloud-native distributed systems at production scale with active on-call responsibility Demonstrated experience as an on-call Incident Commander (IC) for P1 or P2 incidents — structured bridge leadership, not just participant involvement Experience tuning and validating time-series anomaly detection models in a production observability context — this is a Required qualification, not a preferred one; anomaly-based detection is a core function of this role Strong programming proficiency in at least one of Python , Go , or Java at production quality — capable of writing operational tooling that other engineers will rely on Hands-on experience designing SLIs, SLOs, and managing error budgets for customer-facing or business-critical services Deep observability platform experience: Prometheus , Grafana , OpenTelemetry , and at least one log aggregation solution (Loki, Splunk, Elasticsearch) Organizational influence without authority : demonstrated track record of driving SRE practice adoption in engineering teams that did not initially request SRE involvement — this is a Required qualification; technical depth alone is insufficient at this level in a transformation environment Fleet-scale deployment awareness: familiarity with progressive rollout strategies, blast radius management, and configuration drift as a reliability risk in large unattended node deployments Strong cloud platform expertise: AWS , Microsoft Azure , or Google Cloud Platform (GCP) Advanced Kubernetes operational experience: debugging, resource management, networking policies, and workload failure modes. Experience with AI-assisted tooling and development. Preferred Qualifications Experience owning Production Readiness Reviews or service launch gates Hands-on chaos or fault injection experience using LitmusChaos , Chaos Toolkit , or Gremlin TIC (Technical Incident Commander) certification or equivalent structured incident command training Experience operating distributed systems in retail, pharmacy, healthcare, or other operationally sensitive environments where failures have direct patient or customer impact LLM integration for operational use cases (alert summarization, runbook suggestion, incident triage assistance) — design or implementation experience Experience with streaming data platforms: Apache Kafka , Redpanda , Apache Flink , or ksqlDB Familiarity with analytical databases for observability workloads: ClickHouse , Apache Druid , or TimescaleDB Experience with service mesh and traffic management: Istio , Envoy , Linkerd Infrastructure-as-code proficiency at production scale: Terraform , Pulumi , or Ansible What Success Looks Like at 6 Months You own the SLO health and alerting strategy for your assigned domain with full independence — including the ML-based anomaly detection configuration — and the false positive rate is measurably lower than when you joined You have led at least five P1/P2 incident bridges as Incident Commander, with structured postmortems published, action items closed, and at least two incident classes eliminated through root cause remediation rather than symptom suppression You have facilitated your first domain-level PRR and signed off on a service launch At least two engineering teams in your domain have adopted a reliability practice — an SLO health check, a runbook standard, or a deployment gate — that you introduced without top-down mandate You have eliminated measurable toil in your domain: at least three recurring manual tasks automated and documented with before/after comparisons The engineering domain owner you partner with describes SRE as a force multiplier for their team's delivery velocity — not as a gating function that slows them down Education: Bachelor's degree in Computer Science, Engineering, or a related field — or equivalent practical experience Anticipated Weekly Hours 40 Time Type Full time Pay Range The typical pay range for this role is: $92,700.00 - $203,940.00 This pay range represents the base hourly rate or base annual full-time salary for all positions in the job grade within which this position falls. The actual base salary offer will depend on a variety of factors including experience, education, geography and other relevant factors. This position is eligible for a CVS Health bonus, commission or short-term incentive program in addition to the base pay range listed above. Our people fuel our future. Our teams reflect the customers, patients, members and communities we serve and we are committed to fostering a workplace where every colleague feels valued and that they belong. Great benefits for great people We take pride in offering a comprehensive and competitive mix of pay and benefits that reflects our commitment to our colleagues and their families. This full‑time position is eligible for a comprehensive benefits package designed to support the physical, emotional, and financial well‑being of colleagues and their families. The benefits for this position include medical, dental, and vision coverage, paid time off, retirement savings options, wellness programs, and other resources, based on eligibility. Additional details about available benefits are provided during the application process and on Benefits Moments . We anticipate the application window for this opening will close on: 10/31/2026 Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state and local laws.