Staff QA Engineer

Job description

**Job Summary** We are seeking a **Staff QA Engineer** to provide technical leadership for Quality Engineering across the **DDN Horizon Control**, including **Horizon Control & Horizon Store.** This is a senior individual contributor role responsible for defining test strategy, designing automation frameworks, developing comprehensive validation solutions, and driving release quality for a distributed cloud-native system built with Go, Kubernetes, NATS, PostgreSQL, REST APIs, and event-driven workflows. You will work closely with software engineers, architects, product management, DevOps, and program teams to ensure Horizon Control delivers enterprise-grade reliability, scalability, security, and usability. The ideal candidate will be equally comfortable performing exploratory testing, developing automated test frameworks, validating distributed system behavior, troubleshooting complex failures, and driving quality improvements throughout the software development lifecycle. This role requires strong expertise in both **manual and automated testing**. You will be expected to contribute hands-on to test design, automation development, system validation, troubleshooting, root cause analysis, and release readiness while influencing engineering decisions through technical leadership. The ideal candidate is passionate about building quality into the product from the design phase, has experience validating distributed systems at scale, and enjoys solving complex technical challenges across APIs, microservices, event-driven architectures, Kubernetes environments, and AI infrastructure platforms. **Key Responsibilities** - Review architecture, design, and API specifications to identify quality risks early in the development lifecycle. - Define test coverage, automation strategy, quality gates, and release readiness criteria. - Drive quality improvements across functional, integration, system, security, and non-functional testing. - Define and drive the overall test strategy for Horizon Control. - Mentor engineers on test design, automation practices, debugging, and quality engineering methodologies. - Design, implement, and maintain scalable automation frameworks for Horizon Control. - Build and maintain API, integration, and system test automation using Go / Godog / Gherkin / BDD / Playwright / TypeScript - Create reusable test libraries, fixtures, mocks, and reporting solutions. - Integrate automated testing into GitHub Actions CI/CD pipelines. - Ensure automated coverage for critical business workflows and release validation. - Design and execute integration testing across: Horizon Control, Provider Engine, MOSS integrations, Rafay providers, NATS JetStream, PostgreSQL and Kubernetes environments - Validating Authentication workflows, RBAC enforcement, Multi-tenant isolation, Async operations, Event delivery, Reconciliation, Drift detection, Provider interactions and Error handling and recovery - Distributed Systems and Reliability Testing Validate: Transactional outbox patterns, Event-driven workflows, At-least-once delivery semantics, Idempotency handling, Retry mechanisms, Operation state transitions, Concurrent user workflows, Eventual consistency models, Develop failure injection and resiliency testing scenarios. - Non-Functional Testing Design and execute: Performance testing, Scalability testing, Stress testing, Longevity testing, Upgrade testing and Serviceability testing. Develop performance automation using: k6, JMeter and Go benchmarks. Analyze bottlenecks and collaborate with engineering teams to improve performance and scalability. - Drive test planning, execution, defect triage, and release validation activities. - Define release readiness metrics and quality dashboards. - Provide risk assessments and quality recommendations. - Assist with troubleshooting customer-reported issues through defect reproduction and root-cause analysis. - Deliver clear executive-level quality status updates. **Required Skills & Experience** - 12+ years of software quality engineering experience. - Strong programming skills in Go, Python, Java, or a comparable language. - Hands-on experience building automation frameworks from scratch. - Strong experience with API testing and automation. - Strong understanding of REST APIs, OpenAPI, JSON and Event-driven architectures - Experience testing distributed systems, cloud-native applications, or microservices-based architectures. - Experience working with: Kubernetes, Docker, Helm and GitHub Actions - Experience validating asynchronous systems and messaging platforms. - Understanding of Authentication and Authorization, JWT, OAuth, RBAC and Multi-tenancy - Experience executing system, integration security, and non-functional testing. - Strong debugging and root-cause analysis capabilities. - Excellent communication and technical leadership skills. **Preferred Qualifications** - Experience with Go-based automation development. - Hands-on experience with Godog and Gherkin-based BDD automation. - Experience with Playwright and UI test automation. - Understanding of NATS, Kafka, RabbitMQ, or similar messaging technologies. - Familiarity with Rafay or similar infrastructure orchestration platforms. - Experience validating AI/ML infrastructure platforms. - Experience with PostgreSQL and distributed data systems. - Experience with Open Telemetry, Prometheus, and modern observability tooling. - Experience validating FIPS-compliant software environments. - Experience and/or Understanding of traditional block storage and file storage protocols like CIFS, NFS, and iSCSI. - Understanding of Storage concepts such as RAID, SCSI, FC and basic networking concepts.