Senior Automation QA (JS) Engineer

Job description

**This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Senior Automation QA (JS) Engineer based in India (Remote).** As a Senior Automation QA Engineer, you will play a pivotal role in ensuring the reliability, security, and performance of advanced AI-powered platforms. You will design and develop sophisticated automated testing frameworks using Playwright and TypeScript, covering frontend, backend, APIs, and complex asynchronous workflows. Working closely with AI and platform engineering teams, you will establish robust quality standards for applications powered by large language models (LLMs) and autonomous agents. You will develop innovative testing strategies to evaluate nondeterministic AI behavior, detect security vulnerabilities, and validate data integrity across multi-tenant environments. Your work will directly influence release decisions, system resilience, and the trustworthiness of AI-driven products. This remote opportunity offers the chance to shape next-generation quality engineering practices while mentoring colleagues and contributing to high-impact international projects. ### Accountabilities: - **Test Automation Framework Development:** Design, build, and maintain scalable automated test frameworks using Playwright and TypeScript. Develop comprehensive test suites covering frontend applications, backend services, APIs, and contract compatibility to ensure reliable platform behavior as products evolve. - **AI and LLM Quality Assurance:** Create specialized testing strategies for nondeterministic AI systems, including statistical assertions across repeated runs, semantic similarity evaluation, confidence-threshold validation, and techniques for distinguishing model variability from genuine software regressions. - **Workflow Reliability and Resilience Testing:** Validate long-running, asynchronous, and event-driven workflows through failure injection, retry scenarios, worker termination, provider outages, idempotency checks, and eventual-consistency assertions. Verify durable execution and appropriate compensation mechanisms during failures. - **Multi-Tenant Security and Isolation:** Develop automated tests to identify cross-tenant data exposure, configuration leakage, and inaccurate cost attribution. Validate execution sandboxing, network restrictions, and access controls to ensure platform security and tenant isolation. - **Adversarial and AI Security Testing:** Build and maintain adversarial test datasets to identify prompt injection, tool-call manipulation, unauthorized actions, and potential data exfiltration. Align testing practices with recognized AI security frameworks, including the OWASP Top 10 for LLM Applications. - **Human-in-the-Loop Validation:** Verify that confidence thresholds trigger the appropriate escalation procedures and that restricted or irreversible actions cannot proceed without the required human authorization. - **Evaluation Infrastructure and Test Data:** Collaborate with AI evaluation teams to build golden-dataset-driven validation processes, synthetic-data generation pipelines, and repeatable evaluation infrastructure. Ensure datasets are reliable, measurable, and suitable for ongoing regression testing. - **Performance and Non-Functional Testing:** Automate verification of latency, invocation overhead, concurrency, queue backpressure, and AI provider failover times. Analyze performance trends across releases and identify opportunities to improve system efficiency and resilience. - **Observability and Trace Validation:** Use tools such as Langfuse and OpenTelemetry to validate trace completeness, token usage, cost accounting, and anomaly detection. Incorporate observability data into automated testing and troubleshooting workflows. - **CI/CD Integration and Release Quality Gates:** Integrate automated test suites into CI/CD pipelines using GitHub Actions or equivalent tools. Establish mandatory release gates, automate regression detection, and flag metric-level quality issues during pull requests and deployments. - **Quality Evidence and Stakeholder Collaboration:** Produce reproducible, auditable test reports and evidence packages to support client milestone approvals. Communicate testing outcomes, risks, and quality assessments clearly to technical and non-technical stakeholders. - **Architecture and Testability:** Partner with AI and platform engineers from the design stage to define acceptance criteria, improve testability, and incorporate observability and quality considerations into system architecture. - **Environment Management and Mentorship:** Maintain test environments, mocked AI providers, and synthetic data generators to enable efficient and cost-effective testing. Mentor mid-level QA engineers, develop reusable testing frameworks, and contribute to shared quality engineering practices. ## Requirements: - **Professional Experience:** At least 8 years of experience in test automation, API testing, and developing reusable test frameworks adopted by engineering teams. - **Automation Expertise:** Strong proficiency in Playwright and TypeScript is mandatory, with demonstrated experience designing comprehensive automated test suites and integrating them into secure, reliable CI/CD pipelines. - **AI and LLM Testing:** Hands-on experience testing LLM-powered or other nondeterministic systems, including statistical assertions, semantic scoring, prompt sensitivity, model variability, and evaluation of common Retrieval-Augmented Generation (RAG) failure modes. Candidates should be able to demonstrate practical examples of these testing approaches. - **Asynchronous Systems:** Experience testing event-driven architectures, distributed workflows, and background processing systems. Strong knowledge of failure injection, idempotency, retries, and eventual consistency is required; experience with Temporal or a comparable workflow engine is a significant advantage. - **CI/CD and Release Management:** Proven ability to integrate automated testing into continuous integration and delivery pipelines, preferably using GitHub Actions or equivalent tools. Experience defining release gates and making informed quality decisions under tight deadlines is essential. - **Database and Data Validation:** Solid SQL skills, experience validating data integrity, and the ability to develop synthetic test-data pipelines and assess dataset quality. - **Security Testing:** A security-focused mindset with experience in adversarial testing, application security, or prompt-injection testing. Candidates should be comfortable identifying vulnerabilities and developing effective negative test scenarios. - **Analytical and Evaluation Skills:** Strong statistical and analytical capabilities, including interpreting evaluation scores, identifying behavioral patterns, and providing actionable feedback to AI engineers for model and prompt improvements. - **Quality Judgment and Communication:** Demonstrated ability to determine acceptable quality thresholds under delivery pressure, assess technical risks, and defend testing evidence and release recommendations to non-engineering stakeholders. - **Performance Testing:** Familiarity with performance and load-testing tools such as k6 or Locust is an advantage, particularly for validating concurrency, latency, and high-volume background processing. - **AI Observability and Evaluation Tools:** Exposure to tools such as Langfuse, LangSmith, Promptfoo, DeepEval, or Arize Phoenix is desirable. - **Multi-Tenant SaaS Experience:** Previous experience testing tenant isolation, access controls, and data separation in multi-tenant SaaS platforms is beneficial. - **Compliance and Accessibility:** Experience producing compliance evidence for frameworks or regulations such as SOC 2 and GDPR, or performing accessibility testing against Section 508 or WCAG standards, is an advantage. - **Domain Knowledge:** Familiarity with commercial real estate workflows, lease accounting, common area maintenance (CAM) reconciliation, document extraction, OCR, or document layout parsing is beneficial. - **Advanced Testing Practices:** Experience using LLMs to generate test cases, adversarial payloads, or synthetic documents is an advantage. Familiarity with qTest for test management is also desirable. - **Collaboration and Mentorship:** Strong communication, problem-solving, and teamwork skills, with the ability to collaborate across engineering disciplines, share expertise, and promote consistent quality practices. - **Work Arrangement:** Willingness to work remotely from India and collaborate effectively with distributed international teams. ## Benefits: - **Remote Work Flexibility:** Work remotely from India as part of a globally distributed professional community. - **Collaborative Environment:** Join a supportive, open-door culture that encourages knowledge sharing, teamwork, and continuous improvement. - **Professional Development:** Access internal meetups, conferences, workshops, language courses, and Udemy learning resources. - **Company-Funded Certifications:** Opportunities to strengthen your technical expertise through certifications sponsored by the employer. - **Career Growth:** Work on large-scale, high-impact projects and explore internal mobility opportunities across different industries and technology domains. - **Cutting-Edge Technology:** Gain hands-on experience with AI-powered applications, advanced automation frameworks, and modern quality engineering practices. - **Healthcare Coverage:** Company-paid medical insurance. - **Mental Health Support:** Access to resources designed to support employee well-being. - **Financial and Legal Assistance:** Access to financial and legal consultation services. - **Global Collaboration:** Work alongside experienced engineers and technology professionals on innovative solutions with international reach.