What you will work on

Our Vertical Search team's mission is to deliver the best TV search experience: fast, comprehensive, and reliably relevant. For REA Vertical Search, that means helping users discover the right content across typed, voice, and assistant-driven experiences while supporting a growing set of subscription services and Roku Channel discovery surfaces. We partner with search engineers, ML engineers, product managers, and data partners to define quality strategy for relevance-sensitive experiences, build scalable automation and evaluation frameworks, and apply agentic automation to keep pace with expanding catalog breadth and use cases.

Role Overview

We are looking for a Lead, Search Relevance Quality to work closely with the Roku Vertical Search Team to define, measure, and improve Search and Assistant relevance quality. This person will lead quality strategy for relevance-sensitive experiences, build scalable automation and evaluation frameworks, and apply agentic automation to keep pace with expanding catalog breadth and use cases. The role requires balancing automated quality systems with pragmatic human-in-the-loop review where needed until confidence and coverage are high. Quality for Vertical Search will be measured with clear, trusted metrics rather than inferred only from downstream engagement signals. Search and Assistant launches will have reusable relevance-evaluation coverage, stronger release confidence, and faster debugging loops. The team can scale quality across a growing number of subscription services and Roku Channel experiences without linear growth in manual effort.

How AI is used at Roku

At Roku, AI agents do most of the keystroke-level coding. Engineers act as the technical leads for those agents, setting context, planning work, verifying outputs, and recovering when the agent goes off course. If you are excited about working this way, we want to talk to you.

At Roku, we're embracing AI as a powerful tool to amplify human creativity, accelerate innovation, and deliver better results for our customers. Across teams, we're exploring how AI can enhance our work and we're only looking for people who are curious, adaptable, and excited to grow alongside these technologies.

Responsibilities

Define the quality strategy for Vertical Search and Assistant relevance across Roku Premium subscription services, Roku Channel search journeys, and related discovery experiences
Build scalable frameworks to measure search quality across query understanding, retrieval, ranking, result presentation, and assistant response quality
Create automated evaluation pipelines for offline, pre-launch, and continuous post-launch quality monitoring using modern test infrastructure and agentic workflows
Partner with search engineers, ML engineers, product managers, and data partners to define KPI hypotheses, guardrails, and learning plans for quality-related launches and experiments
Develop quality signals and gold-standard datasets for vertical search, including subscription-service discovery, entity matching, ranking relevance, and assistant answer quality
Use agentic automation to generate, maintain, and expand test coverage and evaluation assets while ensuring governance, traceability, and reliability of the resulting workflows
Introduce the right mix of manual review and automation for ambiguous or high-impact experiences until the automated system is proven and trusted
Drive root-cause analysis for relevance regressions and quality escapes, and convert findings into durable frameworks, dashboards, and release gates
Lead end-to-end test and quality architecture for relevance-heavy search and assistant systems
Design evaluation methodologies for search result quality, assistant response quality, and discovery experience consistency
Build APIs, tools, datasets, and automation harnesses for offline and online quality validation
Establish CI/CD quality gates for ranking, retrieval, and assistant-related regressions
Partner with engineering to instrument systems for observability, diagnosis, and actionable quality telemetry
Translate vague quality concerns into measurable test plans, labeled datasets, and scalable automation workflows
Mentor engineers on quality engineering best practices and raise the technical bar for automation design, signal quality, and testability

Experience for success

8+ years in relevance quality engineering, or software engineering roles with significant ownership of automation and system quality
Strong coding skills in Python, Java, or similar languages, with the ability to build production-grade test frameworks and internal tooling
Deep experience with API, integration, service, and UI automation in complex distributed systems
Experience defining test strategy, test design, and quality architecture for large-scale products
Strong analytical ability to define metrics, interpret results, and isolate root causes across data, ranking logic, and system behavior
Demonstrated ability to work cross-functionally with engineering, product, and data stakeholders
Experience introducing scalable automation in environments where quality must be measured, not assumed
Experience with search relevance, ranking systems, information retrieval, recommendations, NLP, ASR/NLU, or LLM/assistant evaluation
Experience building quality frameworks for user-intent-heavy products where correctness is subjective and must be inferred through labels, rubrics, or review processes
Familiarity with agentic AI tooling for test generation, evaluation orchestration, or QA workflow automation
Experience supporting experimentation-driven teams where every meaningful change has KPI hypotheses and guardrails
Experience with subscription, media, streaming, or catalog discovery products
You have built fluency across the agentic engineering toolchain - coding harnesses like Claude Code or Cursor, MCP servers, custom skills, or agent frameworks, and you can describe projects where you shipped real work with these tools; you know how to drive an agent, verify its output, and ramp on an unfamiliar codebase with an agent helping you

Accommodations

Roku welcomes applicants of all backgrounds and provides reasonable accommodations and adjustments in accordance with applicable law. If you require reasonable accommodation at any point in the hiring process, please let us know.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

Senior Software Engineer, Test

Job Description

No Referrers Available