Quality Lead, YouTube Global Vendor Operations

Google's software engineers develop the next-generation technologies that change how billions of users connect, explore, and interact with information and one another. Our products need to handle information at massive scale, and extend well beyond web search. We're looking for engineers who bring fresh ideas from all areas, including information retrieval, distributed computing, large-scale system design, networking and data storage, security, artificial intelligence, natural language processing, UI design and mobile; the list goes on and is growing every day. As a software engineer, you will work on a specific project critical to Google’s needs with opportunities to switch teams and projects as you and our fast-paced business grow and evolve. We need our engineers to be versatile, display leadership qualities and be enthusiastic to take on new problems across the full-stack as we continue to push technology forward.

Responsibilities

• Design Quality Assurance (QA) frameworks to ensure alignment with key business outcomes. Monitor AI QA model performance and accuracy using metrics such as precision, recall, and human interrater agreement.

• Manage the ingestion of human-validated golden labeling from manual overrides to drive continuous, high-fidelity model retraining.

• Implement blind review protocols and complex multi-tiered human+AI arbitration to prevent automation bias.

• Design stratified sampling methodologies that prioritize high-risk or ambiguous interactions (e.g., cases where AI confidence is low) for human review.

• Partner with the training team to remediate quality gaps.

Qualifications Minimum qualifications:

• Bachelor's degree or equivalent practical experience.

• 7 years of experience with QA automation and sampling tools.

• 7 years of experience in Large Language Model (LLM) evaluation.

• Experience with LLM-based evaluation frameworks and prompt engineering.

• Experience in contact center metrics and QA methodologies.

Preferred qualifications:

• MBA or Master's degree in a related field or Council of Supply Chain Management Professionals (CSCMP) certification.

• 7 years of experience using data analysis to drive decision making.

• 7 years of experience building relationships with stakeholders or clients.

• Experience in data center equipment environments or computer component manufacturing.

• Excellent problem-solving skills with the ability to apply structured thinking and logic to your work and provide solutions to challenges.