Sr Applied Scientist, Support Agent Intelligence and Evaluation

Amazon Advertising is one of Amazons fastest growing and most profitable businesses. Amazons advertising portfolio helps merchants retail vendors and brand owners succeed via native advertising which grows incremental sales of their products sold through Amazon. The primary goals are to help shoppers discover new products they love be the most efficient way for advertisers to meet their business objectives and build a sustainable business that continuously innovates on behalf of customers.

Millions of advertisers rely on Amazons self-service support experience to resolve issues unblock campaigns and grow their business. Our Support Agents team is building the science behind intelligent conversational support systems that understand advertiser intent retrieve the right knowledge generate accurate answers and know when to escalate. We serve 2M monthly active advertisers across dozens of locales and languages and every percentage point of improvement in resolution quality translates directly into advertiser success and retention.

We are seeking an Applied Scientist who is passionate about building evaluation science NLP systems and quality measurement at scale. You will define how we measure good designing LLM-as-a-judge evaluation pipelines developing our next-generation Issue Resolution Rate (IRR) metrics and closing the quality gap between English and non-English markets. Your work will directly shape what ships to advertisers and what leadership uses to assess the health of our support experience.

Key job responsibilities
1. Enhance support agent capabilities across the broad suite of Amazon Advertising products expanding coverage depth of resolution and advertiser task completion across Sponsored Products Sponsored Brands DSP AMC and more
2. Design and own the evaluation framework for agent quality including automated LLM-based scoring of answer correctness confidence calibration and conversation-level resolution signals
3. Develop novel metrics that capture whether advertisers actually got the help they needed (beyond surface-level deflection rates)
4. Build and improve retrieval and generation models that power real-time advertiser interactions under strict latency SLAs
5. Drive multilingual science improve non-English resolution rates through cross-lingual retrieval translation quality modeling and locale-aware evaluation
6. Partner with product engineering and business teams to productize research and inform roadmap decisions with data

A day in the life
You might start the morning reviewing overnight evaluation results from your LLM-as-a-judge pipeline then jump into a whiteboard session designing a new resolution metric that captures whether advertisers actually unblocked their campaign. After lunch youre running offline experiments on a cross-lingual retrieval model to close the quality gap for non-English markets and by end of day youre syncing with engineering on latency trade-offs for next weeks A/B test. The constant: your science directly changes the experience millions of advertisers have when they need help.

About the team
This role sits within Amazon Advertisings broader Agentic Intelligence organization a community of multiple Applied Science and Engineering teams building the next generation of AI-powered experiences for advertisers. Youll have access to Principal Engineers and Principal Applied Scientists to pressure-test ideas and elevate your work.

What makes this team unique is the balance: youll drive product-facing science through customer support agents that touch millions of advertisers while also influencing and collaborating with a core AI infrastructure team within Amazon. The team is cross-functional scientists and engineers work shoulder-to-shoulder from problem framing through production deployment.

- 3 years of building machine learning models for business application experience
- PhD or Masters degree and 6 years of applied research experience
- Experience programming in Java C Python or related language
- Experience with neural deep learning methods and machine learning

- Experience with modeling tools such as R scikit-learn Spark MLLib MxNet Tensorflow numpy scipy etc.
- Experience with large scale distributed systems such as Hadoop Spark etc.
- Experience in online or digital advertising

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status disability or other legally protected status.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process including support for the interview or onboarding process please visit for more information. If the country/region youre applying in isnt listed please contact your Recruiting Partner.

The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience qualifications and location. Amazon also offers comprehensive benefits including health insurance (medical dental vision prescription Basic Life & AD&D insurance and option for Supplemental life plans EAP Mental Health Support Medical Advice Line Flexible Spending Accounts Adoption and Surrogacy Reimbursement coverage) 401(k) matching paid time off and parental leave. Learn more about our benefits at WA SEATTLE - 167100.00 - 226100.00 USD annually