Benchmarking Project Lead, Siri Evaluation

Join the team redefining what a deeply personal and integrated assistant can be. As part of the Siri organization you will help shape one of the worlds most widely used AI assistants powered by our next-generation of Apple Intelligence with capabilities like personal context understanding and on-screen awareness built with privacy from the ground up. Your work will have direct meaningful impact for users across iOS iPadOS macOS watchOS and visionOS. This is a rare opportunity to build at the intersection of cutting-edge AI and human-centered design shipping technology that is centered around users and their needs. nnEvaluation is at the heart of how we build our product. As Siri AI becomes more and more powerful and offers ever richer experiences to our users our evaluations have to keep pace. We are seeking a senior manager to help drive our evaluation efforts. The role will involve managing teams working on evaluation development and data science and leading high-impact initiatives to bring state-of-the-art agentic evaluation to the whole Siri team.n

As part of the work on next generation Siri we are developing novel measurements of its quality. To ensure that the evaluation systems we are building are reliable we plan benchmarking their accuracy on a wide range of features locales and platforms using humans in the loop.

Design and execute efficient data collection processes using humans in the loop nLead annotation efforts for various languages and devicesnPlan and manage the budget of annotator resourcing with Finance Annotation Ops and International team and improve the quality of the human judgements through revision of annotator training materials clear annotation questions annotator training and efficient reviewing mechanismsnCollaborate with other engineering teams to design and build a tooling ecosystem for managing and browsing rich datasets n

Agentic Coding proficiency to achieve data-science data collection and visualisation tasksnGood understanding of metrics crowd science data collection annotation analysis statisticsnAbility to work independently and cross-functionally to integrate in partner team reporting systems and pipelinesnExcellent communication skills and the ability to thrive in a highly collaborative work environment

Attunement to computational linguistics language quality human in the loop evaluationnGood engineering practices to create sustainable and easy to use data management pipelinesnPython experience and other tools for data collection and visualisation