AIML Sr Machine Learning Platform Engineer, Evaluation

We are seeking a highly skilled and experienced machine learning engineer to join AIML Evaluation to build the systems that evaluate and refine Apples foundation models and agents. nAs a key member of the team you will help design and develop benchmarks evaluators simulation environments and prompt and context optimization pipelines that drive quality improvements across Apples AI experiences. nYou will collaborate with product teams and the foundation model team to close the loop between observation and improvement contributing datasets environments and reward signals that drive model and agent quality.

Most model evaluation stops at a scorecard. The failing traces get thrown away even though theyre the most relevant training signal we have. Our team puts them to work. We evaluate across offline benchmarks device-in-the-loop simulation and on-device observation in production and we turn what we find into persona-grounded datasets and reward signals for foundation model this role you will build the systems that connect failure analysis to measurable model and agent improvements across a broad range of AI products at Apple. The path from finding a problem to fixing it is short: the loop you build finds the failure produces the data and checks the result. The work reaches millions of users.

Designing and building evaluation infrastructure for agents and foundation technical direction for the evaluation system and bringing partner teams LLM judges reward models calibrated against human prompt and context optimization and integrating simulation environments for agent evaluation and trajectory-based data with product and research teams to identify prioritize and address quality shared foundations such as run lineage a run registry and MCP interfaces for client teams agents.

Strong background in machine learning and distributed proven track record of solving ambiguous problems end to end: defining the problem and the metric building the solution and showing measured leading technical work across multiple codebases teams and building and maintaining ML infrastructure for evaluation training or deployment.n 6 years of professional experience as a software engineer preferably in machine learning or a related field. Bachelors or Masters degree in Computer Science or a related field or equivalent practical experience.

Experience with LLM evaluation LLM-as-judge or reward with prompt optimization agent harness development or post-training (SFT DPO RLHF).nProficiency in Python and ML frameworks such as PyTorch or with agentic systems simulation environments or trajectory-based data with on-device or privacy-preserving of how training data is used in post-training and how data choices affect model communication skills.