Senior Software Engineer, AI Evals

About Sentry

Software runs the world and the pace is faster than ever. Sentry helps developers fix errors and performance issues before users notice so teams can spend less time firefighting and more time building.

Trusted by 200000 organizations Sentry is todays application monitoring standard and our team is building its AI-native future.

About the role

As a Senior Software Engineer on Sentrys AI/ML team youll be responsible for building the evaluation infrastructure that measures the accuracy reliability and real-world performance of our AI systems. This role is critical to ensuring that our debugging agents and AI-powered features behave correctly safely and predictably as they scale. Youll design datasets benchmarks and test harnesses that turn ambiguous AI behavior into measurable signals helping the team ship AI with confidence.

In this role you will

• Design and build robust evaluation frameworks to measure accuracy reliability regressions and edge cases in AI systems

• Create and curate high-quality datasets golden test cases and benchmarks grounded in real production data

• Build automated test harnesses and metrics pipelines to continuously evaluate models prompts and agentic workflows

• Partner closely with applied AI engineers and product leaders to define what good looks like and translate it into measurable criteria

• Own the evaluation lifecycle for major AI initiatives from early experimentation through production monitoring

Youll love this job if you

• Care deeply about correctness rigor and measurement in AI systems

• Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics

• Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team

• Thrive in cross-functional environments and enjoy influencing model design through better evaluation

Qualifications

• Minimum 5 years of professional experience with a Bachelors degree in computer science machine learning or a related field

• Experience building testing evaluation or data infrastructure for complex systems (AI/ML experience strongly preferred)

• Comfort writing production-quality code (we use Python and TypeScript)

• Experience working with structured and unstructured datasets labeling workflows or data quality pipelines

• Familiarity with modern ML systems and evaluation techniques (e.g. offline metrics online evaluation regression testing for models or prompts)

• Bonus: experience evaluating LLMs agentic systems or AI-assisted developer tools

The base salary range (or hourly wage range if applicable) that Sentry reasonably expects to pay for this position is $240000 to $280000 USD. A successful candidates actual base salary (or hourly wage) amount will be determined by a variety of relevant factors including without limitation the candidates work location education work and other relevant experience skills and job-related knowledge. A successful candidate will be eligible to participate in Sentrys employee benefit plans/programs applicable to the candidates position (including incentive compensation equity grants paid time off and group health insurance coverage). See Sentry Benefits for more details about the Companys benefit plans/programs.

Equal Opportunity at Sentry

Sentry is committed to providing equal employment opportunities to its employees and candidates for employment regardless of race color ancestry religion sex national origin sexual orientation age citizenship marital status disability gender identity veteran status or other legally-protected characteristic. This commitment includes the provision of reasonable accommodations to employees and candidates for employment with physical or mental disabilities who require such accommodations in order to (a) perform the essential functions of their jobs or (b) seek employment with Sentry. We strive to build a diverse team with an inclusive culture where every teammate can thrive. Sentry is an open-source company because we believe that everyone everywhere should have the ability and tools to make great software. Software should be accessible. That starts with making our industry accessible.

If you need assistance or an accommodation due to a disability you may contact us at .

Want to learn more about how Sentry handles applicant data Get the details in our Applicant Privacy Policy.

Required Experience:

Senior IC