Enter a job title or keyword

Senior Automation Test Engineer

Our Client

Job Location:

Birmingham, MI - USA

Yearly Salary: USD 125000 - 150000

Posted: 8 September 2026 (14 hours ago)

Application Deadline: 6 December 2026

Vacancies: 1 Vacancy

Job Summary

Our client is a fast-growing healthcare technology company building AI voice agents that answer every call for physician practices scheduling appointments routing patients and resolving routine requests around the clock. They are looking for a Senior Automation Test Engineer who can plug in and build rapidly owning the test strategy and automation infrastructure for their voice AI platform and building the frameworks that make non-deterministic AI behavior measurable and regressions obvious.

Responsibilities

Design build and maintain automated test infrastructure across unit integration end-to-end and load/performance layers for a real-time voice AI platform
Build evaluation and regression harnesses that score conversation outcomes and intent fulfillment for non-deterministic LLM-driven agent behavior
Generate synthetic conversation data and de-identified patient scenarios to expand test coverage for rare edge cases
Develop test frameworks and tooling that enable the broader engineering team to write and run reliable tests efficiently
Define and enforce quality standards and testing best practices across engineering teams
Build simulation and regression frameworks for real-time voice and conversational AI scenarios including heavy rules-based logic layered on top of nondeterministic AI behavior
Own test environments and CI/CD pipeline integration for automated test execution
Investigate and triage production quality issues driving root cause analysis and prevention
Develop testing approaches that satisfy HIPAA and healthcare compliance requirements
Ramp quickly: within the first 30 days understand the tech stack and business and begin shaping a test suite strategy for the platform

Skills/Requirements

10 years in software quality engineering with a strong focus on test automation
Proven experience designing and owning test automation frameworks from the ground up
Proficiency in Python TypeScript or equivalent with strong scripting and tooling skills
Experience with CI/CD integration (GitHub Actions Jenkins or equivalent)
Hands-on AI evaluation experience testing AI-powered applications including prompt and model regression testing
Demonstrated ability to test non-deterministic conversational flows designing assertions and evaluation criteria that tolerate variation without masking real regressions
Experience generating synthetic data at scale realistic conversation transcripts edge-case inputs and de-identified patient scenarios
Ability to move quickly and independently in a fast-scaling environment with minimal ramp-up time
Nice to have: experience with real-time systems telephony or voice platforms; LLM evaluation tooling and prompt/model versioning; evaluating speech-to-text or text-to-speech quality; load/performance testing for high-throughput systems; HIPAA-regulated environment experience; contract testing or service virtualization