Intermediate AI Quality Engineer Agentic Systems

THE OPPORTUNITY THAT AWAITS YOU

Werepartnering with a global healthtech scaleup with roots in South Africa delivering agentic AI systems into real production healthcare environments in the is notan experimental role. These systems are live scaling and solving real clinical workflows now.
Were looking for a sharp AI Quality Engineer to own one of the hardest problems in applied AI: How do you make autonomous systems reliable enough for healthcare

Youlldesign the quality engine behind production AI systems ensuring outputs areaccurate safe measurable and trusted by clinicians. Ifyoureexcited by high-stakes AI complex systems and real-world impact this is the seat.

WHAT YOU WILL BE BUILDING (AND OWN)

• Prompt lifecycle systems version control testing and structured output enforcement

• Golden datasets and evaluation frameworks for LLM performance

• Automated regression detection across prompts models and workflows

• Production monitoring observability tracing and root-cause analysis using Phoenix /OpenTelemetry

• Human-in-the-loop review pipelines labelling workflows and feedback systems

• Statistical quality measurement to continuously improve system performance

OUR REQUIRED EXPERTISE

• 35 years software engineering experience in production environments

• Exposure to AI / LLM systems deployed beyond experimentation

• 2 years in QA evaluation reliability or testing of AI systems

• Strong Python skills (FastAPI/ async preferred)

• Experience with APIs microservices and distributed systems

• Monitoring and observability experience in modern production environments

• Strong analytical thinking with a focus on measurable quality

• Masters Degree or PhD in Computer Science Engineering Artificial Intelligence Data Science or a related technical field

YOUR REWARD

• Work at the intersection of AI and healthcare where reliability truly matters

• Youllhelp build systems clinicians and healthcare operators can trust with direct impact in the US market

• Join a Greenfields environment with real ownership complex technical challenges and the chance to solve problems most companies have not figured out yet

• Competitive salary Perks

• Hybrid/flexible working based out of Stellenbosch Western Cape

For more roles please have a look at our website (). or follow us on LinkedIn ( Experience:

Unclear Seniority