Senior AIML Engineer Validation & Evaluation

As the Senior AI/ML Engineer - Validation & Evaluation within AI Validation & Monitoring (AVM) you will provide practice leadership for validation pathways applied evaluation and consultation. You will translate approved enterprise requirements into practical validation and evaluation methods; lead the development and maintenance of guidance for human-computer interaction (HCI) usability clinical workflow human oversight pilot HCI design legacy-product HCI remediation and revalidation; and review Validation Performance and Safety content for methodological adequacy evidence sufficiency consistency limitations and alignment with approved policy and methods.

You will lead complex but generally non-precedent-setting assessment review and consultation calibrate AVM reviewer comments assure the quality and consistency of review conclusions and revalidation guidance and coach AVM Engineers Associates and others in the department. You will recommend required corrections alternate methods additional evidence limitations fallback or remediation strategies and escalation.

• Translating approved enterprise validation and evaluation requirements into practical pathways and applied methods for pilots full implementations post-deployment changes legacy products and other governed use cases.
• Reviewing intended use pathway selection performance and safety expectations standard-of-practice comparisons acceptance thresholds evaluation criteria evidence sufficiency limitations and residual uncertainty.
• Evaluating the adequacy of retrospective studies prospective designs pilot protocols workflow simulations user acceptance testing human-factors work and alternative validation strategies.
• Reviewing test plans/protocols methods and validation datasets for representativeness traceability production parity functionality robustness calibration subgroup and equity evidence uncertainty and methodological limitations.
• Reviewing HCI clinical workflow usability human oversight automation-bias risk patient-facing behavior training accessibility guardrails task boundaries safe refusal escalation fallback and remediation strategies.
• Leading complex case consultation and resolving non-precedent-setting method questions within approved standards while escalating novel precedent-setting disputed or out-of-method questions to the Principal or Director as appropriate.
• Documenting traceable review conclusions required corrections clarification questions alternate approaches evidence gaps limitations revalidation implications consultation needs and escalation triggers for the AIA Product Lead.
• Developing and maintaining validation-pathway playbooks decision aids reviewer rubrics evidence examples standard findings consultation methods case-library content and escalation criteria.
• Leading reviewer training calibration sessions office hours case review quality assurance and coaching so AVM staff and Governance Operations apply approved methods consistently.
• Guiding remediation and revalidation approaches including when evidence gaps or changes to the model data workflow population interface guardrails autonomy training or intended use require supplemental evaluation or retesting.
• Coordinating with clinical product teams and relevant enterprise partners such as: Kern Center AVSP/CCaTS MCP Evaluate and Deploy FAST Epic DTO IT Architecture data platforms Patient Safety Clinical Informatics Legal and applicable committees when cross-functional expertise is required.
• Converting recurring review gaps and quality findings into improved guidance templates evidence examples standard assessment language training technology requirements and Governance Operations enablement without assuming ownership of the complete assessment study execution routine case management or final approval.
• Providing mentorship guidance and technical leadership to junior engineers within the AIA team.
  

• A masters degree in engineering computer science mathematics health science or a related field with 4 years of experience a bachelors degree with 6 years of experience.
• Extensive experience applying AI and machine learning in production healthcare environments or similar highly regulated or technology focused industries showcasing an understanding of healthcare technology.
• Demonstrated leadership in managing complex projects with a proven ability to navigate intricate project requirements and deliver successful outcomes.
• Proficiency in fostering collaboration across diverse teams and effectively communicating complex technical concepts to non-technical stakeholders.
• Demonstrated expertise in cloud infrastructure environment and software development tools.
• Experience working with large complex and heterogeneous data sets preferably in healthcare.
• Skilled in AI/ML techniques and frameworks.
• Familiarity with best practices in data engineering data science AI Engineering and the MLOps communities.
• Demonstrated initiative in administration education software development and technical reporting.
• A commitment to mentoring and training less-experienced team members coupled with strong interpersonal communication and time management skills.

Preferred Qualifications:

• A Ph.D. or other doctorate is preferred.
• Strong expertise in AI/ML techniques and frameworks such as deep learning natural language processing and Generative AI with proficiency in tools like Python TensorFlow PyTorch sci-kit-learn Keras etc.
• Knowledge of the healthcare domain including clinical workflows electronic health records medical terminologies regulatory requirements and industry standards.
• Familiarity with systems or quality engineering best practices regulatory standards and compliance frameworks with the ability to adapt these effectively to different project scenarios.
• Demonstrated experience leading technical/quantitative teams in a regulated environment.
• Demonstrated experience creating risk management files and verification/validation strategies for digital health technology products within the healthcare industry.
• Strong expertise in user-centered design human factors engineering usability testing methodologies and evaluation across AI product development. Ability to conduct expert reviews using established usability practices and methods. Presents findings in easy-to-understand terms for the business or clinical practice.
• Demonstrated hands-on experience using the TRex assessment application to lead validation pathway reviews for AI tools deployed in Epic ANIMATE or comparable clinical environments including HCI workflow human oversight test plan design and revalidation.