Lead AI Engineer – Agentic Test Automation

Position Overview:

JOB DESCRIPTION

1) Agentic test automation foundation (reusable patterns reference implementations)

Design and implement agentic testing patterns that can be adopted by multiple Underwriting teams (and later other domains).

Create reference implementations (sample repos / templates) demonstrating:

o Test generation assistance (from requirements APIs contracts schemas)

o Test maintenance assistance (auto-updating selectors/contracts flaky test triage)

o Failure analysis assistance (root cause suggestions log correlation defect drafting)

Establish a standard architecture for test code organization tagging data management and execution across UI API service layers.

2) Coverage standards templates and governance

• Define and publish coverage standards (what good looks like) including:

o Minimum coverage expectations by service/component

o Test type mix (unit vs API vs UI vs contract vs integration)

o Risk-based prioritization and traceability to requirements

• Provide templates usable across teams:

o Test plan templates

o Test case/spec templates (Gherkin-style or equivalent)

o Definition of Ready / Definition of Done quality checklists

• Create a scalable tagging/metadata strategy (e.g. feature service risk priority data sensitivity) to support reporting and quality gates.

3) GenAI-assisted reporting and quality insights across microservices

• Build automated reporting that aggregates test service data across multiple microservices such as:

o Test execution results (Karate/Playwright CI runs)

o Service health signals (logs/metrics/traces if available)

o Defect signals (issue tracker metadata if available)

• Generate GenAI-driven summaries:

o Release readiness narratives

o Failure clustering and trend analysis

o What changed insights (commit/PR correlation)

• Produce outputs consumable by engineering leadership and teams (dashboards markdown summaries in PRs artifacts in CI).

4) Quality gates via agents

• Build automated review agents that evaluate user stories/requirements for minimum required clarity and data before development/testing starts:

o Required fields present (acceptance criteria testable outcomes data needs dependencies)

o Ambiguity detection and missing edge cases

o Data/privacy considerations and environment needs

• Integrate gates into workflow (PR checks issue templates GitHub Actions) to reduce churn and rework.

Required Technical Skills (must-have)

GenAI / LLM agentic development

• Hands-on experience building LLM-powered agents (tool-using multi-step reasoning guardrails).
• Experience with prompting patterns structured outputs (JSON schemas) evaluation and reducing hallucinations.
• Ability to design agent workflows for:

o Test generation/augmentation

o Requirements review and completeness validation

o Report generation and summarization

GitHub platform GHCP (Copilot) for engineering workflows

• Strong proficiency with GitHub Copilot in day-to-day development.
• Deep experience with GitHub platform capabilities:

o GitHub Actions (CI/CD pipelines reusable workflows composite actions)

o PR checks branch protections CODEOWNERS templates

• • Automation via GitHub APIs/webhooks (as needed)

Test automation engineering (framework expertise)

• Advanced experience designing and implementing automation with:

o Karate (API testing contract-like checks data-driven testing mocks)

o Playwright (UI automation selectors strategy parallelization trace/video artifacts)

• Strong understanding of test design and coverage:

o Happy path scenarios

o Negative/validation scenarios

o Edge/boundary scenarios

o Data setup/teardown strategies and test isolation

Cross-service reporting and data aggregation

• Proven ability to aggregate and normalize results from multiple microservices and multiple pipelines.
• Experience producing actionable automated reports (trend analysis failure clustering service correlation).

Automated requirements review agents

• Experience implementing automated checks that validate:

o Acceptance criteria completeness

o Required test data and environment dependencies

o Non-functional requirements (performance security observability) when applicable

Deliverables / What success looks like (for the posting)

• A reusable agentic testing automation kit adopted by multiple teams.
• Published coverage standards templates and onboarding documentation.
• A working GenAI-assisted reporting pipeline aggregating results across microservices.
• Automated quality gates integrated into GitHub workflows that measurably reduce story churn.