Enter a job title or keyword
AI Quality Engineer
Job Location:
Job Location:
Merrifield, VA - USA
Monthly Salary:
Not provided by the employer
Monthly Salary:
Posted:
8 September 2026 (9 hours ago)
Posted:
Application Deadline:
6 December 2026
Application Deadline:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
Title: AI Quality Engineer
Location: Hybrid (Vienna VA) or Remote
Duration: Initial 6 months plus extensions
Work Requirement: USC GC or Authorized to work in US
Job Description
One of our Financial client is seeking an experienced AI Quality Engineer to establish and execute the quality validation certification and production readiness processes for AI-powered SDLC and enterprise automation solutions.
As *** continues to expand its AI-enabled capabilities we require an independent validation function responsible for ensuring these solutions are accurate reliable secure explainable compliant and production ready for Production deployment.
This is not a traditional manual testing role. It is a specialized engineering position focused on validating AI systems Retrieval-Augmented Generation (RAG) architectures agent workflows orchestration frameworks observability platforms security controls and production readiness requirements. The successful candidate will serve as an independent reviewer and certification authority responsible for evaluating the quality and effectiveness of AI-generated outputs and agent behavior.
The AI Quality Engineer will work closely with AI Engineers Platform Engineers Architects Quality Engineering teams Security Risk and Product Owners to develop repeatable evaluation frameworks testing methodologies benchmarking standards and governance processes for enterprise AI solutions. This individual will play a critical role in ensuring AI solutions meet enterprise expectations for accuracy transparency auditability and operational excellence before production deployment.
In addition to validating AI-powered solutions this role will serve as the independent quality authority for ***s Internal Developer Portal (IDP) and Developer Experience (DevEx) platform initiatives. The successful candidate will evaluate and certify new portal capabilities self-service workflows automation features developer onboarding experiences and platform enhancements to ensure they meet usability reliability operational readiness governance and business value expectations before release to engineering teams.
Key Responsibilities:
Define and implement AI certification validation and production readiness standards for enterprise AI agents and intelligent automation solutions.
Develop and execute evaluation frameworks that measure:
* Accuracy
* Relevance
* Groundedness
* Completeness
* Hallucination rates
* Retrieval quality
* Recommendation quality
* User satisfaction
Design and maintain AI validation datasets benchmark scenarios regression test suites and golden test datasets using platforms such as LangSmith and Azure AI Foundry.
Validate Retrieval-Augmented Generation (RAG) solutions built using Azure AI Search Azure AI Foundry LangChain LangGraph and enterprise knowledge repositories.
Perform architecture reviews and quality assessments for AI solutions deployed on Azure Container Apps Azure Functions Azure Databricks Azure SQL Cosmos DB and related cloud-native services.
Evaluate AI orchestration workflows including prompt execution agent routing tool calling guardrails retrieval pipelines human-in-the-loop processes and MCP integrations.
Validate security governance auditability and compliance controls including Entra ID integration RBAC managed identities Key Vault integrations data protection policies and sensitive data handling requirements.
Create production readiness and certification checklists covering observability monitoring logging resiliency supportability recoverability and operational readiness.
Analyze LangSmith traces agent execution logs AI evaluation metrics and telemetry to identify quality concerns and improvement opportunities.
Partner with development teams to identify and remediate quality accuracy security and performance issues prior to production deployment.
Create and maintain AI certification reports scorecards dashboards and executive summaries for leadership and governance review boards.
Drive continuous improvement of AI validation methodologies testing strategies and model evaluation frameworks across the organization.
Establish independent quality gates that AI solutions must satisfy before production approval.
Lead certification and production readiness reviews for Internal Developer Portal (IDP) capabilities self-service workflows engineering automation and developer experience improvements.
Required Qualifications:
Bachelors Degree in Computer Science Software Engineering Data Science Artificial Intelligence Information Systems or a related technical field.
5 years of software quality engineering test architecture platform engineering software development machine learning engineering or related experience.
2 years of experience working directly with Generative AI Large Language Models (LLMs) RAG architectures AI agents or AI-powered business applications.
Strong understanding of AI system evaluation methodologies including accuracy testing hallucination detection groundedness validation and AI quality measurement.
Experience with one or more of the following:
* Azure AI Foundry
* Azure OpenAI
* LangChain
* LangGraph
* LangSmith
* AI agent frameworks
* RAG architectures
Experience with cloud-native application architectures and Azure services including Container Apps Functions Cosmos DB Azure AI Search Azure SQL Key Vault and Databricks.
Experience evaluating or supporting developer platforms internal developer portals DevOps platforms CI/CD pipelines platform engineering initiatives or self-service engineering capabilities.
Experience designing automated test strategies quality frameworks regression testing and certification processes.
Strong understanding of DevSecOps CI/CD observability monitoring and enterprise software development practices.
Experience evaluating APIs microservices integrations and distributed systems.
Strong analytical and troubleshooting skills with the ability to investigate complex technical issues.
Excellent documentation communication and stakeholder engagement skills.
Ability to operate independently and provide objective unbiased quality assessments.
Preferred Qualifications:
Experience building or validating enterprise AI agents and multi-agent systems.
Experience within Platform Engineering Developer Experience (DevEx) Site Reliability Engineering (SRE) or DevOps organizations.
Experience with AI observability tracing and evaluation platforms such as LangSmith.
Experience validating Azure AI Search implementations vector databases semantic search embeddings and knowledge retrieval solutions.
Experience implementing AI governance Responsible AI model risk management or AI compliance frameworks.
Experience in financial services banking insurance healthcare or other regulated industries.
Familiarity with Azure DevOps GitHub GitHub MCP Azure DevOps MCP enterprise developer platforms and SDLC tooling.
Experience with performance engineering chaos testing resilience testing and production readiness reviews.
Knowledge of identity and access management concepts including Entra ID managed identities Key Vault integration and RBAC controls.
Experience creating quality scorecards executive dashboards KPI frameworks and reporting metrics.
Experience in Quality Engineering Test Architecture Security Engineering Platform Engineering or AI Engineering roles.
Ideal Candidate Profile:
A highly technical engineer who understands both modern software engineering and AI systems. This individual can independently assess whether an AI solution is accurate secure explainable compliant observable and ready for production. They serve as the organizations independent quality authority for AI-powered solutions and help establish confidence in enterprise AI deployments before release to production.
About INSPYR Solutions:
Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions we strategically align industry and technical expertise with our clients business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services project and talent solutions. By always striving for excellence and focusing on the human aspect of our business we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at .
Location: Hybrid (Vienna VA) or Remote
Duration: Initial 6 months plus extensions
Work Requirement: USC GC or Authorized to work in US
Job Description
One of our Financial client is seeking an experienced AI Quality Engineer to establish and execute the quality validation certification and production readiness processes for AI-powered SDLC and enterprise automation solutions.
As *** continues to expand its AI-enabled capabilities we require an independent validation function responsible for ensuring these solutions are accurate reliable secure explainable compliant and production ready for Production deployment.
This is not a traditional manual testing role. It is a specialized engineering position focused on validating AI systems Retrieval-Augmented Generation (RAG) architectures agent workflows orchestration frameworks observability platforms security controls and production readiness requirements. The successful candidate will serve as an independent reviewer and certification authority responsible for evaluating the quality and effectiveness of AI-generated outputs and agent behavior.
The AI Quality Engineer will work closely with AI Engineers Platform Engineers Architects Quality Engineering teams Security Risk and Product Owners to develop repeatable evaluation frameworks testing methodologies benchmarking standards and governance processes for enterprise AI solutions. This individual will play a critical role in ensuring AI solutions meet enterprise expectations for accuracy transparency auditability and operational excellence before production deployment.
In addition to validating AI-powered solutions this role will serve as the independent quality authority for ***s Internal Developer Portal (IDP) and Developer Experience (DevEx) platform initiatives. The successful candidate will evaluate and certify new portal capabilities self-service workflows automation features developer onboarding experiences and platform enhancements to ensure they meet usability reliability operational readiness governance and business value expectations before release to engineering teams.
Key Responsibilities:
Define and implement AI certification validation and production readiness standards for enterprise AI agents and intelligent automation solutions.
Develop and execute evaluation frameworks that measure:
* Accuracy
* Relevance
* Groundedness
* Completeness
* Hallucination rates
* Retrieval quality
* Recommendation quality
* User satisfaction
Design and maintain AI validation datasets benchmark scenarios regression test suites and golden test datasets using platforms such as LangSmith and Azure AI Foundry.
Validate Retrieval-Augmented Generation (RAG) solutions built using Azure AI Search Azure AI Foundry LangChain LangGraph and enterprise knowledge repositories.
Perform architecture reviews and quality assessments for AI solutions deployed on Azure Container Apps Azure Functions Azure Databricks Azure SQL Cosmos DB and related cloud-native services.
Evaluate AI orchestration workflows including prompt execution agent routing tool calling guardrails retrieval pipelines human-in-the-loop processes and MCP integrations.
Validate security governance auditability and compliance controls including Entra ID integration RBAC managed identities Key Vault integrations data protection policies and sensitive data handling requirements.
Create production readiness and certification checklists covering observability monitoring logging resiliency supportability recoverability and operational readiness.
Analyze LangSmith traces agent execution logs AI evaluation metrics and telemetry to identify quality concerns and improvement opportunities.
Partner with development teams to identify and remediate quality accuracy security and performance issues prior to production deployment.
Create and maintain AI certification reports scorecards dashboards and executive summaries for leadership and governance review boards.
Drive continuous improvement of AI validation methodologies testing strategies and model evaluation frameworks across the organization.
Establish independent quality gates that AI solutions must satisfy before production approval.
Lead certification and production readiness reviews for Internal Developer Portal (IDP) capabilities self-service workflows engineering automation and developer experience improvements.
Required Qualifications:
Bachelors Degree in Computer Science Software Engineering Data Science Artificial Intelligence Information Systems or a related technical field.
5 years of software quality engineering test architecture platform engineering software development machine learning engineering or related experience.
2 years of experience working directly with Generative AI Large Language Models (LLMs) RAG architectures AI agents or AI-powered business applications.
Strong understanding of AI system evaluation methodologies including accuracy testing hallucination detection groundedness validation and AI quality measurement.
Experience with one or more of the following:
* Azure AI Foundry
* Azure OpenAI
* LangChain
* LangGraph
* LangSmith
* AI agent frameworks
* RAG architectures
Experience with cloud-native application architectures and Azure services including Container Apps Functions Cosmos DB Azure AI Search Azure SQL Key Vault and Databricks.
Experience evaluating or supporting developer platforms internal developer portals DevOps platforms CI/CD pipelines platform engineering initiatives or self-service engineering capabilities.
Experience designing automated test strategies quality frameworks regression testing and certification processes.
Strong understanding of DevSecOps CI/CD observability monitoring and enterprise software development practices.
Experience evaluating APIs microservices integrations and distributed systems.
Strong analytical and troubleshooting skills with the ability to investigate complex technical issues.
Excellent documentation communication and stakeholder engagement skills.
Ability to operate independently and provide objective unbiased quality assessments.
Preferred Qualifications:
Experience building or validating enterprise AI agents and multi-agent systems.
Experience within Platform Engineering Developer Experience (DevEx) Site Reliability Engineering (SRE) or DevOps organizations.
Experience with AI observability tracing and evaluation platforms such as LangSmith.
Experience validating Azure AI Search implementations vector databases semantic search embeddings and knowledge retrieval solutions.
Experience implementing AI governance Responsible AI model risk management or AI compliance frameworks.
Experience in financial services banking insurance healthcare or other regulated industries.
Familiarity with Azure DevOps GitHub GitHub MCP Azure DevOps MCP enterprise developer platforms and SDLC tooling.
Experience with performance engineering chaos testing resilience testing and production readiness reviews.
Knowledge of identity and access management concepts including Entra ID managed identities Key Vault integration and RBAC controls.
Experience creating quality scorecards executive dashboards KPI frameworks and reporting metrics.
Experience in Quality Engineering Test Architecture Security Engineering Platform Engineering or AI Engineering roles.
Ideal Candidate Profile:
A highly technical engineer who understands both modern software engineering and AI systems. This individual can independently assess whether an AI solution is accurate secure explainable compliant observable and ready for production. They serve as the organizations independent quality authority for AI-powered solutions and help establish confidence in enterprise AI deployments before release to production.
About INSPYR Solutions:
Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions we strategically align industry and technical expertise with our clients business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services project and talent solutions. By always striving for excellence and focusing on the human aspect of our business we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at .
INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race color religion sex national origin age disability genetics or any other protected status. INSPYR Solutions complies with all applicable laws governing nondiscrimination in employment in every location in which the company has facilities. Applicants requiring reasonable accommodation during the application or interview process should contact for assistance.
Information collected and processed through your application with INSPYR Solutions (including any job applications you choose to submit) is subject to INSPYR Solutions Privacy Policy and INSPYR Solutions AI and Automated Employment Decision Tool Policy: By submitting an application you are consenting to being contacted by INSPYR Solutions through phone email or text.
Required Experience:
IC
