Enter a job title or keyword

Ai QA Engineer (Agentic & Generative)

Tekfortune

Job Location:

Dallas, IA - USA

Monthly Salary: Not provided by the employer

Posted: 29 September 2026 (3 days ago)

Application Deadline: 27 December 2026

Vacancies: 1 Vacancy

Job Summary

Tekfortune is a fast-growing consulting firm specialized in permanent contract & project-based staffing services for worlds leading organizations in a broad range of this quickly changing economic landscape virtual recruiting and remote work are critical for the future of work. To support the active project demands and skills gaps our staffing experts can help you find the best job for you.

Role: AI QA ENGINEER (AGENTIC & GENERATIVE)
Location: Dallas TX Hybrid Couple of days in a month
Duration: 6 Month
Required Skills: AI
Job Description:
We are seeking a hands-on AI Engineer to design and execute end-to-end testing strategies for agentic AI solutions including multi-agent systems in production-grade environments. This role partners with the Agentic Operations Team to ensure resiliency reliability accuracy latency orchestration correctness and scale. You will establish QA frameworks build reusable test artifacts drive macro-level validations across complex workflows and lead the QA function for Agentic AI from Dev to Prod.
Key Responsibilities

Quality Strategy & Leadership
Agentic & MultiAgent Testing
Reliability Resiliency and Latency
Accuracy & Macro-Level Validations
Scale & Orchestration
Dev Prod Readiness
Define and own the QA strategy for agentic/multi-agent AI systems across dev staging and prod.
Mentor a team of QA engineers; establish testing standards coding guidelines for test harnesses and review practices.
Partner with Agentic Operations Data Science MLOps and Platform teams to embed QA in the SDLC and incident response.
Design tests for agent orchestration tool calling planner-executor loops and inter-agent coordination (e.g. task decomposition handoff integrity and convergence to goals).
Validate state management context windows memory/knowledge stores and prompt/graph correctness under varying conditions.
Implement scenario fuzzing (e.g. adversarial inputs prompt perturbations tool latency spikes degraded APIs).
Create resilience testing suites: chaos experiments failover retries/backoff circuit-breaking and degraded mode behavior.
Establish latency SLOs and measure end-to-end response times across orchestration layers (LLM calls tool invocations queues).
Ensure reliability through soak tests canary verifications and automated rollbacks.
Define ground-truth and reference pipelines for task accuracy (exact match semantic similarity factuality checks).
Build macro validation frameworks that validate task outcomes across multi-step agent workflows (e.g. complex data pipelines content generation verification agent loops).
Instrument guardrail validations (toxicity PII hallucination policy compliance).
Design load/stress tests for multi-agent graphs under scale (concurrency throughput queue depth backpressure).
Validate orchestrator correctness (DAG execution retries branching timeouts compensation paths).
Engineer reusable test artifacts (scenario configs synthetic datasets prompt libraries agent graph fixtures simulators).
Integrate tests into CI/CD (pre-merge gates nightly canary) and production monitoring with alerting tied to KPIs.
Define release criteria and run operational readiness (performance security compliance cost/latency budgets).
Build post-deployment validation playbooks and incident triage runbooks.

Required Qualifications

7 years in Software QA/Testing with 2 years in AI/ML or LLM-based systems; hands-on experience testing agentic/multi-agent architectures.
Strong programming skills in Python or TypeScript/JavaScript; experience building test harnesses simulators and fixtures.
Experience with LLM evaluation (exact/soft match BLEU/ROUGE BERTScore semantic similarity via embeddings) guardrails and prompt testing.
Expertise in distributed systems testing latency profiling resiliency patterns (circuit breakers retries) chaos engineering and message queues.
Familiarity with orchestration frameworks (LangChain LangGraph LlamaIndex DSPy OpenAI Assistants/Actions Azure OpenAI orchestration or similar).
Proficiency with CI/CD (GitHub Actions/Azure DevOps) observability (OpenTelemetry Prometheus/Grafana Datadog) and feature flags/canaries.
Solid understanding of privacy/security/compliance in AI systems (PII handling content policies model safety).
Excellent communication and leadership skills; proven ability to work cross-functionally with Ops Data and Engineering.

Preferred Qualifications

Experience with multi-agent simulators agent graph testing and tooling latency emulation.
Knowledge of MLOps (model versioning datasets evaluation pipelines) and A/B experimentation for LLMs.
Background in cloud (AWS) serverless containerization and event-driven architectures.
Prior ownership of cost/latency/SLAs for AI workloads in production.