Skip to content
AI Engineer Jobs

Senior AI QA Automation Engineer

Encora

Location
Remote (Costa Rica)
Employment
Full-time
Level
Senior Level
Posted 4 days ago

About the role

Encora, an AI-native engineering services leader, is seeking a Senior AI QA Automation Engineer to build automated evaluation frameworks for AI agents and RAG solutions. The role focuses on ensuring high-quality performance through rigorous testing, CI/CD quality gates, and production monitoring of AI systems.

Skills

Python AI Testing Test Automation Playwright Selenium API Testing Azure AI Foundry RAG Evaluation Prompt Testing CI/CD Monitoring Quality Engineering Regression Testing End-to-end Automation Performance Testing

Perks

  • Remote Work

Full job details

Job Title: Senior AI QA Automation Engineer

Key Skills: Python, AI Testing, Test Automation, Playwright, Selenium, API Testing, Azure AI Foundry Evaluations, RAG Evaluation, Prompt Testing, CI/CD, Monitoring

Experience: 5+ YOE.

Location: Costa Rica, Peru, Colombia, Bolivia.

Mode: Remote

We at Coforge are hiring Senior AI QA Automation Engineer (#006fu00000bWsOCAA0-4) with the following skill set.

Key Responsibilities

  • Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.

  • Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.

  • Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.

  • Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.

Required Skills & Qualifications

  • 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.

  • Strong Python skills with experience building reusable automated testing frameworks.

  • Hands-on experience with Playwright and/or Selenium.

  • Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.

  • Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.

  • Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.

  • Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.

  • Experience implementing CI/CD quality gates and production monitoring.

  • Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria.

  • Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.

Preferred Skills:

  • Experience testing Generative AI, RAG solutions, copilots, or AI agents.

  • Experience with LLM evaluation frameworks and advanced prompt testing.

  • Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants.

  • Experience with adversarial testing and Responsible AI testing.

  • Experience monitoring AI quality, drift, reliability, and failure patterns in production environments.

Posted On: 12-10-2026

At Coforge, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.