Software Engineer in Test - AI

O.C. Tanner is the global leader in software and services that improve workplace culture through meaningful employee experiences. Our Culture Cloud is a suite of apps designed to enhance the employee experience with strategic recognition, service awards, wellbeing, leadership, and events that help people thrive at work. Our Culture by Design approach provides expert services to organizations looking to create great workplaces.

Our global team of 1,500 people hail from 58 countries and speak 62 languages. As programmers, researchers, designers, client professionals and craftspeople we create the tech, tools and awards that connect employees to purpose at thousands of companies. Join us as we help people all over the world thrive at work.

Location: Salt Lake City, UT

In this role, you will build automation, evaluate AI-driven behavior, validate responsible AI controls, and identify risks early so our products are reliable, scalable, production-ready, and trusted by millions of users.

Key Responsibilities

• Become a Subject Matter Expert on the AI platform and possess a deep understanding of system interactions, upstream/downstream dependencies, and data flows.
• Design, develop, and maintain automated test frameworks for AI platform services, APIs, agents, prompt-based workflows, and RAG-enabled applications.
• Develop automated AI evaluation suites and regression benchmarks that continuously measure model behavior and detect quality degradation before release.
• Build and execute functional, integration, end-to-end, regression, performance, and reliability tests for AI-driven systems and services.
• Define evaluation strategies, quality metrics, and acceptance criteria for AI-generated outputs, including accuracy, relevance, consistency, grounding, safety, and business value.
• Validate responsible AI controls, permissions, guardrails, data handling practices, and business rules that protect customer trust.
• Partner with Engineering, Product, and Support throughout the software lifecycle to drive risk-based testing strategies, influence quality to ensure production readiness.
• Integrate automated testing, AI evaluations, and release validation into CI/CD pipelines.
• Investigate, triage, and communicate defects, quality issues, and production incidents, driving root cause analysis and continuous improvement.
• Utilize observability tools, logs, metrics, and traces to investigate production issues and perform root cause analysis.
• Establish and promote testing standards, automation patterns, and AI quality practices that improve reliability and delivery speed.

Required Qualifications

• 3+ years of experience in software testing, quality engineering, test automation, or software development.
• Hands-on experience developing automated tests, testing tools, or quality frameworks using Python, Selenium and Playwright.
• Experience testing APIs, microservices, distributed systems, backend services, or event-driven architectures.
• Strong experience testing AI-enabled applications using technologies such as LLMs, LangChain, LangGraph, or similar platforms.
• Hands-on experience evaluating AI-generated outputs using datasets, scoring rubrics, golden test sets, benchmarking frameworks, or automated quality checks.
• Experience testing RAG systems, including retrieval quality, embeddings, grounding, citations, context accuracy, and streaming responses.
• Experience working with SQL and NoSQL technologies, including relational, document, key-value, or vector databases.
• Experience integrating automated testing into CI/CD pipelines and modern software delivery practices.
• Experience investigating production issues and using incident and defect data to improve system quality and reliability.
• Strong understanding of test automation, performance testing, risk-based testing, release validation, and responsible AI principles.
• Excellent collaboration and communication skills, with the ability to clearly articulate quality risks and acceptance criteria.

Preferred Qualifications

• Experience with AWS, Kubernetes, Docker, and cloud-native architectures.
• Experience testing event-driven systems using Kafka or similar messaging platforms.
• Familiarity with application security testing and OWASP Top 10 principles.
• Proficiency in Python and at least one additional programming language such as Java, Ruby, or Go.
• Experience with reliability engineering practices, including observability, production readiness reviews, incident analysis, SLOs, and continuous improvement.
• Experience conducting performance, load, stress, scalability, or resilience testing.