QA Developer, AI Assistant

Role overview

Our team develops the AI assistant built into Poka. Workers and supervisors use it every day on manufacturing floors. We build it with prompts, tools, information retrieval and agents.

As a quality assurance (QA) specialist on this team, you make sure what we ship works: the assistant's answers and the features around it. You do it with automation, AI and tools you build, not with manual testing. You work with the devs, the product team and the design team to define what "good" means, then you build the evals, automated tests and tools that check it.

How this is different from traditional QA

• No manual test passes. If something needs to be checked more than once, you automate it, with code, AI or whatever tool fits.
• A large part of what you test has no single answer. The same question can get 50 good answers phrased differently, and the assistant can answer differently from one run to the next. You help define what "good" means and measure it.
• You write Python every week.

If you're looking for a role focused on test plans and manual regression testing, this isn't it.

What you'll be responsible for

• Automating QA for the team's features and building the tools that make it easier, including ways for non-developers to add evals and analyze the results.
• Building and maintaining eval suites based on real usage: actual user questions, expected behavior and the edge cases that show up in production.
• Defining with the team what a good answer looks like for our users, writing scoring rubrics and using LLM-as-judge where it makes sense, calibrated against human-labeled samples.
• Running regression checks in CI every time we change a prompt, a tool, retrieval or the Claude model version, covering answer quality, cost and speed. You build the guardrails, evals and tests that tell us whether a release gets the green light or not.
• Reviewing real conversations and grouping failures into patterns: made-up answers, wrong tool calls, missed context, inappropriate tone. Then working with developers to turn them into fixes.
• Contributing to adversarial testing: prompt injection, data leaking between customers, getting the assistant to do things it shouldn't.
• Doing timeboxed exploratory testing on features that change what the assistant can decide or do, like new agents, tools or actions, including checking that actions stay safe when something fails or gets retried.
• Joining feature kickoffs to help define acceptance criteria, including AI-specific ones.

What you bring

• AEC, DEC or BAC in a field related to the position, or equivalent experience
• 2 to 4 years of relevant experience
• A background in test automation or as an SDET
• You're comfortable with several programming languages, including Python and TypeScript
• Real curiosity about how LLMs behave and why they fail
• You're comfortable judging quality when there's no single answer, and explaining your reasoning
• Curiosity about our users' world. You can't judge an answer without understanding what a supervisor on a manufacturing floor actually needs.

Assets

• You've helped set up an eval harness, even in a personal project
• Experience using AI tools to speed up testing
• Basic statistics: pass rates, variance, sample size
• Experience in security or adversarial testing
• Knowledge of the test pyramid and the shift-left approach
• B2B experience or experience on industrial platforms
• Knowledge of Jira

The technologies you'll use

• Python
• TypeScript
• Claude Code
• Eval framework (promptfoo)
• AWS
• Docker
• And many others

We strongly believe that a passion for new technologies, a positive attitude and real team spirit are key to succeeding at Poka.

Above all, we're looking for someone curious and resourceful. You care about the success of the product and your team. You like complex problems, especially the ones without an obvious solution. You want to write clean, high-quality code and keep learning.

If that sounds like you, you're probably the person we're looking for!