AI Engineer, Evals & Systems Integration

What Makes Us a Great Place to Work

We are proud to be consistently recognized as one of the world’s best places to work. We are currently the top ranked consulting firm on Glassdoor’s Best Places to Work list and have earned the #1 overall spot a record seven times. Extraordinary teams are at the heart of our business strategy, but these don’t happen by chance. They require intentional focus on bringing together a broad set of backgrounds, cultures, experiences, perspectives, and skills in a supportive and inclusive work environment. We hire people with exceptional talent and create an environment in which every individual can thrive professionally and personally.

Who You’ll Work With

You will be part of a team building Bain’s integrated suite of agentic decisioning tools that gives large organizations always-on intelligence across their most important growth decisions. The role sits at an intersection few organizations can offer: the analytical depth and client access of a top-tier consultancy, the technical ambition of a product company, and engineering work at the frontier of applied AI.

You will work inside Bain’s standing product team, building software that helps the world’s largest companies answer the questions that drive their business: where to invest, where to focus, and how to win. The impact of what you build is direct and immediate, and the platform compounds in value with every release.

The platform has two layers working together: an agentic core and a set of decision modules, each solving a distinct commercial problem. The build lifecycle is connected to the real world: as case teams work alongside clients across industries and geographies, that feedback flows straight back to engineering and into the product. Every client engagement makes the platform smarter, broader, and more resilient.

This is a startup inside an established company. You get genuine early-stage ownership and real influence over architecture and product direction, backed by Bain’s global scale, client relationships, and leadership commitment; the pace of a startup with the runway of a global firm.

We are looking for engineers who see how to make the platform’s evaluations sharper and bring that instinct to every sprint without being asked, and who are comfortable with problems that do not have a known answer. The work involves evaluation frameworks for LLM-based and agentic systems, integration testing, and data problems with no off-the-shelf answer. Experience with agentic systems matters: tool and context protocols such as MCP, evaluation harnesses, and entity resolution across messy enterprise data. This is applied AI in production, at scale, with real users depending on what you build. The field is moving quickly and the team is built to move with it, so the willingness to keep learning counts for as much as what you already know: real-time decision engines, cloud-native infrastructure, and data challenges without off-the-shelf answers. This is applied AI in production, at scale, with real users depending on what you build. Native AI fluency matters here, and so does the drive to keep learning: because the field is moving fast and this team is built to move with it.

Where You’ll Fit Within the Team

The AI Engineer, Evals and Systems Integration works directly under the AI Engineering Lead, Evals and Systems Integration, contributing to the evaluation framework and integration testing infrastructure across the Growth AI Performance System platform. This is a hands-on implementation role: you will build and run the test suites, evaluation pipelines, and data quality checks that the evaluation framework depends on, working across the data ontology, Agentic Core, and platform and security layers under the direction of the Evals Lead.

You will develop broad exposure to the full Growth AI Performance System stack: from the knowledge graph and semantic layer through agentic orchestration and retrieval pipelines to the product module outputs that clients see. This cross-cutting view makes it an excellent role for an engineer who wants to develop deep expertise in how complex AI systems fail, how to measure quality systematically, and what it takes to make a production AI platform reliable enough for enterprise clients to depend on.

You will work across the platform and product modules focused on marketing ROI, brand, and pricing strategy, building the evaluation datasets, regression tests, and integration checks that validate quality at each layer and across the integration points between them.

The role has developing client proximity. As you grow into the position, you will engage with consulting teams to understand where system outputs fell short in live engagements, and contribute to the structured feedback loops that translate client experience into evaluation improvements.

What You’ll Do

Building evaluation infrastructure

• Build and maintain automated test suites and evaluation pipelines for components of the Growth AI Performance System platform (including knowledge graph outputs, agentic workflow execution, retrieval pipeline quality, and end-to-end product module outputs), under the direction of the Evals Lead
• Implement evaluation pipelines that integrate into CI/CD workflows so that quality gates run automatically before changes are deployed; ensure evaluation runs reliably and surfaces failures clearly to the engineering teams responsible for fixing them
• Build and maintain evaluation datasets, regression test cases, and scoring rubrics for specific platform components; contribute to the evaluation dataset library that enables systematic, reproducible quality measurement across the platform
• Design experiments to measure component and integration quality, detect regressions, and help diagnose the root cause of quality failures; develop statistical reasoning skills to interpret evaluation results accurately
• Contribute to the development and maintenance of evaluation documentation: test coverage maps, failure mode taxonomies, scoring convention guides, and deployment-readiness checklists that support cross-team use and client audit trails

Integration and cross-layer testing

• Build integration tests that validate the interfaces between platform layers (including knowledge graph output conformance, API contract correctness, agentic context passing, and data pipeline output structure), catching misalignment before it propagates downstream
• Work alongside the Data Ontology and Knowledge team to implement evaluation checks for entity resolution quality, taxonomy consistency, and data contract correctness; build tests that run automatically when ontology changes are proposed
• Work alongside the Agentic Core team to implement evaluation checks for multi-agent orchestration reliability, tool use correctness, and end-to-end workflow quality; contribute to failure mode analysis when agentic systems produce unexpected outputs
• Contribute to evaluation tooling that supports the Platform and Security team’s audit and compliance requirements; ensure evaluation pipelines handle sensitive client data correctly and that test outputs are logged appropriately

Stakeholder engagement and communication

• Engage with consulting teams under the direction of the Evals Lead to understand where system outputs fell short in live client engagements; support the process of tracing failures through the evaluation framework and feeding findings back to engineering teams
• Develop the communication skills needed to explain evaluation findings, quality metrics, and system behavior clearly to both technical peers and non-technical stakeholders including consulting team members
• Contribute to evaluation reports and quality documentation that give engineering and product teams the evidence they need to make deployment decisions; develop written communication habits that support cross-team use and, over time, client-facing communication
• Stay current with emerging evaluation methodologies and AI-specific risk and safety approaches; contribute to team knowledge sharing on evaluation best practices

About You

Technical

• Solid Python proficiency; experience writing and maintaining automated test suites, evaluation pipelines, and data quality checks in a production engineering environment
• Working experience with LLM behavior, failure modes, and evaluation metrics; familiarity with how prompt changes, tool definitions, and model routing decisions affect agent outputs, and how to detect those effects systematically
• Ability to design experiments that measure system quality and catch regressions; developing statistical reasoning skills to interpret evaluation results accurately and avoid over-reading noise
• Experience with integration testing or systems testing, able to design tests that validate not just individual components but the correctness of interfaces between them
• Familiarity with CI/CD pipelines and the ability to wire evaluation tests into deployment workflows so that quality gates run automatically
• Exposure to or genuine interest in knowledge graphs, ontologies, or semantic data models; willingness to develop practical understanding of how knowledge layer quality affects downstream AI system behavior
• Solid engineering fundamentals: clean, testable code; version control; automated testing; and debugging distributed systems
• Active use of AI-assisted coding tools (Cursor, Claude Code, or equivalent) to accelerate development and build evaluation tooling faster
• Awareness of responsible AI principles and AI-specific risks including prompt injection, data leakage, and agentic workflow failure modes; eagerness to develop depth in AI safety evaluation as the role matures

Experience

• 3 or more years in software engineering, data engineering, or applied AI roles with hands-on experience building and operating automated test suites or evaluation pipelines
• Track record of building test or evaluation infrastructure that is reliable enough for others to depend on, not just writing tests, but owning them and keeping them current as the systems they test evolve
• Experience working across multiple system components and integration points; comfort debugging failures that span more than one service or data layer
• Exposure to LLM evaluation, agentic system testing, or knowledge graph quality assessment is a strong advantage; genuine curiosity about how complex AI systems fail and how to measure quality systematically is equally valued
• Experience in agile, cross-functional product or consulting-adjacent environments; strong troubleshooting and root cause analysis skills

Education

Associate’s / Bachelor’s degree in computer science, engineering, mathematics, or a related technical field; or an equivalent combination of education, training, and experience

U.S. Compensation Information

Compensation for this role includes base salary, annual discretionary performance bonus, 401(k) plan with an annual employer contribution based on years of service and Bain’s best in class benefits package (details listed below).

Some local governments in the United States require a good-faith, reasonable salary range to be included in job postings for open roles. The estimated annualized compensation for this role is as follows:

• In Massachusetts, the good-faith, reasonable annualized full-time salary range for this role is between $114,000 – $136,500; placement within this range will vary based on several factors including, but not limited to experience, education, licensure/certifications, training and skill level
• In Illinois, the good-faith, reasonable annualized full-time salary range for this role is between $109,000 – $130,500; placement within this range will vary based on several factors including, but not limited to experience, education, licensure/certifications, training and skill level
• Annual discretionary performance bonus
• This role may also be eligible for other elements of discretionary compensation
• 4.5% 401(k) company contribution, which increases after 3 years of service and is 100% vested upon start date

Bain & Company’s comprehensive benefits and wellness program is designed to help employees achieve personal independence, protection and stability in the areas most important to you and your family.

• Bain pays 100% individual employee premiums for medical, dental and vision programs, offering one of the most comprehensive medical plans for employees without impacting your paycheck
• Generous paid time off, including parental leave, sick leave and paid holidays
• Fully vested 401(k) company contribution
• Paid Life and Long-Term Disability insurance

For all other locations, the good-faith, reasonable annualized full-time salary range for this role is commensurate with competitive geographic market rates for this role and will vary based on several factors including, but not limited to experience, education, licensure/certifications, training and skill level.

It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.