AI Operations Engineer

About the Role

We are seeking an AI Operations Engineer to help support and optimize our next-generation AI platform focused on compliance, surveillance, and intelligent workflow automation.This role will work closely with AI engineers, product managers, and domain experts to evaluate AI outputs, manage training and evaluation datasets, improve prompts and agent behaviors, and help operationalize LLM-powered systems across the platform.

** Work Location: Durham, NC/Hybrid-2 days in office**

The ideal candidate is detail-oriented, analytical, and excited about working hands-on with modern AI technologies including agents, MCP servers, vector search, RAG pipelines, document/media processing, and large language models.

Responsibilities

• Test and evaluate AI agents, prompts, and workflows across multiple use cases
• Create, maintain, and curate labeled datasets for AI evaluation and tuning
• Review AI-generated outputs for quality, accuracy, safety, and consistency
• Update and optimize prompts, agent instructions, and AI “skills”
• Help improve retrieval quality for vector search and RAG pipelines
• Assist with evaluation frameworks and benchmark creation for LLM systems
• Work with engineering teams to validate MCP connectors and integrations
• Analyze model failures, hallucinations, and edge cases to improve system performance
• Participate in human-in-the-loop review workflows for AI-generated content
• Document testing procedures, findings, and recommended improvements
• Collaborate with compliance and domain experts to refine AI behaviors and detection logic

What working at ACA offers: 

We offer a competitive compensation package where you’ll be rewarded based on your performance and recognized for the value you bring to our business. Our Total Rewards package includes medical and dental coverage and 401(k) plans, and a wide range of paid time off options as well as a flexible work environment. You’ll also be granted time off for designated ACA Paid Holidays, Summer Fridays, Personal/Family Care, and other leaves of absence when needed to support your physical, financial, and emotional well-being. We also offer unique benefits such as Student Debt Forgiveness and Pet Insurance. 

About ACA: 

ACA Group is the leading governance, risk, and compliance (GRC) advisor in financial services. We empower our clients to reimagine GRC and protect and grow their business. Our innovative approach integrates consulting, managed services, and our ComplianceAlpha® technology platform with the specialized expertise of former regulators and practitioners and our deep understanding of the global regulatory landscape. 

What we commit to: 

ACA is firmly committed to a policy of nondiscrimination, which applies to recruiting, hiring, placement, promotions, training, discipline, terminations, layoffs, transfers, leaves of absence, compensation and all other terms and conditions of employment. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability or protected veteran status, or any other legally protected basis, in accordance with applicable law.