AI Operations Engineer
BlytzPay
- Location
- Onsite (Draper, Utah)
- Employment
- Full-time
- Level
- Mid Level
Posted 3 weeks ago
About the Role
BlytzPay is seeking an AI Operations Engineer to build and refine agentic AI skills. This role involves troubleshooting AI logic, managing human-in-the-loop evaluations, and contributing to architectural strategy.
Skills
Agentic AI
LLM Memory Management
Prompt Engineering
Tool Orchestration
Data Sanitization
Evaluation Metrics
Data Pipelines
Technical Communication
Agentic Coding
Troubleshooting
Full job details
Role
- Skill Construction: Use agentic coding to build agentic skills (third-party connections and tool-use logic).
- Troubleshooting: When our internal users are unable to train the agent out of a mistake via feedback, you step in. You’ll dig into the code, refine the prompts, and fix the hallucination or logic error.
- Human-in-the-Loop Management: Work directly with internal employees to manage evals. You’ll help them understand how to test the agent and translate their qualitative feedback into quantitative technical improvements.
- Architectural Strategy: You will work within the existing agentic architecture, providing feedback on how the system handles memory, context, and tool-triggering.
Requirements
- Agentic AI Expertise: Deep understanding of LLM memory management, context windows, and tool interaction/orchestration.
- Familiarity with Data Science Principles: Knowledge of data sanitization, evaluation metrics, and data pipelines.
- Exceptional Communication and Interpersonal Skills: Ability to translate complex technical concepts for non-technical stakeholders and act as a liaison to internal staff.
- Bachelors Degree in CS/Data Science