As AI agents increasingly write and operate infrastructure, reliability systems must be able to reason and act at machine speed. In this role, you will set the technical direction and drive execution for an AI agent that autonomously gathers context, investigates incidents, identifies likely root causes, builds a safe mitigation plan, and executes it - with humans supervising and able to intervene at any point. The immediate challenge is building an agent that is both highly accurate and broadly trusted. Experienced incident responders need a transparent collaborator that helps them investigate, continuously surfaces relevant evidence, and keeps them in control. Engineers who encounter incidents only occasionally need an agent that leads the investigation and guides them to a safe outcome. You will lead two tightly connected areas: - Agent quality - skills, tools, context, memory, offline evaluation, backtesting, online experimentation, and learning from production outcomes. - Responder experience - a multi-turn interface that gives responders visibility into the agent's work and brings together the operational data they need to make decisions. This is a deeply hands-on technical leadership role at the intersection of large-scale distributed systems, incident response, applied AI, and product engineering. You will personally design and ship critical parts of the system while leading senior engineers across team and organizational boundaries. The long-term opportunity is to help create self-healing infrastructure that resolves production incidents safely at Meta scale.

Responsibilities

Define and drive the technical vision, architecture, and execution roadmap for AI-powered incident investigation and mitigation
Personally design, code, evaluate, and ship major capabilities in a production agentic system
Improve timely root-cause accuracy through better reasoning, skills, tool use, context, memory, planning, and domain knowledge
Build the offline evaluation, historical replay, and online experimentation systems needed to measure changes and continuously improve agent behavior
Create a transparent, multi-turn experience that supports collaborative investigation with expert responders and agent-led guidance for less frequent responders
Enable the agent to reason across telemetry, code, configuration, deployments, service dependencies, operational knowledge, and incident history
Build closed-loop learning systems that turn production outcomes and user feedback into evaluations, experiments, and product improvements
Advance the system toward progressive autonomy with explicit authorization, validation, auditability, rollback, and human oversight
Build an extensible platform that combines shared capabilities with domain-specific skills, tools, and agents contributed by infrastructure and product teams
Partner closely with on-call engineers, reliability experts, product groups, data science, and product management to drive accuracy, adoption, and realized operational impact
Ensure the system meets a high bar for reliability, latency, scale, security, privacy, and cost
Influence technical strategy across organizations and mentor senior engineers working across distributed systems and applied AI

Minimum Qualifications

Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
10+ years of software engineering experience, including experience building and operating large-scale distributed, infrastructure, or production AI systems
Experience setting technical direction and leading complex engineering efforts across multiple teams or organizational boundaries
Experience applying AI to production problems at scale, including evaluation of system quality and failure modes
Recent hands-on experience designing, coding, debugging, and shipping complex production systems
Experience building systems that support critical, always-on production workloads
Experience diagnosing complex production systems using telemetry, code, configuration, deployments, and dependency information
Experience translating customer needs into technical architecture and driving adoption of a platform or developer product
Experience coding in languages such as C++, Java, Python, Rust, or equivalent
Experience influencing senior engineers and leaders without direct organizational authority

Preferred Qualifications

Experience improving agent quality through context engineering, skills, memory, evaluation, automated experimentation, hill climbing, fine-tuning, or reinforcement learning
Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements)
Experience building multi-turn, human-agent experiences for complex or high-stakes workflows
Experience building production AI agents that use tools and reason across telemetry, code, configuration, deployments, and operational knowledge
Track record of taking an ambitious technical product from early adoption to trusted, broad production use
Experience designing autonomous or semi-autonomous production actions with identity, authorization, independent validation, auditability, rollback, and escalation
Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews)
Experience with incident management, SRE, observability, on-call operations, or automated remediation at significant scale
Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies
Experience building extensible platforms that allow domain teams to contribute specialized capabilities while preserving a coherent user experience

$219,000/year to $301,000/year + bonus + equity + benefits

Software Engineer, AI for Incident Response (Technical Leadership)

About the role

Skills

Benefits

Perks

Full job details