Meta is seeking a Production Systems Engineer to lead the reliability, scalability, and operational excellence of our AI infrastructure fleet. This role sits at the intersection of large-scale AI compute systems and production engineering, supporting the hardware and software stack that powers Meta's AI training and inference workloads across tens of thousands of accelerators and servers. You will drive the strategy for fleet health, automated diagnostics, and lifecycle management of AI-optimized hardware, partnering closely with hardware design, silicon, software, and data center operations teams to ensure our AI systems are production-ready and operating at peak efficiency.

Responsibilities

Define and drive the long-term technical strategy for fleet health monitoring, automated remediation, and lifecycle management of AI compute infrastructure at hyperscale
Lead cross-functional initiatives spanning hardware design, firmware, systems software, and data center operations to bring new AI server platforms from pre-production validation through mass deployment
Architect and oversee development of large-scale test automation frameworks for AI accelerator systems, covering GPU, TPU, and custom silicon configurations across diverse fleet topologies
Establish organizational standards for root cause analysis methodologies, failure mode classification, and escalation workflows for AI system incidents at fleet scale
Drive vendor and partner engagement on hardware specifications, firmware quality, and test methodology improvements to continuously raise the reliability bar for AI infrastructure
Develop and champion data-driven visibility platforms that surface fleet health signals, predictive failure indicators, and capacity risk across AI training and inference clusters
Lead ambiguous, high-impact investigations into systemic hardware and software failures affecting AI workload availability, producing durable solutions and documented learnings
Mentor and provide technical direction to other engineers across the production systems organization, shaping engineering practices and raising the collective capability of the team
Collaborate with capacity engineering and AI platform teams to translate fleet reliability requirements into hardware procurement, qualification, and deployment criteria
Influence the roadmap of internal tooling and automation systems to reduce mean time to detection and resolution for AI fleet incidents at scale

Minimum Qualifications

Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
12+ years of experience in production systems engineering, server hardware support, or infrastructure reliability engineering at hyperscale
12+ years of experience troubleshooting and root-causing failures across the full hardware and software stack, including CPUs, accelerators, memory, networking, storage, firmware, and operating systems
Experience leading large-scale technical programs across hardware, firmware, and software engineering organizations with demonstrated organizational impact
Experience designing and implementing automated diagnostics, test frameworks, or remediation systems for large server fleets
Experience with Linux systems administration and scripting or programming in Python or equivalent languages for infrastructure tooling

Preferred Qualifications

Experience supporting AI training or inference infrastructure, including GPU clusters, high-speed interconnects such as InfiniBand or RoCE, and AI accelerator hardware at fleet scale
Experience engaging directly with silicon vendors or ODMs on firmware qualification, failure analysis, and hardware specification development for AI compute systems
Experience in post-production hyperscale environments managing full hardware lifecycle progressions for AI or HPC server platforms
Experience with predictive fleet health modeling, telemetry pipelines, or data visualization platforms applied to large-scale hardware operations

$208,000/year to $289,000/year + bonus + equity + benefits

Production Systems Engineer, Fleet AI Systems

About the role

Skills

Benefits

Perks

Full job details