Staff AI Engineer - AI Labs

Technology Scouting & Evaluation

• Run short, instrumented spikes and benchmarking on new models, tools and frameworks: LLMs, agentic systems, vector databases, orchestration frameworks, copilots, assistants and more.

• Compare vendor and open-source options, documenting trade-offs across quality, cost, latency, security and integration complexity.

• Deliver concise decision memos with clear recommendations: adopt, watch, or avoid.

Evaluation Harnesses & Sandboxes

• Design and maintain evaluation environments (e.g. datasets, prompts, scenarios, telemetry) to test models under realistic constraints.

• Build automation and tooling to measure quality, robustness, latency and cost, including regression tracking over time.

• Ensure every evaluated technology has benchmark coverage and a documented risk and limitations view.

Prototyping & Technical Validation

• Build enough of a system to understand how a technology behaves under realistic conditions, not just in vendor demos or isolated examples.

• Explore architecture, integration patterns, operational constraints, security boundaries, and failure modes through working prototypes.

• Determine what must be true for a proof of concept to become a viable production capability.

• Prefer focused prototypes that answer specific technical questions over prematurely building production systems.

Recommendations, Readiness & Hand-offs

• Translate technical findings into clear decision memos for both technical and non-technical stakeholders.

• For validated technologies, produce readiness guidance covering recommended patterns, guardrails, known limitations, operational considerations, and integration requirements.

• Coordinate hand-offs to the engineering teams responsible for productionization.

• Support those teams during the transition when deep context from the evaluation is required, without becoming the permanent owner of the resulting system.

• Track what happens after Lab recommendations and use those outcomes to improve future evaluation methods.

Governance, Risk & Standards

• Work with Security, Legal, Compliance and other AI teams to document risk assessments, mitigations and governance recommendations for each evaluated technology.

• Maintain checklists, decision templates and lightweight standards reusable across evaluations and by partner teams.

• Incorporate learnings from third-party AI tooling already in use, such as external copilots and the AWS AI suite, into adoption guidelines.

Collaboration, Mentoring & Community

• Partner with other AI teams and domain teams to ensure clear boundaries and smooth collaboration.

• Participate in hiring as a technical evaluator and culture champion.

• Mentor engineers in the Lab and adjacent teams on evaluation methods, benchmarking and experimental design.

• Share knowledge through internal write-ups, tech talks and occasional external meetups and conferences.

Technical depth

• 8+ years of software engineering experience, including significant experience operating at senior or Staff-level scope.

• Deep hands-on experience building and evaluating systems based on LLMs and modern AI tooling.

• Strong software engineering fundamentals and the ability to rapidly build high-quality experimental systems.

• Experience building agentic or multi-step AI systems involving tool use, orchestration, state, retrieval, or external integrations.

• Strong knowledge of cloud infrastructure, preferably AWS, and the ability to run experimental workloads securely and cost-consciously.

• Experience with observability, telemetry, testing, and benchmarking of complex systems.

• Ability to reason about system architecture, reliability, scalability, asynchronous workflows, and distributed components where relevant.

• Track record of designing experiments or benchmarks that influenced meaningful technical decisions.

Benchmarking & evaluation

• Track record designing and running benchmarks that compare AI models and tools under real constraints.

• Experience constructing evaluation datasets: task selection, labelling, holdout discipline, and keeping a set useful as models improve.

• Working knowledge of LLM-as-judge methods and their failure modes, alongside human evaluation, inter-annotator agreement, and a view on when each is appropriate.

• Able to reason about statistical significance on small samples, and to state confidence honestly rather than over-reading a result.

• Familiarity with regression tracking, telemetry and versioning, so that a result stays reproducible months later.

Decision-making

• Able to turn ambiguous "we should try this new thing" ideas into well-scoped evaluation plans with clear hypotheses and metrics.

• Comfortable making trade-off calls across quality, latency, cost and vendor lock-in, and documenting them clearly.

• Experience writing short, opinionated decision memos that help others move fast.

Collaboration & communication

• Can explain technical results to non-specialists in concrete, concise terms.

• Experience working with platform, product and operations teams to align evaluations with real use cases.

• Able to influence without authority, aligning teams around shared standards and guardrails.

Mindset

• Curious and biased toward experimentation, combined with disciplined measurement and risk awareness.

• Comfortable in a small, high-leverage team without embedded PMs. You structure your own work and keep stakeholders informed.

• Builder attitude: you prefer reusable tools, templates and playbooks over one-off work.