Software Development Engineer - Content Quality and AI Evaluation, Technical Content Experience
Amazon
- Location
- Onsite (Arlington, Virginia)
- Compensation
- $143k - $194k/yr
- Employment
- Full-time
- Level
- Mid Level
Posted 2 days ago
About the role
Join the Content Platform team to build automated quality, governance, and evaluation systems for AWS documentation. You will develop tools to measure content quality, evaluate AI agent performance, and create a shared metadata service to support internal builders and customers.
Skills
Software development
System architecture
C#
C++
Java
Perl
Object oriented design
Distributed systems
API development
AI evaluation
Large language models
Knowledge graph
Operational excellence
Data analysis
Regression testing
Metadata management
Benefits
- Health insurance
- Dental insurance
- Vision insurance
- 401(k) matching
- Paid time off
- Parental leave
- Life insurance
Perks
- Sign-on payments
- Restricted stock units
Full job details
Build the systems that define what "good" means for AWS documentation — and that measure whether AI agents are actually producing it. You'll work on an automated quality and governance engine that scores documentation across dimensions like readability, accessibility, link integrity, security, and discoverability, and on the evaluation platform that grades the AI agents now authoring and reviewing that content.
You'll also own the shared metadata service that answers "what do we actually know about this page?" — one API where many producing systems write typed, versioned metadata about pages and the sections inside them, and every consumer reads the complete cross-producer picture in a single call instead of integrating with each producer separately.
And you'll build the front door to our AI-first content platform: a home page and AI chat assistant, available both in the browser and in chat, that answers documentation questions for thousands of internal builders using retrieval over a knowledge graph and large language models.
This is applied AI with a hard measurement problem at its center. Millions of AWS customers read the content your systems evaluate, and the engineers and writers who produce it rely on your quality signals every day.
Key job responsibilities
- Build and extend automated quality checks that score documentation across measurable dimensions, exposed through APIs and agent tooling so problems are caught while content is being written
- Build and operate the AI evaluation platform — golden datasets, model-as-judge scoring, and regression gates — that teams use to prove an AI agent is good enough to ship and to catch regressions when models change
- Extend the rules layer that turns quality scores into pass, warn, and block decisions, with configurable thresholds, automated ticketing, and self-healing remediation
- Build and evolve the shared metadata service for pages and sections — namespaced multi-producer writes, typed and versioned records, single-call cross-producer reads, change events for downstream consumers, and the search and discovery layer on top of it
- Build features on the platform home page and its AI chat assistant across web and chat surfaces, including response streaming, conversational context, and feedback collection
- Own operational excellence for production services: on-call, availability and latency targets, and cost
- Partner with content strategy, product management, and applied science teams to turn quality standards into rules a machine can measure and enforce
A day in the life
You'll start your day looking at quality signals from overnight runs — did a new check fire on more pages than expected, is a model-generated classification drifting, did the chat assistant's answer quality dip after a model upgrade? You might dig into a check that's reporting false positives, then tune its thresholds and remediation guidance so writers get advice they can act on rather than noise.
Much of your week is building. You'll add a new quality dimension end to end, from the analysis logic through the API and agent-facing tooling, wire a team's AI agent into the evaluation platform so their agent gets scored against a curated dataset on every change, or onboard a new producer onto the metadata service so its data reaches every consumer without anyone building another integration. When an evaluation says an agent got worse, you'll help figure out whether the agent, the judge, or the dataset is wrong — that ambiguity is the interesting part of the job.
You'll work directly with content strategists who define quality standards, product managers who own quality and governance outcomes, and engineers on peer teams whose AI agents your evaluations grade. Your customers are the writers, service teams, and AI agents producing AWS documentation — and ultimately the millions of AWS customers who read it.
About the team
The Content Platform team builds the quality, governance, and evaluation infrastructure for AWS's documentation ecosystem. Our quality service scores content against measurable dimensions and returns results to authors and AI agents in seconds. Our rules layer turns those scores into decisions about what ships. Our evaluation platform is how the wider organization proves an AI system is good enough to release — a shared capability rather than something every team rebuilds. Our metadata service is the single place to ask what any system knows about a documentation page or section, and the substrate that quality, search, and analytics systems build on.
We also own the home page and AI chat assistant for our content platform, the entry point where builders start authoring sessions, publish documentation, and ask questions in the browser or in chat.
We're part of the organization responsible for the AWS Documentation website and the AI-first content platform behind it. The team values pragmatic engineering, operational excellence, and making quality the default rather than an afterthought. Basic Qualifications: - 3+ years of non-internship professional software development experience
- 2+ years of non-internship design or architecture (design patterns, reliability and scaling) of new and existing systems experience
- 1+ years of software development engineer or related occupational experience
- 1+ years of designing and developing large-scale, multi-tiered, multi-threaded, embedded or distributed software applications, tools, systems, and services using: C#, C++, Java, or Perl experience
- 1+ years of Object Oriented Design experience
- Bachelor's degree or foreign equivalent in Computer Science, Engineering, Mathematics, or a related field
- Experience programming with at least one software programming language Preferred Qualifications: - 3+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experience
- Bachelor's degree in computer science or equivalent
USA, VA, Arlington - 143,700.00 - 194,400.00 USD annually
You'll also own the shared metadata service that answers "what do we actually know about this page?" — one API where many producing systems write typed, versioned metadata about pages and the sections inside them, and every consumer reads the complete cross-producer picture in a single call instead of integrating with each producer separately.
And you'll build the front door to our AI-first content platform: a home page and AI chat assistant, available both in the browser and in chat, that answers documentation questions for thousands of internal builders using retrieval over a knowledge graph and large language models.
This is applied AI with a hard measurement problem at its center. Millions of AWS customers read the content your systems evaluate, and the engineers and writers who produce it rely on your quality signals every day.
Key job responsibilities
- Build and extend automated quality checks that score documentation across measurable dimensions, exposed through APIs and agent tooling so problems are caught while content is being written
- Build and operate the AI evaluation platform — golden datasets, model-as-judge scoring, and regression gates — that teams use to prove an AI agent is good enough to ship and to catch regressions when models change
- Extend the rules layer that turns quality scores into pass, warn, and block decisions, with configurable thresholds, automated ticketing, and self-healing remediation
- Build and evolve the shared metadata service for pages and sections — namespaced multi-producer writes, typed and versioned records, single-call cross-producer reads, change events for downstream consumers, and the search and discovery layer on top of it
- Build features on the platform home page and its AI chat assistant across web and chat surfaces, including response streaming, conversational context, and feedback collection
- Own operational excellence for production services: on-call, availability and latency targets, and cost
- Partner with content strategy, product management, and applied science teams to turn quality standards into rules a machine can measure and enforce
A day in the life
You'll start your day looking at quality signals from overnight runs — did a new check fire on more pages than expected, is a model-generated classification drifting, did the chat assistant's answer quality dip after a model upgrade? You might dig into a check that's reporting false positives, then tune its thresholds and remediation guidance so writers get advice they can act on rather than noise.
Much of your week is building. You'll add a new quality dimension end to end, from the analysis logic through the API and agent-facing tooling, wire a team's AI agent into the evaluation platform so their agent gets scored against a curated dataset on every change, or onboard a new producer onto the metadata service so its data reaches every consumer without anyone building another integration. When an evaluation says an agent got worse, you'll help figure out whether the agent, the judge, or the dataset is wrong — that ambiguity is the interesting part of the job.
You'll work directly with content strategists who define quality standards, product managers who own quality and governance outcomes, and engineers on peer teams whose AI agents your evaluations grade. Your customers are the writers, service teams, and AI agents producing AWS documentation — and ultimately the millions of AWS customers who read it.
About the team
The Content Platform team builds the quality, governance, and evaluation infrastructure for AWS's documentation ecosystem. Our quality service scores content against measurable dimensions and returns results to authors and AI agents in seconds. Our rules layer turns those scores into decisions about what ships. Our evaluation platform is how the wider organization proves an AI system is good enough to release — a shared capability rather than something every team rebuilds. Our metadata service is the single place to ask what any system knows about a documentation page or section, and the substrate that quality, search, and analytics systems build on.
We also own the home page and AI chat assistant for our content platform, the entry point where builders start authoring sessions, publish documentation, and ask questions in the browser or in chat.
We're part of the organization responsible for the AWS Documentation website and the AI-first content platform behind it. The team values pragmatic engineering, operational excellence, and making quality the default rather than an afterthought. Basic Qualifications: - 3+ years of non-internship professional software development experience
- 2+ years of non-internship design or architecture (design patterns, reliability and scaling) of new and existing systems experience
- 1+ years of software development engineer or related occupational experience
- 1+ years of designing and developing large-scale, multi-tiered, multi-threaded, embedded or distributed software applications, tools, systems, and services using: C#, C++, Java, or Perl experience
- 1+ years of Object Oriented Design experience
- Bachelor's degree or foreign equivalent in Computer Science, Engineering, Mathematics, or a related field
- Experience programming with at least one software programming language Preferred Qualifications: - 3+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experience
- Bachelor's degree in computer science or equivalent
The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.
USA, VA, Arlington - 143,700.00 - 194,400.00 USD annually