aws-bench: Test AI Agents Against Real AWS Infrastructure

AWS released aws-bench as a research preview on July 24, 2026. It does something most agent demos avoid: it puts an AI agent in front of real AWS resources and checks whether the job actually got done. The initial registry includes 78 basic tasks and 47 advanced tasks across seven scenarios. Read-only diagnosis tasks are judged against a reference answer. Tasks that create or modify infrastructure are verified against live AWS state.

That is a much harder test than asking a model to write Terraform in a chat window. A cloud agent has to discover the environment, choose the right API, handle permissions, avoid collateral changes, and leave the account in the expected state. aws-bench turns those behaviors into a repeatable experiment.

It also creates real risk. The framework provisions an AWS Organization and disposable member accounts. It deploys CDK stacks, hands scoped credentials to agents running in containers, and executes mutation tasks. Run it in a dedicated benchmark management account. Never point it at production, a shared security account, or an organization whose service-control policies and billing you cannot change.

Why static agent benchmarks are not enough

Static benchmarks are useful for syntax, reasoning, and fixed-answer questions. They are weak evidence for infrastructure operations.

Consider the instruction: “Find why the application in Region A cannot read its replica in Region B and fix the configuration.” A text benchmark can grade an explanation. It cannot prove that the agent inspected the right route tables, noticed a KMS-policy mismatch, changed only the intended resource, preserved unrelated controls, and returned the environment to a healthy state.

The AWS announcement says the tasks come from analysis of real AWS usage and cover investigation, troubleshooting, and infrastructure creation. The open-source aws-bench repository makes the execution model visible: scenarios are CDK stacks, agents run in sandboxed containers with scoped credentials, and verifiers write a binary reward of 1.0 for pass or 0.0 for fail.

aws-bench is not a certification that an agent is safe for production. It is a higher-quality signal. You still need organization-specific evaluations, least privilege, approvals, observability, and rollback.

Architecture and trust boundaries

The system has four important boundaries:

1. A management account creates and closes disposable member accounts.
2. A scenario deploys real infrastructure into an isolated account with CDK.
3. An agent runs inside a sandboxed container with task-scoped AWS credentials.
4. A verifier judges the answer or checks the final live state.

The boundary between the agent and the verifier matters most. If the same model creates the answer and decides whether it is correct, the score measures self-consistency, not success. Mutation tasks avoid that trap by checking AWS state programmatically. Introspection tasks use an LLM judge because natural-language diagnosis can have several valid phrasings, but the reference answer still anchors the result.

This is closely related to the release gates in AgentCore evaluations for CI/CD. The difference is scope. AgentCore Evaluations measures agent behavior and output; aws-bench adds disposable real cloud environments and stateful infrastructure tasks.

What ships in the first registry

The project defines a dataset as a versioned collection of tasks and scenarios. The launch registry lists:

It also publishes per-scenario datasets: api-and-observability, compute-and-data, databases-and-storage, ec2-multiregion, reference-architectures, serverless-apps, streaming-and-iot, and troubleshooting-multiservice.

The apparent mismatch between seven curated basic/advanced scenarios and eight named per-scenario datasets is not an error. The registry can publish scenarios independently, while a given curated release selects a subset. Pin dataset versions in serious comparisons. A benchmark name without a version can move underneath your model score.

The two task families need different interpretation:

• Introspection tasks are read-only. The agent diagnoses an environment and writes an answer. An LLM judge compares it with a reference answer.
• Mutation tasks create or modify resources. A programmatic verifier checks live AWS state and the framework resets changes afterward.

An introspection pass says the explanation met the judge’s criteria. It does not prove that a human would find the answer concise, safe, or useful. A mutation pass proves the expected state, but it may not reveal unnecessary API calls or a dangerous route taken before the final state was reached. Capture execution traces as a second layer of evidence.

Requirements and account design

The repository currently requires macOS or Linux, Python 3.12 or newer, uv, Docker Compose v2, and Docker buildx 0.17.0 or newer. The AWS principal must be able to create an AWS Organization and member accounts. aws-bench expects us-east-1, so set both AWS_REGION and AWS_DEFAULT_REGION consistently.

Use a new management account with a payment method and budget alarms. Do not attach it to your production organization. Give the human setup principal the documented management permissions, then let the framework scope agent and verifier roles inside disposable accounts.

Before installing anything, record these decisions:

The benchmark is research software. Read its IAM and account-management code before authorizing it. The project’s security model is stronger than handing an agent administrator credentials in a shared account, but “disposable” does not mean “free of consequences.” Quotas, billing, account-closure limits, and organization policies are real.

Install and validate locally

Clone the official repository and let uv build the pinned environment:

git clone https://github.com/aws-bench/aws-bench.git
cd aws-bench

uv sync
uv run aws-bench --help

docker compose version
docker buildx version
aws sts get-caller-identity --profile aws-bench-management

Do not export a permanent access key if IAM Identity Center is available. Configure a profile and refresh the SSO session:

aws configure sso --profile aws-bench-management
aws sso login --profile aws-bench-management

export AWS_PROFILE=aws-bench-management
export AWS_REGION=us-east-1
export AWS_DEFAULT_REGION=us-east-1

The AWS IAM best-practices guide recommends temporary credentials and federation for human users. A benchmark is not a reason to move backward to static keys.

Run the quickstart safely

Initialize the environment first. This phase creates the organization structure, member accounts, and quota requests needed by the selected dataset:

uv run aws-bench env init \
  --env-name isolated-agent-lab \
  -d aws-bench-quickstart \
  --wait-for-quotas

Inspect the resulting accounts and roles before setup. Then deploy the scenario resources:

uv run aws-bench env show --env-name isolated-agent-lab

uv run aws-bench env setup \
  --env-name isolated-agent-lab \
  -d aws-bench-quickstart

uv run aws-bench env verify \
  --env-name isolated-agent-lab

If the agent uses an Amazon Bedrock model, the project can generate a bearer token for the disposable environment. It is optional; agents using another provider supply their own provider credentials.

eval $(uv run aws-bench env creds --eval)

That command intentionally modifies the current shell environment. Run it in a dedicated terminal. Never paste its output into a ticket, build log, or chat.

Now run one agent/model pair against the quickstart dataset:

uv run aws-bench run \
  --env-name isolated-agent-lab \
  -d aws-bench-quickstart \
  -a claude-code \
  -m global.anthropic.claude-sonnet-5 \
  --yes

The model ID is an example from the project README. Use a model your account can access. The agent and model are separate dimensions: claude-code with one model is a different system from the same harness with another model. Record both.

Results land under jobs/<timestamp>/, and each trial contains reward.json. Preserve the job metadata, exact git commit, dataset version, agent version, model ID, Region, and attempt count. Without that context, a score cannot be reproduced.

Read the results beyond pass rate

Pass rate is the first number, not the final analysis.

For a run with tasks (T) and attempts (A), the simple pass rate is:

pass rate = successful trials / total trials

Add at least these measures:

• median and P95 task duration;
• model and tool cost per trial;
• API-call count;
• retries and throttles;
• permission-denied rate;
• number of resources changed;
• cleanup success;
• variance across repeated attempts.

Agent systems are nondeterministic. One attempt per task can reward luck. Run several attempts, report confidence intervals, and separate introspection from mutation tasks. A model might explain failures well and make poor changes, or do the reverse.

The AWS DevOps Agent guide discusses incident investigation as an operational workflow. Use that mindset here: the quality of evidence and the path taken matter, not just the final sentence.

Compare agents fairly

Keep the dataset, scenario state, number of attempts, maximum time, and tool access constant. Otherwise you are comparing configurations, not agents.

Use the oracle agent to validate a scenario and its reference solution before blaming a candidate model. If the oracle fails, the environment or verifier may be wrong.

Run candidates in a randomized or alternating order to reduce temporal effects such as service capacity. Reset the scenario between trials. Store failures, not only aggregate scores. A 90% result that deletes an unrelated bucket in one case may be unacceptable, while an 80% read-only diagnostic agent can still be useful with a human operator.

This is also why the Terraform and MCP infrastructure-agent guide favors plan, policy, and approval boundaries. A benchmark should measure whether an agent respects those boundaries, not remove them to make the score higher.

Build organization-specific tasks

The public registry cannot encode your naming rules, tagging policy, network topology, data classification, or change process. The durable value of aws-bench may be its framework for private scenarios.

A strong internal task contains:

1. a reproducible scenario defined as code;
2. a natural-language instruction that resembles a real ticket;
3. narrowly scoped agent permissions;
4. an expected result plus forbidden side effects;
5. a verifier independent from the agent;
6. a reference solution;
7. reliable reset and cleanup behavior.

For example, “Make the Lambda function private” is incomplete. A useful verifier checks that the public resource policy was removed, the intended service principal still works, the function configuration remains otherwise unchanged, logging is still enabled, and no unrelated functions changed.

Add negative tests. Ask the agent to perform an action that should be refused because approval is missing. Present a malicious string in a resource tag or log entry and verify that it is treated as untrusted data, not instruction. Deny one API and check whether the agent escalates safely instead of searching for a bypass.

The AWS Agent Registry governance guide can hold ownership and lifecycle metadata. The benchmark provides performance evidence. You need both: knowing an agent exists is not the same as knowing it is qualified for a job.

Cleanup is a test, not housekeeping

After a run, reset scenario state if you plan to continue:

uv run aws-bench env reset \
  --env-name isolated-agent-lab

uv run aws-bench env verify \
  --env-name isolated-agent-lab

When finished, remove deployed resources and terminate the disposable accounts:

uv run aws-bench env cleanup \
  --env-name isolated-agent-lab

uv run aws-bench env terminate \
  --env-name isolated-agent-lab

Then confirm the organization, CloudFormation stacks, S3 buckets, log groups, KMS keys, networking resources, and service-specific artifacts are gone or intentionally retained. Account closure can be asynchronous. Keep the management account and budget alarms active until billing is quiet.

If cleanup fails, stop. Don’t start another dataset and hope the old resources disappear. The repository documents AccountContaminatedError as a signal to run cleanup. A contaminated account invalidates reproducibility and can leak state between trials.

Security review before wider use

Threat-model three actors: the human running the benchmark, the agent under test, and untrusted content inside the scenario.

The human principal can create organizations and accounts, which is a powerful administrative capability. The agent has scoped credentials but may still create cost or damage inside its scenario. Scenario content can contain prompt injection. The verifier can be fooled if it checks only the happy-path field.

Minimum controls include:

• dedicated management and member accounts;
• no network path to production;
• temporary human and model-provider credentials;
• service-control policies that match the scenario without granting unrelated services;
• CloudTrail and cost monitoring;
• resource quotas and bounded run time;
• separate verifier credentials;
• code review for scenarios and verifiers;
• post-run cleanup validation.

For agent sandboxes, the principles in Docker sandboxes and microVM security still apply. A container narrows process isolation; it is not a magic trust boundary for credentials, the Docker socket, mounted files, or the host network.

When aws-bench is worth the effort

Use it when you build or select agents that will inspect or change AWS, and when the difference between plausible output and correct state matters. It is especially useful for model comparisons, harness changes, tool-schema revisions, regression testing, and research on cloud-operations agents.

Skip it for a simple Q&A bot that never gets AWS credentials. Use lighter offline evaluation first when the tool contract is still changing daily. Don’t use it to justify autonomous production access. A benchmark pass supports a staged rollout; it does not replace approval, least privilege, or incident response.

aws-bench moves cloud-agent evaluation in the right direction: real resources, isolated accounts, explicit tasks, independent verification, and repeatable cleanup. The teams that benefit most will resist chasing one leaderboard number and use it to study how their agents fail.

Put the benchmark in CI without creating unattended drift

Do not run the full live benchmark on every pull request. Account creation, quotas, model calls, and infrastructure deployment are slower and riskier than unit tests. Use a layered pipeline.

On each pull request, run the framework’s tests, lint, type checking, scenario schema validation, and any offline task fixtures. On merge to a protected branch, run a small pinned scenario in an already approved disposable environment. Run the full basic or advanced dataset on a schedule or before a model, harness, tool, or policy release.

The live job needs a global timeout and a guaranteed cleanup stage. In GitHub Actions, GitLab CI, or another runner, configure cleanup to execute after success, failure, or cancellation. A shell trap helps inside one process, but it does not protect against a terminated runner. Add a separate scheduled janitor that lists benchmark environments older than the allowed duration and alerts a human before cleanup.

Keep environment provisioning and benchmark execution in separate roles. The setup role can create accounts and stacks. The trial role should receive only the scoped scenario permissions. The reporting role can read job artifacts but should not be able to mutate the scenario. This separation makes CloudTrail evidence easier to interpret and limits what a compromised runner can do.

Publish a release scorecard rather than one pass percentage. Include dataset and version, git commits for framework and tasks, agent and model IDs, attempts, pass rate by task family, confidence interval, duration, cost, permission failures, cleanup status, and links to failed trials. Block promotion on cleanup failure even when every task passed. A benchmark that leaves resources behind has failed an operational requirement.

For trend reporting, compare only like-for-like runs. When a dataset changes, start a new series or rerun the previous agent on the new version. Otherwise a score increase may come from easier tasks rather than a better system.

Trusted resources

• AWS announcement for aws-bench
• aws-bench source and getting-started guide
• aws-bench datasets repository
• Harbor agent-evaluation framework
• AWS Organizations documentation
• AWS IAM security best practices
• AWS CDK bootstrapping
• AWS Budgets documentation