AI agents are moving deeper into production software, and that creates a testing problem that traditional CI/CD pipelines were never really designed to handle. A normal application can usually be tested with predictable inputs and outputs, but an AI agent may behave differently after even a small change to its prompt, model, tool configuration, or supporting code. AWS is now addressing that challenge with a reference implementation that brings automated AI agent evaluation directly into GitHub Actions.
The new approach allows developers to run behavioural tests whenever relevant changes are submitted through a pull request. If an agent's evaluation scores fall below a configured threshold, the GitHub Actions job can fail. When that evaluation job is configured as a required status check through GitHub branch protection or repository rulesets, the failing result can also prevent the pull request from being merged.
Bringing Agent Behaviour Into The CI/CD Pipeline
The idea is relatively simple: AI agent behaviour should be tested before code reaches production, just like application logic, security checks, code quality, and integration tests. The difference is that an agent cannot always be validated with a simple expected-output comparison, because large language models are inherently non-deterministic and can take different paths even when given the same request.
AWS's implementation uses Amazon Bedrock AgentCore Evaluations to determine whether an agent continues to behave acceptably after changes are made. Those changes might involve the agent's source code, its system prompt, the underlying model, an MCP server, infrastructure configuration, or the evaluation scripts themselves. Instead of assuming that a successful deployment means the agent still works properly, the pipeline actively exercises the agent and scores how it responds.
AWS published the technical guide and supporting reference implementation on September 8. The example includes the agent itself, infrastructure-as-code, authentication setup, evaluation scripts, and the GitHub Actions workflow needed to connect everything together.
How The Reference Pipeline Works
In AWS's example, the agent is built using the Strands framework and runs on Amazon Bedrock AgentCore Runtime. It can connect to tools exposed through a Model Context Protocol server, while AWS Cloud Development Kit code provisions the supporting runtime environment, IAM roles, and Amazon Cognito resources required by the application.
When a qualifying pull request is opened against the main branch, GitHub Actions deploys the development stack and starts the agent. A predefined set of evaluation prompts is then sent to the agent so that its behaviour can be observed under controlled test conditions. The resulting interactions generate OpenTelemetry traces, which are collected through Amazon CloudWatch and passed into AgentCore Evaluations for scoring.
Once the evaluation completes, the results are returned to the GitHub Actions workflow. The workflow can then decide whether the agent performed well enough for the job to pass or whether the change should be treated as a regression.
AWS Tests More Than Just The Final Answer
One of the more useful parts of the approach is that AWS does not limit evaluation to the text eventually shown to the user. The reference implementation examines several different parts of the agent's behaviour, including whether the requested task was completed, whether the response was appropriate, whether the right tool was selected, and whether the tool was called with the correct parameters.
AWS demonstrates this with four built-in evaluators: GoalSuccessRate, Correctness, ToolSelectionAccuracy, and ToolParameterAccuracy. Together, these checks provide a much broader picture of performance than simply comparing one generated answer against another.
This matters because an agent can produce a response that looks plausible while still making mistakes internally. It might call the wrong service, pass the wrong customer identifier, choose an unnecessary tool, or misunderstand the original task. Behavioural evaluation gives developers a way to catch those problems before the code is merged.
A Small Prompt Change Can Cause A Real Regression
AWS's reference example deliberately introduces a problem by modifying the agent's system prompt. The application still runs, but the change causes several behavioural metrics to fall below the required standard. GoalSuccessRate, Correctness, and ToolParameterAccuracy fail, while ToolSelectionAccuracy continues to pass.
That example highlights why separating evaluation categories is useful. The agent may still know which tool it should call, but that does not mean it has understood the task correctly or supplied the right information to that tool. Looking at a single overall score could make it much harder to see where the regression actually occurred.
AWS sets the passing threshold at 0.8 in the sample workflow. If one of the required evaluation metrics drops below that value, the script exits with a failure, causing the GitHub Actions job to fail as well. After AWS restores the original system prompt and reruns the pipeline, the agent clears all four evaluation checks.
Turning A Failed Evaluation Into A Merge Blocker
A failing GitHub Actions job does not automatically stop someone from merging a pull request. That behaviour only becomes enforceable when the repository configures the evaluation job as a required status check using protected branches or GitHub rulesets.
Once that is enabled, agent quality becomes part of the repository's merge policy. A code change might compile successfully, pass its unit tests, and satisfy every conventional quality check, but it can still be blocked because the agent no longer behaves within acceptable limits.
For organisations building systems where agents can call APIs, access business applications, retrieve private data, or trigger real actions, that kind of protection can be especially important. It turns behavioural testing from something developers perform manually into something enforced automatically during development.
Building Larger Regression Suites With Evaluation Datasets
The September 8 reference implementation uses a fixed collection of test prompts, but AgentCore also includes support for dataset-based evaluation. That feature is intended for broader test suites where teams want to define many scenarios and run them repeatedly against an agent.
A scenario can contain considerably more than the initial prompt. Developers can define an expected response, natural-language assertions describing the behaviour the agent should demonstrate, and an expected trajectory showing which tools should be called during execution.
Those fields can then be connected to different evaluators. Correctness can compare the generated response against an expected answer, GoalSuccessRate can assess whether behavioural assertions were satisfied, and trajectory-based evaluators can compare the agent's actual tool activity against the execution path defined in the test dataset.
AWS currently describes dataset evaluation as a public preview. Even so, it points toward a more structured way of testing agents, where teams maintain behavioural test suites in much the same way they already maintain unit and integration tests.
Versioning Makes Agent Tests Easier To Reproduce
AgentCore also supports versioning for evaluation datasets. Teams can maintain an editable Draft, publish that Draft as an immutable numbered version, and then continue changing the working copy without affecting the published baseline.
That is useful when comparing agent performance over time. If an evaluation suddenly changes, developers need to know whether the problem came from the agent, the model, the system prompt, or the test dataset itself. Immutable dataset versions make those comparisons much easier to reproduce.
AWS also records whether the working Draft contains unpublished changes. This gives development teams a clearer view of whether the evaluation scenarios currently being edited still match the version used in earlier regression tests.
Testing The Execution Path, Not Only The Response
Another important distinction is the ability to evaluate the path an agent takes while completing a task. AgentCore can evaluate an entire session, a single trace, or even specific tool-call spans within that trace.
This makes it possible to test whether an agent selected the right tools, called them in the right order, and supplied appropriate parameters. For multi-step applications, that can be just as important as the final response.
Imagine an agent that needs to retrieve an account, verify eligibility, query another system, and then perform an action. A final response could appear correct even if the agent skipped a verification step along the way. Trajectory evaluation gives developers a way to catch that kind of problem.
Dealing With The Non-Deterministic Nature Of AI
AI agents cannot always be tested with the rigid assumptions used for traditional software. The same request may produce slightly different wording, execution paths, tool choices, or intermediate decisions each time it runs. That does not necessarily mean something is wrong.
AWS therefore recommends building datasets around expected behaviour rather than expecting identical outputs. Developers can define what a successful result looks like and which actions matter, while still allowing some flexibility in how the agent gets there.
This is a major change in testing philosophy. Instead of asking whether the output exactly matches a stored string, teams increasingly need to ask whether the agent achieved the intended goal, followed acceptable behaviour, and stayed within defined operational boundaries.
LLM Judges And Deterministic Checks Serve Different Purposes
AgentCore supports both model-based evaluation and programmatic evaluation. Some checks, including Correctness and GoalSuccessRate, can use an LLM acting as a judge to determine whether an agent's behaviour meets the required standard.
This is useful for tasks where quality cannot easily be reduced to a simple true-or-false comparison. However, model-based evaluation introduces its own variability, because the judge model is also non-deterministic. AWS warns that the same trace can sometimes receive slightly different scores across repeated evaluations.
For that reason, AWS recommends leaving some safety margin when choosing evaluation thresholds. Setting the limit too close to the expected score could result in tests that intermittently pass and fail without any meaningful behavioural change.
Trajectory Matching Offers More Predictable Testing
For scenarios where developers want deterministic results, AgentCore provides programmatic trajectory matching. Rather than asking another model to judge behaviour, the system directly compares recorded tool calls with the expected execution path defined by the test.
AWS supports exact-order, in-order, and any-order trajectory checks. Exact-order testing requires the agent to follow the specified sequence precisely, while in-order testing allows extra steps as long as the required calls appear in the correct sequence. Any-order testing simply checks that the expected tools were used, regardless of order.
Because these comparisons are performed programmatically, they avoid the variability of LLM-based judging. They are particularly useful when an application has strict requirements around tool usage or workflow sequencing.
Custom Evaluators Can Add Application-Specific Rules
Not every organisation will want to rely entirely on AWS's built-in evaluators. AgentCore therefore allows teams to create custom code-based evaluators using AWS Lambda, with support for Python and JavaScript.
These custom checks can enforce application-specific requirements that would be difficult or unnecessary to evaluate using another language model. A developer could verify that a specific tool was never called without prior validation, check whether a required field was present, or confirm that an agent avoided performing a prohibited action.
Using custom programmatic checks alongside model-based evaluators gives teams more flexibility. Semantic quality can be judged with an LLM, while strict business rules can remain deterministic.
Different Evaluation Modes Cover Development And Production
AgentCore supports several evaluation modes depending on where an agent is in its lifecycle. On-demand evaluation is intended for controlled testing during development and CI/CD, while online evaluation can sample interactions from production traffic. Batch evaluation allows larger groups of sessions to be scored asynchronously.
Dataset testing also supports on-demand and batch runners. The on-demand runner collects spans and submits evaluations from the client side, while the batch runner delegates much of the trace collection and evaluation process to the service itself.
This creates the possibility of using a consistent evaluation strategy from development through production. Teams could test a small set of critical scenarios during every pull request, run larger benchmark datasets before a release, and continue evaluating real interactions after the application is deployed.
Stored Traces Can Be Evaluated Without Running A Live Agent
AWS also describes another way of running CI evaluations. Instead of deploying the current agent and executing new requests, developers can capture representative OpenTelemetry traces from a staging environment and evaluate those stored traces during the CI process.
This can reduce deployment time and the cost of repeatedly invoking a live agent. It can also be useful for testing evaluation rules against known behaviour.
There is an important limitation, however. Stored traces reflect behaviour captured from the staging deployment, not behaviour generated from the code contained in the current pull request. As a result, the method is useful for some benchmarking scenarios but cannot fully replace executing the modified agent when the goal is to detect regressions introduced by new code.
GitHub Authentication Avoids Long-Lived AWS Credentials
Security is another part of the reference implementation. The GitHub Actions workflow uses OpenID Connect federation to assume an AWS IAM role, which removes the need to store permanent AWS access credentials inside the repository.
When the agent needs access to OAuth-protected MCP tools, the example obtains a machine-to-machine token through Amazon Cognito. This allows the CI workflow to authenticate with protected services while keeping long-lived secrets to a minimum.
AWS does point out that the sample configuration gives the pipeline access to all tools. It does not attempt to validate different role-based access-control scenarios, so teams deploying similar systems in production would still need to test their own permission boundaries separately.
More Reliable Testing Comes With Extra Time And Cost
Running behavioural evaluations on every pull request adds overhead. AWS says its end-to-end reference workflow takes around 10 minutes, largely because it needs to deploy the environment, initialise the runtime, invoke the agent, wait for telemetry, perform the evaluations, and tear everything down afterwards.
Trace propagation can take between 30 and 90 seconds in AWS's testing. The sample workflow deals with this by retrying trace collection every 30 seconds for up to 10 minutes before abandoning the attempt.
Model-based evaluation also carries a usage cost. In AWS's example, four evaluators are executed across five test prompts, producing 20 judge-model invocations for every pull request. For a small project that may be manageable, but larger regression suites or high-volume repositories could generate significantly more evaluation traffic.
Teams can reduce some of that cost by using deterministic trajectory checks or custom code-based evaluators where an LLM judge is not actually necessary.
Why This Matters As AI Agents Become More Autonomous
The value of this approach becomes clearer as AI agents move beyond simple conversational assistants. Modern agents are increasingly expected to query databases, call APIs, interact with internal systems, use MCP tools, process information, and perform multi-step tasks on behalf of users.
That makes behavioural regressions much more serious. A chatbot choosing slightly different wording may be harmless, but an agent selecting the wrong tool or supplying the wrong parameters can lead to incorrect actions.
Traditional software testing alone cannot fully detect these problems because the underlying application may still function perfectly from a technical perspective. Agent evaluation adds another layer of testing that focuses specifically on whether the AI component continues to make acceptable decisions.
AI Development Is Starting To Adopt More Traditional Engineering Discipline
AWS's GitHub Actions implementation also reflects a broader change taking place across AI development. Agent-based applications are gradually moving away from experimental workflows where developers simply try a few prompts manually and decide whether everything looks acceptable.
Instead, teams are beginning to introduce repeatable datasets, quality thresholds, regression suites, trace inspection, deterministic checks, and automated merge gates. These are familiar ideas from conventional software engineering, adapted for systems that do not always produce deterministic outputs.
The challenge is that agent testing requires a more flexible definition of correctness. The goal is not always to reproduce exactly the same response, but to ensure the agent still understands the task, follows the appropriate process, uses the right tools, and reaches an acceptable outcome.
Final Thoughts
AWS bringing AgentCore Evaluations into GitHub Actions is a practical step toward making AI agent testing part of everyday software development. Instead of treating behavioural quality as something developers check manually after deployment, teams can evaluate agents automatically whenever important code, prompts, infrastructure, or tool configurations change.
There are still trade-offs. Evaluation pipelines require additional infrastructure, introduce extra runtime and cost, and LLM-based judges cannot provide perfectly identical scores every time. Building useful regression datasets also takes effort because developers need to define what good agent behaviour actually looks like.
Even so, the direction is becoming increasingly clear. As AI agents gain more responsibility inside real applications, simply verifying that the code runs successfully will no longer be enough. Development teams will also need confidence that the agent still behaves correctly—and putting those checks directly into CI/CD may eventually become just as normal as running unit tests before a pull request is merged.
Comments 0