Building an AI agent is becoming easier. Proving that the same agent can safely handle a real business process is much harder. That is the problem Synthesized is trying to address with Test Data Agent, a new infrastructure capability designed to create realistic datasets, permissions, business relationships and application states for testing AI agents before they are allowed into production environments.
The technology recently entered limited availability for existing customers and ecosystem partners, with wider availability planned for later in the third quarter of 2026. Synthesized also says the system is being tested with design partners at Tier 1 global banks.
Rather than judging whether an AI agent produced a good answer, Test Data Agent focuses on something deeper: whether the environment used to test that agent actually resembles the complicated conditions it will encounter in the real world.
Testing AI Agents Is Different From Testing Normal Software
Traditional software testing is comparatively predictable.
A developer provides an input, the application performs a defined operation and the test checks whether the output matches what was expected.
AI agents behave differently.
An agent may make several decisions before completing a task. It could choose between different tools, query databases, update applications, change system state and then modify its strategy depending on what happened earlier.
Even when given exactly the same instruction twice, an agent may not follow precisely the same route.
Anthropic highlighted this challenge in its January 2026 guidance on evaluating AI agents, noting that agents operate across multiple turns, use tools and adapt to intermediate results.
AWS has made a similar point in its production-agent guidance: identical prompts do not necessarily produce identical outcomes because an agent's decisions depend on context.
That means evaluating an agent once is rarely enough.
Teams need repeated trials to understand whether the agent succeeds consistently rather than occasionally arriving at the correct answer.
A Good Final Answer Does Not Mean the Agent Worked Correctly
One of the biggest dangers in agent testing is focusing only on the response displayed to the user.
An agent could produce a perfectly convincing final message while doing something wrong behind the scenes.
It may have selected the wrong tool.
It may have supplied incorrect parameters.
It may have changed the wrong database record.
Or it could claim that an action was completed when the underlying application state was never changed at all.
Anthropic therefore distinguishes between an agent's transcript and the actual resulting state of the environment.
If an agent says, "The invoice has been approved," the test cannot simply verify that those words were generated.
It also needs to confirm that the correct invoice was actually approved in the relevant system, by an authorised process, with the correct values and permissions.
AWS similarly recommends evaluating tool choice, parameter passing, execution paths and actual task completion alongside the final output.
For enterprise applications, this distinction is critical.
Real Businesses Contain Messy Data and Exceptions
Controlled demonstrations often make AI agents look extremely capable because everything surrounding them has been carefully prepared.
Production environments are rarely so tidy.
A real enterprise agent might encounter incomplete customer records, duplicate transactions, unusual currencies, missing approvals, conflicting permissions, outdated information or dependencies spread across several applications.
These edge cases are precisely where failures become dangerous.
An AI purchasing agent may work correctly when processing an ordinary invoice but fail when the supplier record is incomplete.
A banking agent might handle standard transactions but behave incorrectly when a customer account carries an unusual restriction.
Testing therefore needs to reproduce not only the happy path but also the awkward situations businesses experience every day.
That is where Synthesized believes Test Data Agent can help.
Test Data Agent Builds the Environment Around the Scenario
According to Synthesized, teams can describe a particular business scenario and the system will identify what information and system conditions are required to test it properly.
It can then generate synthetic information, mask sensitive values or create representative subsets of existing datasets.
Importantly, the goal is not simply to generate random rows in a database.
The system is intended to preserve referential integrity, statistical characteristics and business relationships across connected systems.
For example, an invoice is not useful on its own.
It may need to correspond to a valid supplier, purchase order, currency, payment terms, business unit and user with appropriate permissions.
If those relationships are incorrect, the test environment may look realistic while behaving nothing like production.
Synthesized says Test Data Agent is designed to recreate those relationships rather than merely produce isolated synthetic records.
Synthetic Data Helps Protect Sensitive Information
Using actual production data for AI testing introduces obvious privacy and security concerns.
Enterprise databases may contain customer information, employee details, financial records and commercially sensitive data.
Moving that information into development or external testing environments may violate internal policies or regulatory requirements.
Synthetic data provides an alternative.
Instead of copying real customer records, organisations can create artificial datasets that reproduce important characteristics of the original information without exposing the underlying people or transactions.
However, synthetic data also creates its own challenge.
If it is too simplistic, the AI agent may perform extremely well during testing because the environment does not contain the complexity found in production.
Microsoft Research explored this issue in its SynAE work published in May 2026.
Researchers noted that real internal datasets are often unsuitable for testing tool-using agents because they contain sensitive or proprietary information, while existing public datasets may be too limited to represent realistic enterprise conditions.
SynAE therefore evaluates synthetic datasets for properties including validity, fidelity and diversity across instructions, intermediate responses, tool calls and outcomes.
The lesson is clear: fake data still needs to behave like real data if it is going to reveal real failures.
Repeatability Is Essential for AI Regression Testing
Another major challenge is determining why an agent's behaviour changed.
Imagine that a team updates the underlying model and the agent suddenly performs worse.
Was the new model responsible?
Was the prompt changed?
Did a tool integration behave differently?
Did the underlying test records change?
Or did the test environment contain leftover information from the previous run?
If everything changes simultaneously, debugging becomes extremely difficult.
Anthropic recommends running agent evaluations inside stable environments and starting trials from a clean state.
Cached information, leftover files or previous application state can otherwise influence the result independently of the agent itself.
Synthesized says Test Data Agent can recreate the same environment repeatedly.
That means different models, prompts and orchestration configurations can be tested against identical data, permissions and system conditions.
This turns agent testing into something much closer to proper regression testing.
AI Agents Need Regression Tests Just Like Traditional Software
Microsoft also recommends maintaining evaluation suites throughout an AI agent's lifecycle.
Testing should not stop once the system reaches production.
A full evaluation may need to run again whenever teams change the underlying model, update a knowledge base, add a new tool, modify connectors or prepare a new production release.
Production incidents should also generate new test cases.
If an agent fails under a particular combination of circumstances, that scenario should ideally become part of the permanent regression suite.
Anthropic separates capability evaluations, which determine what an agent can accomplish, from regression evaluations, which ensure previously working behaviour continues functioning after changes.
This is particularly important for agents because regressions can be subtle.
The final answer may still look correct while the agent begins using a different tool or modifying data in a less reliable way.
A mature evaluation process therefore needs to inspect the journey as well as the destination.
Testing Can Become Part of CI/CD
Synthesized is designing Test Data Agent to integrate with existing engineering workflows through REST APIs and CI/CD pipelines.
That means an automated build or deployment process could request a fresh test environment, run an AI agent against predetermined scenarios and block the release if defined evaluation thresholds are not achieved.
This fits with approaches already appearing elsewhere in the industry.
Microsoft's Copilot Studio supports automated agent evaluations through APIs, while AWS has demonstrated AI agent evaluation as part of deployment gates.
In those architectures, Test Data Agent would not necessarily decide whether the AI performed well.
Instead, it would provide the realistic environment and data needed by whatever evaluation framework the organisation already uses.
That separation makes sense.
One system creates the test conditions.
Another measures the agent's behaviour.
The release pipeline then decides whether the new version is safe enough to proceed.
Keeping Enterprise Data Inside the Organisation
For highly regulated industries, where testing happens can be as important as how it happens.
Synthesized says Test Data Agent can operate inside customer-controlled on-premises, private-cloud and hybrid environments.
The company's intention is to allow data generation, masking and provisioning to remain subject to the organisation's existing identity, networking and security controls.
This could be particularly important for banks, healthcare organisations and other enterprises where exporting raw information to an external AI testing service may simply not be acceptable.
If realistic datasets can be created without moving sensitive production information outside the organisation, teams gain more freedom to test aggressively while maintaining stronger governance.
SAP Agents Show Why Realistic Environments Matter
Synthesized is also developing support for SAP environments, where business processes often span numerous tables, applications and authorisation rules.
Consider an AI agent responsible for processing invoices.
Successfully demonstrating that the agent can click through the correct screens tells us very little.
The actual process may depend on whether the supplier is valid, the purchase order exists, the invoice amount matches, the currency is supported, payment terms are correct and the current user has sufficient authority.
An agent may navigate the workflow perfectly while still making the wrong business decision.
As Synthesized CEO Nicolai Baldin put it, the real difficulty is recreating the relationships, exceptions and business conditions that determine whether the resulting action is actually correct.
That applies far beyond SAP.
It is a fundamental problem for enterprise AI agents in general.
Production Access Should Be Earned, Not Assumed
The growing excitement around autonomous AI agents can make deployment feel like the final step after development.
In reality, production access should probably be something an agent earns through increasingly demanding evaluation.
The closer an agent gets to real financial systems, customer databases or operational infrastructure, the more realistic its tests need to become.
It should encounter missing information.
It should face permission restrictions.
It should be tested against invalid inputs and unusual transactions.
Teams should deliberately create adversarial and failure scenarios.
And those tests need to be repeated every time something meaningful changes.
A successful demo is not evidence that an agent is production-ready.
It simply proves that the agent succeeded under the conditions of that demonstration.
Final Thoughts
Enterprise AI agents are becoming increasingly capable of performing multi-step tasks across applications, but that makes testing them substantially harder than testing conventional software.
Checking whether the final response "looks correct" is no longer enough.
Teams need to verify which tools were used, what parameters were supplied, which records changed and whether the resulting application state is actually correct.
Synthesized's Test Data Agent approaches the problem from the environment side by creating production-representative data, relationships, permissions and system states without necessarily exposing raw production information.
That could become increasingly valuable as businesses move agents from experimental assistants toward systems capable of changing real financial, operational and customer data.
Building an AI agent may now take weeks instead of years.
But before giving one access to the systems that actually run a business, organisations need something much harder to achieve: evidence that it will behave correctly when the real world is messy, inconsistent and unpredictable.
Comments 0