An agent saying done is not an acceptance test

Microsoft and Hugging Face's 22 October ThinkingBox Training article describes using executable checks of final backend state and side effects as evaluation signals and training rewards. It is research infrastructure, not a ready-made guarantee for a business integration. We focus on the method rather than reproducing headline gains. (Microsoft / Hugging Face)

The framework supports isolated tool environments and agent-outcome checks. The training repository describes a specialist training stack; its actual LICENSE.TXT is MIT. That code licence does not settle the rights or operating requirements of every dependency, dataset or model used with it. (Microsoft, Microsoft, Microsoft)

Turn a task into observable invariants

Consider a hypothetical CRM request: create an enquiry, assign it to the correct queue and prepare a follow-up for review. The evaluator should inspect the resulting records and permissions, not reward a fluent summary or a particular sequence of tool calls.

Design the test data before running the agent

Start with synthetic records that make the expected result unambiguous. Include an existing matching enquiry, missing required evidence, a permission denial and an interruption after a partial write. Keep the expected state separate from instructions shown to the agent so the check does not merely confirm that it echoed the answer.

Illustrative evaluator structure:
initialise isolated backend with known fixtures
run one bounded task with restricted tools
read terminal records and relevant side effects
check required outcomes AND forbidden changes
record result, tool failures, cost and review notes
reset the environment before the next trial

For tasks involving judgement, define the rubric and review disagreements. A database check can establish that a field changed, but cannot establish that a vague business judgement was correct unless the acceptance definition covers it. Avoid hiding a subjective judgement inside a binary success flag.

Repeatability and failure cost deserve separate columns

A task solved once may fail on another run, while repeated trials can expose duplicate actions, inconsistent routing or tool-error handling. Choose the repetition plan in advance and show both overall outcomes and per-case failures. Do not read 'solved at least once' as the probability that the next unattended run will be safe.

• Report incorrect state, missing action, unauthorised action and evaluation failure separately.
• Track the cost per attempt and human intervention, not only the count of successful runs.
• Preserve prompts, model/tool versions and fixtures so a later change can be compared.
• Use held-out cases when adjusting prompts, tools or training so improvement is not just memorisation.

Inspect the repository before adopting the stack

Decide whether you need a full training pipeline or a smaller application-level acceptance suite. Review the documented hardware/runtime dependencies, maintenance, licence files and isolation model before executing anything. Repository popularity is not evidence that a stack fits a small team's workload.

The immediate operational lesson is modest: specify how you will know the work is done before granting write access. Custom software and internal tools can incorporate such checks around a defined workflow. No ThinkingBox benchmark or reliability result is claimed for KIURO or the illustrative CRM task.