Single prompts are only the beginning
A production agent may retrieve policy, call tools, write JSON, ask follow-up questions, and hand off to a human. Evaluation should follow that workflow instead of testing one isolated answer.
- Score each step and the final workflow outcome.
- Record where the agent needed tools or human review.
- Treat validation failures as workflow failures, not cosmetic issues.
What to log
Log the input, prompt version, model version, retrieved context, tool calls, raw output, validator result, human correction, and final user-visible answer.
How AAA.win can grow here
AAA.win can use static tasks as the evidence layer, then add richer workflow traces as the site matures. This gives readers both readable rankings and deeper operational proof.
How to reuse the method
evaluation methodology can become a small internal evaluation. The point is not to create the largest task set. The point is to cover real work, real risk, and real output formats.
- Define unacceptable failures first.
- Prepare representative samples next.
- Compare candidates with one shared rubric.
What evidence to keep
Save the input, prompt version, model version, run date, raw output, human rating, and failure tags. This makes future retesting and stakeholder review much easier.
Pre-launch checklist
Before using this method in production, run a small retest with real inputs, edge cases, and a plan for what happens when the agent fails.
- Is there a clear human-review rule?
- Are model version and evaluation date recorded?
- Which outputs are not allowed to be sent or written automatically?
- Is there a fallback path when the agent fails?
A practical next step
If you are evaluating this method, start with ten real samples: three normal cases, three edge cases, two high-risk cases, and two cases with strict language or formatting requirements. Run two or three candidate agents and compare quality, repair time, and critical failures.