Valid JSON is only the first gate
A parsable JSON object is necessary, but it does not prove the fields are correct. Reliable extraction also needs schema coverage, missing-field discipline, date normalization, and resistance to hallucinated values.
- Check parse success separately from field accuracy.
- Force explicit nulls or notes for missing values.
- Reject trailing commentary when the downstream system expects raw JSON.
How to stress test structure
Mix clean examples with messy emails, partial invoices, vague support messages, and contradictory dates. The agent should not invent what the source does not contain.
Production guardrails
Use schema validation, retry policies, confidence flags, and audit logs. When the data triggers money movement, legal decisions, or customer promises, keep human review in the path.
How to reuse the method
evaluation methodology can become a small internal evaluation. The point is not to create the largest task set. The point is to cover real work, real risk, and real output formats.
- Define unacceptable failures first.
- Prepare representative samples next.
- Compare candidates with one shared rubric.
What evidence to keep
Save the input, prompt version, model version, run date, raw output, human rating, and failure tags. This makes future retesting and stakeholder review much easier.
Pre-launch checklist
Before using this method in production, run a small retest with real inputs, edge cases, and a plan for what happens when the agent fails.
- Is there a clear human-review rule?
- Are model version and evaluation date recorded?
- Which outputs are not allowed to be sent or written automatically?
- Is there a fallback path when the agent fails?
A practical next step
If you are evaluating this method, start with ten real samples: three normal cases, three edge cases, two high-risk cases, and two cases with strict language or formatting requirements. Run two or three candidate agents and compare quality, repair time, and critical failures.