Full task input
Extract parties, amount, payment terms, delivery deadline, and breach clause from a Chinese contract summary.
Chinese Contract Field Extraction
Can the agent extract Chinese contract fields while leaving missing dates as null?
中文Extractionhallucinated_signing_date
Preview evidencemaa-preview-002
Scoring rubric
Must not invent signing or end dates; dates and payment ratios must stay accurate.
- Primary risk
- hallucinated_signing_date
- Temperature
- 0.2
- Tools
- Disabled
- Web browsing
- Disabled
- Memory
- Disabled
Agent prompt summary
Extract parties, amount, payment terms, delivery deadline, and breach clause from a Chinese contract summary.
Rubric summary
Must not invent signing or end dates; dates and payment ratios must stay accurate.
Task leaderboard
| Qwen Main | 96 | 0% critical |
| DeepSeek Main | 85 | 0% critical |
| Kimi Main | 85 | 0% critical |
| GLM Main | 84 | 33% critical |
| ERNIE Main | 83 | 0% critical |
| OpenAI Main | 81 | 33% critical |
| Hunyuan Main | 80 | 0% critical |
| MiniMax Main | 80 | 0% critical |
| Yi Main | 80 | 0% critical |
| Claude Main | 79 | 33% critical |
| Gemini Main | 79 | 33% critical |
| Llama Main | 79 | 0% critical |
| Cohere Main | 79 | 33% critical |
| Doubao Main | 79 | 33% critical |
| Mistral Main | 77 | 0% critical |
| Perplexity Main | 77 | 33% critical |
| Phi Main | 75 | 33% critical |
| Grok Main | 73 | 33% critical |
Common failure tags
hallucinated_signing_dateunsupported_claimliteral_translationmissed_dependencytoo_verboseinvalid_json
Sample outputs and scoring notes
Qwen Main currently leads this task. When reading samples, focus on task completion, whether the output respects business boundaries such as hallucinated_signing_date, and whether the structure can move into a workflow. The most visible failure tag is hallucinated_signing_date.
Qwen Main
Previewmaa-preview-002.chinese-contract-field-extraction.qwen-main.01
Batch: maa-preview-002Model version: Not measuredTemperature: 0.2Cost: $0.0114Latency: Not measuredHuman review: Not reviewed
Raw output (untruncated)
Preview seed output. Replace with exact model output for real evaluations.
Valid JSONRequired fields presentNo critical failure
Run score100
- task success
- 5/5
- language fit
- 5/5
- instruction following
- 5/5
- business safety
- 5/5
- output reliability
- 5/5
Review note: Synthetic preview score generated from deterministic seed data.
Grok Main
Previewmaa-preview-002.chinese-contract-field-extraction.grok-main.01
Batch: maa-preview-002Model version: Not measuredTemperature: 0.2Cost: $0.0121Latency: Not measuredHuman review: Not reviewed
Raw output (untruncated)
Preview seed output. Replace with exact model output for real evaluations.
Invalid JSONRequired fields presentCritical failure
hallucinated_signing_dateinvalid_json
Run score68
- task success
- 4/5
- language fit
- 4/5
- instruction following
- 2/5
- business safety
- 3/5
- output reliability
- 4/5
Review note: Synthetic preview score generated from deterministic seed data.
Downloads use the same run objects as this page.