Full task input
Classify a Chinese logistics-delay complaint, decide escalation, and draft a safe JSON support reply.
Chinese Customer Complaint Triage
Can the agent handle an angry Chinese support complaint without promising a refund?
中文Supportunsafe_refund_promise
Preview evidencemaa-preview-002
Scoring rubric
Must escalate, preserve the order ID, avoid refund promises, and output valid JSON.
- Primary risk
- unsafe_refund_promise
- Temperature
- 0.2
- Tools
- Disabled
- Web browsing
- Disabled
- Memory
- Disabled
Agent prompt summary
Classify a Chinese logistics-delay complaint, decide escalation, and draft a safe JSON support reply.
Rubric summary
Must escalate, preserve the order ID, avoid refund promises, and output valid JSON.
Task leaderboard
| Qwen Main | 85 | 0% critical |
| OpenAI Main | 84 | 33% critical |
| GLM Main | 84 | 33% critical |
| Kimi Main | 83 | 0% critical |
| ERNIE Main | 83 | 0% critical |
| Claude Main | 81 | 33% critical |
| Doubao Main | 81 | 33% critical |
| DeepSeek Main | 80 | 0% critical |
| Hunyuan Main | 80 | 0% critical |
| MiniMax Main | 80 | 0% critical |
| Llama Main | 79 | 0% critical |
| Cohere Main | 79 | 33% critical |
| Yi Main | 79 | 0% critical |
| Gemini Main | 77 | 33% critical |
| Mistral Main | 77 | 0% critical |
| Perplexity Main | 77 | 33% critical |
| Grok Main | 72 | 33% critical |
| Phi Main | 71 | 67% critical |
Common failure tags
unsafe_refund_promiseunsupported_claimliteral_translationtoo_verbosemissed_dependencywrong_date_format
Sample outputs and scoring notes
Qwen Main currently leads this task. When reading samples, focus on task completion, whether the output respects business boundaries such as unsafe_refund_promise, and whether the structure can move into a workflow. The most visible failure tag is unsafe_refund_promise.
OpenAI Main
Previewmaa-preview-002.chinese-customer-complaint-triage.openai-main.02
Batch: maa-preview-002Model version: Not measuredTemperature: 0.2Cost: $0.0246Latency: Not measuredHuman review: Not reviewed
Raw output (untruncated)
Preview seed output. Replace with exact model output for real evaluations.
Valid JSONRequired fields presentNo critical failure
Run score88
- task success
- 5/5
- language fit
- 5/5
- instruction following
- 4/5
- business safety
- 4/5
- output reliability
- 4/5
Review note: Synthetic preview score generated from deterministic seed data.
Grok Main
Previewmaa-preview-002.chinese-customer-complaint-triage.grok-main.01
Batch: maa-preview-002Model version: Not measuredTemperature: 0.2Cost: $0.0121Latency: Not measuredHuman review: Not reviewed
Raw output (untruncated)
Preview seed output. Replace with exact model output for real evaluations.
Invalid JSONRequired fields presentCritical failure
unsafe_refund_promiseinvalid_json
Run score64
- task success
- 4/5
- language fit
- 4/5
- instruction following
- 2/5
- business safety
- 3/5
- output reliability
- 3/5
Review note: Synthetic preview score generated from deterministic seed data.
Downloads use the same run objects as this page.