Skip to content
AAA.win
EN
EnglishEN中文ZH日本語JAEspañolES
SearchSubmit a tool
Menu

Japanese Product Specification Extraction

Can the agent extract Japanese product specifications without guessing material?

日本語Extractionhallucinated_material
Preview evidencemaa-preview-002

Full task input

Extract model, dimensions, weight, material, paper size, connection, and warranty.

Scoring rubric

Must keep units, set missing material to null, and output valid JSON.
Primary risk
hallucinated_material
Temperature
0.2
Tools
Disabled
Web browsing
Disabled
Memory
Disabled

Agent prompt summary

Extract model, dimensions, weight, material, paper size, connection, and warranty.

Rubric summary

Must keep units, set missing material to null, and output valid JSON.

Task leaderboard

Qwen Main9133% critical
Gemini Main850% critical
OpenAI Main830% critical
Claude Main830% critical
DeepSeek Main830% critical
Kimi Main8133% critical
Mistral Main800% critical
Cohere Main800% critical
GLM Main800% critical
Hunyuan Main800% critical
Yi Main800% critical
Llama Main790% critical
ERNIE Main790% critical
MiniMax Main790% critical
Doubao Main790% critical
Perplexity Main7733% critical
Phi Main750% critical
Grok Main7267% critical

Common failure tags

hallucinated_materialunsupported_claimliteral_translationtoo_verbosegeneric_ai_copyinvalid_json

Sample outputs and scoring notes

Qwen Main currently leads this task. When reading samples, focus on task completion, whether the output respects business boundaries such as hallucinated_material, and whether the structure can move into a workflow. The most visible failure tag is hallucinated_material.

Qwen Main

maa-preview-002.japanese-product-spec-extraction.qwen-main.01

Preview
Batch: maa-preview-002Model version: Not measuredTemperature: 0.2Cost: $0.0115Latency: Not measuredHuman review: Not reviewed

Raw output (untruncated)

Preview seed output. Replace with exact model output for real evaluations.
Valid JSONRequired fields presentCritical failure
hallucinated_materialliteral_translation
Run score96
task success
5/5
language fit
5/5
instruction following
5/5
business safety
4/5
output reliability
5/5

Review note: Synthetic preview score generated from deterministic seed data.

Grok Main

maa-preview-002.japanese-product-spec-extraction.grok-main.01

Preview
Batch: maa-preview-002Model version: Not measuredTemperature: 0.2Cost: $0.0124Latency: Not measuredHuman review: Not reviewed

Raw output (untruncated)

Preview seed output. Replace with exact model output for real evaluations.
Invalid JSONRequired fields missingCritical failure
hallucinated_materialinvalid_json
Run score64
task success
4/5
language fit
4/5
instruction following
2/5
business safety
3/5
output reliability
3/5

Review note: Synthetic preview score generated from deterministic seed data.

Downloads use the same run objects as this page.

Version · v4.3.6-indexnow-delivery-verified

Latest releases

IndexNow production delivery verification

Verified production IndexNow receipt with a new key: Microsoft's external proof returned HTTP 200, single-URL GET and POST both returned HTTP 200, and eight batches covering 386 canonical URLs each returned HTTP 200 with zero retries. A second run found the inventory unchanged and sent no request.

IndexNow proof-origin protocol fix

Production Bing validation exposed HTTP 500 at the external root proof because TLS termination was followed by an internal rewrite using the wrong protocol. This release corrects that rewrite boundary, but remains unverified until the external proof returns HTTP 200, the automatic submission runs, a later canary returns HTTP 200, and the idempotent rerun passes.

IndexNow root-proof request compatibility

After v4.3.3, the exact root-level {key}.txt proof returned HTTP 200, but a full request carrying keyLocation still returned HTTP 403; a minimal homepage request with the same production key and no keyLocation returned HTTP 202. This release omits that field, while the automatic full run and idempotent rerun remain deployment checks.

View all releases