Arena #2

Methodology

AAA.win evaluates specific agents on specific multilingual business tasks under documented conditions.

This page explains how AAA.win turns multilingual business work into reviewable agent evaluations: how tasks are selected, how runs are controlled, how scores should be read, and where the current preview should not be over-claimed.

Method

Methodology

18agents20tasks1080runs

18agents

20tasks

4languages

1080runs

Core principles

Scoring

Each run is scored on task success, language fit, instruction following, business safety, and output reliability.

Runs

Each agent runs each task 3 times with tools, browsing, and memory disabled.

Critical failures

A critical failure is unsafe, misleading, unusable, or structurally invalid in a real business workflow.

Vendor policy

Vendors cannot pay to change scores. Sponsored placements, if introduced later, will be labeled separately.

Evaluation process

Collect real tasks

Tasks come from recurring business workflows such as support, writing, and structured extraction.

Convert tasks into runnable prompts

Each task keeps metadata for language, task type, difficulty, and primary risk.

Run each task-agent pair repeatedly

Repeated runs reveal stability instead of rewarding one lucky output.

Score and tag failures

Each run receives a score, critical-failure flag, format check, failure tags, and judge notes.

Generate reports and review them

Reports are generated from structured data, then reviewed before strong public claims.

Scoring dimensions

Task success

Did it solve the real job?

The output must complete the task, not merely sound fluent.

Answers the core request.
Preserves constraints.
Produces usable output.

Language fit

Would a local team send it?

Language evaluation includes tone, format, politeness, and market convention.

Natural local phrasing.
Correct date and support conventions.
No literal localization.

Instruction following

Can it respect format and boundaries?

Many failures come from ignoring schema, fields, or role limits.

Valid JSON.
Complete required fields.
Concise when requested.

Business safety

Does it avoid dangerous promises?

Unsafe refunds, invented security claims, and false contract data matter more than style.

No unauthorized commitments.
No fabricated facts.
Escalates when needed.

Reliability

Can downstream systems use it?

Automation needs stable structure and honest handling of missing information.

Stable dates and numbers.
Nulls when data is missing.
No discussion-as-action errors.

Cost fit

Is the high score worth it?

Buyers should weigh score against cost, risk level, and workflow importance.

Prioritize safety in high-risk flows.
Consider cost in low-risk flows.
Retest with your own tasks.

Critical failures and tags

Failure tags explain what went wrong and where readers should inspect the raw output.

literal_translation

observed seed runs

unsupported_claim

observed seed runs

weak_cta

observed seed runs

unsafe_refund_promise

observed seed runs

missing_field

observed seed runs

too_verbose

observed seed runs

Run conditions

Batch

The public preview uses maa-preview-002.

Runs

Each task-agent pair runs 3 times.

Tools

Tools, browsing, and memory are disabled in the current batch.

Languages

Real task sets currently cover Chinese, English, Japanese, and Spanish.

Limits

The current arena is a useful comparison framework, not a permanent universal model ranking.

Agent names refer to current configurations.
Preview seed outputs should be replaced before strong public claims.
The task set does not cover every industry or regulation.
Close scores should be interpreted by language, task type, and risk.
High-stakes outputs still need human review.

Publication rules

Transparent

Publish batch conditions

Reports should show batch ID, date, tool settings, and runs per pair.

Reviewable

Keep raw evidence

Scores should trace back to prompts, outputs, tags, and notes.

Fresh

Rerun after model changes

Old results should keep dates and be replaced by new batches when conditions change.

Independent

Vendors cannot buy scores

Sponsored placements, if introduced later, must be clearly separated from ranking.