Protocol
- Freeze repository commit, environment, issue, and acceptance tests.
- Use tasks representing bug fixes, small features, tests, and constrained refactors.
- Record tools, permissions, model identifiers, prompts, run time, and human interventions.
- Run the repository's test, lint, type, security, and dependency checks where applicable.
- Have maintainers review the diff for correctness, scope, security, and maintainability.
Planned measurements
- Accepted-task rate
- Regression and security-finding rate
- Unexpected-file-change rate
- Human review and repair time
- Unsupported test or completion claim rate
Publication gate
The page remains preview and noindex until every gate is met.
- Repository and task licensing reviewed
- Baseline and candidate runs reproducible
- Acceptance is based on tests plus human review
- Failed and incomplete runs remain visible
- No vendor benchmark is relabeled as an AAA.win result
Why results are not publishable yet
These are real missing run artifacts and external review conditions—not completed evidence.
- No rights-cleared, reproducible licensed repository snapshot, task set, and executable acceptance tests with stable identifiers and expected-answer records has been published.
- No candidate run has frozen the product surface, model identifier, prompt, settings, tools, region, runtime, dependency versions, and hardware or service environment.
- No completed run log records sample size, exclusions, failures, interventions, start date, end date, and the exact configuration used for every candidate.
- No versioned scoring rubric and scoring implementation have been published with examples that another reviewer can reproduce.
- No blind or independent human-review record signed by repository maintainers and security reviewers has resolved disagreements and critical-failure labels.
- No inspectable input-to-output evidence bundle exposes failed, incomplete, excluded, and successful cases rather than only selected examples.
Sources checked
Open the original pages before relying on a time-sensitive product decision.