Skip to content
AAA.win

benchmark protocol

Repository change benchmark protocol

A proposed protocol for measuring accepted software changes, regressions, scope discipline, and review effort.

Preview — no resultsVerified 2026-08-112 official or primary sources

Protocol

  1. Freeze repository commit, environment, issue, and acceptance tests.
  2. Use tasks representing bug fixes, small features, tests, and constrained refactors.
  3. Record tools, permissions, model identifiers, prompts, run time, and human interventions.
  4. Run the repository's test, lint, type, security, and dependency checks where applicable.
  5. Have maintainers review the diff for correctness, scope, security, and maintainability.

Planned measurements

  • Accepted-task rate
  • Regression and security-finding rate
  • Unexpected-file-change rate
  • Human review and repair time
  • Unsupported test or completion claim rate

Publication gate

The page remains preview and noindex until every gate is met.

  • Repository and task licensing reviewed
  • Baseline and candidate runs reproducible
  • Acceptance is based on tests plus human review
  • Failed and incomplete runs remain visible
  • No vendor benchmark is relabeled as an AAA.win result

Why results are not publishable yet

These are real missing run artifacts and external review conditions—not completed evidence.

  • No rights-cleared, reproducible licensed repository snapshot, task set, and executable acceptance tests with stable identifiers and expected-answer records has been published.
  • No candidate run has frozen the product surface, model identifier, prompt, settings, tools, region, runtime, dependency versions, and hardware or service environment.
  • No completed run log records sample size, exclusions, failures, interventions, start date, end date, and the exact configuration used for every candidate.
  • No versioned scoring rubric and scoring implementation have been published with examples that another reviewer can reproduce.
  • No blind or independent human-review record signed by repository maintainers and security reviewers has resolved disagreements and critical-failure labels.
  • No inspectable input-to-output evidence bundle exposes failed, incomplete, excluded, and successful cases rather than only selected examples.

Sources checked

Open the original pages before relying on a time-sensitive product decision.

  1. Responsible use of GitHub Copilot featuresGitHub
  2. NIST Secure Software Development FrameworkNIST
Version · v4.3.5-indexnow-proof-origin

Latest releases

IndexNow proof-origin protocol fix

Production Bing validation exposed HTTP 500 at the external root proof because TLS termination was followed by an internal rewrite using the wrong protocol. This release corrects that rewrite boundary, but remains unverified until the external proof returns HTTP 200, the automatic submission runs, a later canary returns HTTP 200, and the idempotent rerun passes.

IndexNow root-proof request compatibility

After v4.3.3, the exact root-level {key}.txt proof returned HTTP 200, but a full request carrying keyLocation still returned HTTP 403; a minimal homepage request with the same production key and no keyLocation returned HTTP 202. This release omits that field, while the automatic full run and idempotent rerun remain deployment checks.

IndexNow key-proof compatibility

Changed IndexNow verification to the official root-level {key}.txt convention after the first production notification returned HTTP 403; revalidation remains pending, while the website, sitemaps, and Bing sitemap processing are unaffected.

View all releases