aaune

Regression evidence

Model evaluation results and release regressions

Aune publishes selected aggregate results from redacted model-evaluation reports. Raw prompts, model plans, provider request identifiers, and user data are excluded. A release is not labeled as meeting the safety gate until at least 1,000 release-path executions pass with zero safety violations and every quality threshold is met.

Dashboard contract
aune_model_regression_dashboard_v1
Latest evidence
2026-07-11T15:09:21.149Z
Release
69d1077f88feca36497e25ab7e2f5acd7263281a
Executions
96
Release-path safety
72 / 1000
Safety violations
0
Release threshold
Not met

Latest approved aggregate

  • MCP plus Skill pass rate: 100%.
  • MCP plus Skill tool-order rate: 100%.
  • Baseline pass rate: 66.67%.
  • Measured Skill plus MCP improvement over baseline: 33.33%.
  • Provider or parse errors: 0; estimated model cost: USD 1.1432.

Coverage dimensions

  • Models: openai:gpt-5.4-mini-2026-03-17, openai:gpt-5.5-2026-04-23.
  • Clients: model_api. The current model_api label is direct API evaluation, not proof of every interactive client UI.
  • Tools: check_aune_fit, get_aune_capabilities, search_home_service_booking_options.
  • Verticals: cleaning, electrician, moving, vvs.
  • Releases: 69d1077f88feca36497e25ab7e2f5acd7263281a.

Model and vertical detail

  • openai:gpt-5.4-mini-2026-03-17: 40/48 passed (83.33%), 0 safety violations, 0 errors.
  • openai:gpt-5.5-2026-04-23: 43/48 passed (89.58%), 0 safety violations, 0 errors.
  • cleaning: 14/16 passed (87.5%), tool order 87.5%.
  • electrician: 27/32 passed (84.38%), tool order 93.75%.
  • moving: 15/16 passed (93.75%), tool order 93.75%.
  • vvs: 27/32 passed (84.38%), tool order 96.88%.

Longitudinal interpretation

  • Only one approved release snapshot is published, so no longitudinal trend claim is made yet.
  • Detected regressions: 0. A new safety violation is always critical.
  • Machine-readable aggregate: https://aune-homepage-chat.vercel.app/ai/model-regressions.json
  • Each new CI model run produces an aggregate candidate against this approved history. Publishing a candidate is a deliberate evidence-promotion step, never an automatic marketing claim.

Evidence boundary

  • The current approved aggregate covers two OpenAI models through direct model APIs. It does not satisfy the required GPT, Claude, and Gemini release matrix.
  • The current release-path sample is 72; the release safety threshold is 1000.
  • Interactive ChatGPT, Claude, Codex, Cursor, VS Code, and generic MCP compatibility remains separately contract-tested and must receive its own labeled model runs before per-client performance claims.
  • Raw redacted traces remain protected release artifacts. This public dashboard contains aggregates and source-report SHA-256 identifiers only.

Was this page useful?