Case Study · Docket 9f4983d · August 2026

The Council reviews its own bug fix.

We put a real commit from AI Council's own codebase before the bench: a fix its author considered finished and shipped. Five agents voted. All five said revise. They were right. This is the unedited session, condensed only for length.

The case

Commit 9f4983d fixed a real bug: ai-council test, the API connectivity check, was failing against newer "thinking" models. OpenAI's gpt-5.4-mini rejects the legacy max_tokens parameter, and Gemini's gemini-3.6-flash spends its output budget on internal reasoning before emitting text, so a tiny token limit returned an empty response that looked like a failure.

The fix, 18 lines added and 5 removed in packages/cli/src/cli.ts, switched OpenAI to max_completion_tokens, raised Gemini's maxOutputTokens from 10 to 256, and made empty responses throw with a diagnostic finishReason:

-        max_tokens: 5
+        max_completion_tokens: 16

-          generationConfig: { maxOutputTokens: 10 }
+          generationConfig: { maxOutputTokens: 256 }

+        const text = data.candidates?.[0]?.content?.parts?.[0]?.text;
+        if (text) { ... } else {
+          const reason = data.candidates?.[0]?.finishReason ?? "unknown";
+          throw new Error(`Empty response from Gemini (finishReason=${reason})`);
+        }

The author believed this was done. It compiled, it worked, it shipped. Then we asked the Council:

The proceedings

Proceedings docket № 9f4983d
$ ai-council review --commit=9f4983d
  1. Senior Developer Revise 0.90

    "Directionally correct... but it assumes max_completion_tokens is universally accepted. If users set a legacy model that doesn't support it, the test could fail even though the API key and network are fine. Add a fallback path." Also flagged: the Gemini check reads only the first candidate, "a brittle single-candidate assumption."

  2. Security Engineer Revise 0.78

    "The test prints the first 8 characters of the API key when configured. Even partially redacted secrets can aid credential correlation in shared terminals." Also: raw provider error bodies flow into thrown messages; map non-2xx responses to a generic message and keep provider detail behind a debug level.

  3. Performance Engineer Revise 0.86

    "The jump from 10 to 256 output tokens is a meaningful regression in worst-case budget for a command meant to be a quick connectivity check. Revise downward to the minimum that reliably succeeds, or make the larger budget conditional on models known to need it."

  4. Pragmatist Revise 0.87

    "If this CLI supports arbitrary model overrides via AI_COUNCIL_MODEL, this could regress users who point at a model that still only accepts max_tokens. Try max_completion_tokens, and on an invalid-parameter error, retry once with max_tokens."

  5. Gemini Structural Thinker Revise 0.88

    "Gemini thinking models frequently emit multiple elements in content.parts: parts[0] may be a thought block while the text lives in parts[1]. Reading strictly parts[0].text risks a false 'Empty response' error even when valid text exists."

    parts?.find(p => Boolean(p.text))?.text
Final ruling Revise 5 revise · 0 approve · confidence 0.93

From the judge's rationale: "The change is directionally correct and likely fixes the immediate connectivity-test failures... However, I would not approve as-is because there are real production risks. Keep the fail-fast behavior and the larger Gemini budget, but make the parsing and error handling more defensive before shipping."

What the Council caught

  • A latent bug in the fix itself. The Gemini agent's parts[0] finding is the sharpest: the very models this fix targets, thinking models, are the ones most likely to return a thought block first. The fix for empty responses could itself produce false "empty response" errors. That's a bug the author missed while staring directly at the file.
  • A compatibility regression. Three agents independently converged on the same risk: max_completion_tokens isn't accepted by every OpenAI-compatible endpoint (older proxies and self-hosted gateways still require max_tokens), and the CLI explicitly supports arbitrary model overrides.
  • A secret-hygiene slip nobody asked about. The security agent wasn't reviewing the diff in isolation; it flagged the surrounding function printing an 8-character API-key prefix. Specialists read code with their own mandate, not just the changed lines.
  • The cost of the easy fix. The performance agent priced the 25× token-budget increase against the command's purpose. Not wrong to raise it, just wrong to raise it unconditionally.

What this session shows

No single reviewer produced this ruling. Each finding came from a different mandate, and the unanimous REVISE emerged from independent votes, not from one model's pass over the diff. This is the difference between an opinion and a verdict: the Council's judgment is the aggregate of specialists who don't share blind spots, weighted by how confident each one actually is.

A note on method: this is a real, unedited session against a real shipped commit in AI Council's own repository. Agent remarks above are condensed from their full written reasoning; votes, confidence scores, and the ruling are verbatim. The standard review bench seats five of the Council's six agents; the Software Architect sits on the specialized benches convened by the arch, security, and sanity commands.

Put your own docket before the bench.

Every review looks like this: votes, confidence, debate, and one ruling on your diffs, with your keys.

Get License Get Started

Related reading