TB TripleBench

Pitfall Lab · run-002 · n = 1

Same invisible .env bug, three AI fixes — diff hygiene decided it

This bug has probably bitten you without you knowing: an invisible character in a config file. No error, no crash — your app just can't read your config. We had Codex, Claude Code and Gemini CLI fix it against a hidden acceptance gate they never saw.
Last round was about who fixes fastest. This round is about who finishes clean.

The bug, in plain words

Ever edited a .env file in Windows Notepad? On save it may silently prepend 3 bytes — a marker called a BOM — think of it as an encoding hint some editors write at the start of a file. Completely invisible. But when your app reads the config, the first variable name becomes \ufeffAPI_KEY — that \ufeff prefix is the invisible character instead of API_KEY.

Result: no error, no crash, and your app just cannot find API_KEY. This kind of issue can burn hours because nothing looks obviously broken.

EFBBBF46 'F'49 'I'52 'R'53 'S'54 'T'3D '='…

The three red bytes are the BOM — invisible in your editor. This is a real bug in python-dotenv, a widely used third-party config loader for Python (upstream PR #640).

The setup

1

Same starting point
pinned to the commit before the official fix
fa4e6a90

2

Same prompt
all three got the identical task, word for word

3

Hidden gate
5 hidden checks they never saw, revealed only after they finished

Scoreboard

The scoreboard below describes only this one task, not the tools' overall capability.

Codex

hidden gate ✓
time1m47sfastest
hygieneAcleaned up after itself

API-level regression test

Hit a broken test environment, found a workaround on its own, and deleted the lockfile it generated before handing in the diff.

Claude Code

hidden gate ✓
time2m25smiddle
hygieneA+cleanest diff

parser-level regression test

Touched exactly the two files that needed touching, slotted the test into the existing parametrized table, and left a comment explaining why.

Gemini CLI

hidden gate ✓
time3m27sslowest (one platform retry)
hygieneBleft unrelated files behind

new test file, 3 scenarios

Correct fix and the broadest test coverage — but the diff shipped with an unrelated uv.lock and trailing-whitespace noise.

Times are operator-observed wall clock for the local CLI process, not a normalized compute benchmark. Gemini's run includes one "429 no capacity" platform retry.

The fun part: all three changed the same line

The core fix is identical — when reading the file, strip the invisible character if it leads:

class Reader:
    def __init__(self, stream: IO[str]) -> None:
        self.string = stream.read()
+       if self.string.startswith("\ufeff"):
+           self.string = self.string[1:]
        self.position = Position.start()

What actually separated them was the exit work in the diff. Claude Code slotted its regression test into the existing test table with a comment. Codex tested at the API layer and cleaned up its temp files. Gemini wrote the broadest new test file — and accidentally committed an unrelated uv.lock along with it.

Exit checklistCodexClaude CodeGemini CLI
Core fix correct✅✅✅
Regression test included✅✅✅ 3 entry points
Diff contains only related changes✅✅⚠️ stray uv.lock
No formatting noise✅✅⚠️ trailing whitespace
Temp files cleaned up✅ deleted proactively— (none created)❌ left behind

What the hidden gate checked

After the run, we applied 5 hidden checks. Here are the 3 BOM behavior cases that are easiest to read:

  1. BOM + single key — must parse a clean FIRST; a key with the invisible prefix is an instant fail
  2. BOM + multiple keys — keys after line one must be unaffected
  3. No BOM at all — guards against over-fixing normal behavior

The other two checks were Unicode value preservation and the parser test suite — guarding against a fix that strips legitimate Unicode or regresses existing parsing behavior.

All three passed all five. No disaster story this round — and that's exactly why cleanliness became the real differentiator.

This round's takeaway

Code that runs ≠ code you can merge.

All three got the job done. Whether the result was clean is what decides if you'd merge it without reading every line. If you remember one thing: after an AI says "fixed", check its diff for files that have no business being there. A ten-second look at the diff is usually cheaper than a rollback.

Evidence & reproduction

  • Real repo: theskumar/python-dotenv, pinned at fa4e6a90b45428212452afc6ee0d5c8103b9301d (before official fix PR #640)
  • Full diffs, hidden gate cases and the timeline are above; scoring follows Methodology v1
  • Raw terminal logs are archived — open an issue on GitHub to request them or to challenge a result

Single task, n = 1. This is not an overall ranking. Previous round: run-001 · One bug, three agents, one hidden test