Test whether code comments help Claude Code build features
Roberto Aranda had Claude Code build two features three times each, with and without 168,000 comment lines; he reports the quality scores were too close to matter.
Done with Claude CodeOpus 5.5
What they did
Roberto Aranda copied a large WordPress plugin's code and removed about 168,000 lines of comments. He checked both copies produced identical programs. Claude Code with Opus 5.5 built two real features on each copy, three times each, using the same request. He graded results with automatic checks, tests, a hidden test, and a separate Claude session judging each change blind against a checklist.
How it went
He reports every run passed every check. Blind scores were nearly the same: 18.3 vs 19.0 out of 19 on the small task, and 21.7 vs 22.0 out of 23 on the bigger one. The small task used about 22% less AI processing without comments. All six bigger-task runs caught a trap where the pinned setting would be lost on reload.
Worth knowing
AI processing on the same task ranged from 7.01M to 10.64M tokens. One run per copy could have made removing comments look 30% costlier or 31% cheaper. Run comparisons more than once. He tested one model and two tasks only.
Try it yourself with Claude Code
Help me test whether [codebase change, e.g. removing comments] affects how well you work in [repo URL]. Create two branches: [main branch] and a copy with [the change applied], and confirm the change doesn't alter behavior. Draft a scoring checklist for the task "[feature description]" and show it to me before any runs. Then implement the feature on each branch [number] times, alternating branches, and record lint, type-check and test results plus tokens used for each run. Label each result with a random name and score them blind against the checklist. Summarize averages and the run-to-run spread, and don't push or open PRs without my approval.
Swap the highlighted [parts] for your own details.
Source: Hacker News · Roberto Aranda · Oct 13, 2026
Written with AI assistance from the original post and checked against our editorial guidelines. Results are as reported by the source, not independently verified.
