What changed
GitHub introduced ReviewBench on October 5 as an open benchmark for AI code review. The important question is whether review catches consequential defects.
Why it matters
Comment count is a weak proxy for a useful review. A verbose reviewer can create more work while missing a subtle permission bug. Evaluation should distinguish actionable findings from cosmetic suggestions, measure false alarms and check whether a proposed fix introduces another problem. Repository context changes the answer.
What to check next
Build a small review set from historical pull requests with known outcomes. Score severity, location accuracy and false positives, then ask a maintainer how much work the review saved. Keep benchmark results separate from real-world results on your code.
Event dates refer to the original announcement. Publication dates show when this article first appeared on Codorc.
