What changed
IBM Research published work on repeated-run agent consistency on September 15. Its central distinction is between average success and succeeding every time the same task is repeated.
Why it matters
A single successful demonstration can hide unstable decisions. For a developer, this matters when an agent sometimes chooses a different file, skips a check or takes an unnecessary action. An average score cannot tell you whether a specific workflow is dependable. Keep the distribution of outcomes and inspect the divergent traces.
What to check next
Repeat a fixed set of tasks under identical conditions, without showing earlier solutions. Record both average success and the share that succeed on every run. Separate recoverable failures from unsafe actions. Codorc has not reproduced the paper’s experiments; this is a practical evaluation implication.
Event dates refer to the original announcement. Publication dates show when this article first appeared on Codorc.
