What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus

Researchers are evaluating terminal-based AI benchmarks to distinguish between genuine performance limitations and failures caused by flawed task design. The study finds that high failure rates often stem from missing context or environmental errors rather than actual model inability. This analysis highlights the need for more rigorous testing criteria to ensure evaluation corpora accurately measure genuine agentic capabilities.