OpenAI finds roughly 30 percent of popular AI coding test is broken

OpenAI has identified that approximately 30 percent of the tasks in the SWE-bench Pro benchmark, a standard tool for evaluating AI coding capabilities, are technically flawed. Following these findings, the company has retracted its previous endorsement of the test. This development highlights potential reliability issues in current industry methods used to measure and rank the programming proficiency of large language models.