BenchMIRT: What are LLM benchmarks actually measuring?

Researchers have introduced BenchMIRT, a framework designed to analyze the underlying linguistic and reasoning capabilities tested by standard large language model benchmarks. By deconstructing common evaluation metrics, this tool reveals the specific patterns and heuristics models rely on to achieve high scores. This work highlights potential gaps between high benchmark performance and genuine task comprehension, offering developers a more granular method to assess whether models are learning intended skills or merely identifying statistical shortcuts.