The Cost of Compression: A Rate-Distortion Limit on Factual Hallucination

A new research paper identifies a fundamental trade-off between data compression and factual accuracy in language models. The authors argue that models often hallucinate because the mathematical requirements of compressing vast amounts of information into limited parameters force the system to prioritize patterns over specific facts. This suggests that factual errors are not merely a result of incomplete training data, but an inherent limitation of how current models store and retrieve information.