Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale

Researchers have analyzed how large language models perform when asked to retrieve information from massive, million-token document sets placed directly in their context window. The study suggests that while models can bypass traditional vector-based search methods by processing entire corpora at once, their reliability at this scale remains inconsistent. This research highlights the current technical limitations of using expansive context windows as a primary retrieval system for high-volume data.