The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought

Researchers have identified that transformer models using low-precision softmax attention can match the expressive power of systems previously thought to require higher precision or hardmax functions. This finding confirms that common architectural choices in modern language models are theoretically capable of complex reasoning, such as chain-of-thought processing, without needing the idealized components typically required in academic proofs.