Dust: Pretraining Transformers Without Backpropagation

Researchers have introduced Dust, a method for training Transformer models that avoids standard backpropagation by using local loss functions and forward-only signals. This approach aims to address the memory constraints and sequential dependencies typically associated with the backpropagation algorithm. By decoupling layer updates, the technique potentially offers a more hardware-efficient path for scaling neural networks.