IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining

Apple researchers have introduced IDEA Prune, a training pipeline designed to improve the efficiency of generative language models by integrating pruning directly into the pretraining process. This approach aims to produce smaller, deployable models that achieve better performance than those trained to a specific size from scratch. By optimizing the architecture during development, the method offers a way to reduce the computational resources required for running complex language models on devices with strict hardware constraints.