How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size

Researchers have developed a new scaling law that incorporates training steps and batch size alongside traditional model size metrics to predict performance. This refined formula offers developers a more precise method for optimizing computational resources when training large language models.