Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers

Gigatoken is a new MIT-licensed Rust-based Byte Pair Encoding tokenizer capable of processing text at 24.53 GB/s. By significantly outperforming established libraries like HuggingFace Tokenizers and tiktoken in high-core environments, this tool provides a more efficient way to handle large-scale data preprocessing for machine learning workflows.