Presentation: From S3 to GPU in One Copy: Rethinking Data Loading for ML Training

Vortex is an open-source columnar file format designed to accelerate machine learning training by streamlining data transfer from cloud storage to GPUs. By utilizing zero-copy memory pipelines and advanced encoding techniques, the format removes common CPU and storage bottlenecks that typically slow down data ingestion. This approach allows for higher throughput when streaming information directly from S3, potentially reducing the time required to prepare datasets for large-scale model training.