Presentation: Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads
LinkedIn engineer Sundara Raman Ramachandran detailed techniques for integrating large language models into high-traffic search and recommendation systems while maintaining strict latency requirements. By distilling a 7B model into a 400M parameter version and utilizing single-token embeddings for job contexts, the team achieved a 10x increase in throughput. These methods demonstrate how companies can deploy complex AI ranking models within production environments that demand near-instant response times.
Covered by 1 source
- IInfoQ AI↗Sundara Raman Ramachandran5d ago