← Back to Model Beat
Models·6d ago·all news from September 9, 2026

transformers v5.17.0 — Release 5.17.0

# Release v5.17.0 ## New Model additions ### HYV4 Hy4-Preview is a 780B-parameter mixture-of-experts language model that activates 49B parameters per token. Each MoE layer holds 256 routed experts plus one always-active shared expert and routes every token to 8 of them. The context window is 1M tokens. The architecture combines four features: - **Multi-head Latent Attention (MLA)** compresses keys and values into a low-rank latent () that expands back to one key/value per query head. - **DeepSeek Sparse Attention (DSA)** selects keys per query with a lightweight indexer. Following [IndexShare](https://huggingface.co/papers/2603.12201), only the layers marked in run an indexer; layers reuse the previous full layer's selection. - **Gated MLA with learnable attention sinks**, where each head owns a sink logit that participates in the softmax and contributes no value, as in [GPT-OSS](./gpt_oss). - **Independent Hyper-Connections (iHC)** replace the plain residual path with parallel residual streams that are collapsed before, and…

Covered by 1 source

Related stories

ModelsUS Says Alibaba, DeepSeek Have ‘Systematically’ Siphoned AI ModelsSep 8 · 37 sourcesModelsMistral AI Raises €3 Billion With Samsung Leading the RoundSep 8 · 102 sourcesModelsNew Deepseek model V4.1-Flash cuts memory needs for AI agentsSep 8 · 29 sourcesModelsGPT-6 Astra: The next generation in intelligence for workSep 7 · 8 sources