分词器v1发布候选版
该分词库正在进行一次重大的更新，采用基于位流的编码和解码方法，重点在于提高性能和效率。此次更新旨在使分词成为机器学习工作流程中的更重要组成部分，可能会加速或减缓机器学习过程。
01
证据
- HHugging Face Blog公司披露2026-09-21
Results What V1 Is The Split: Bitstreams Instead Of A Regex The Word Cache The Merge Loop Method What This Adds Up To Getting It Progress Towards V1 Release Candidate: Implemented 1.0.0 After 1.0.0 The tokenizer has not historically been the bottleneck within ML workflows. Compu…
- HHugging Face Blog公司披露2026-09-21
This is why we have chosen to heavily focus on performance for the upcoming version 1 of tokenizers. Tokenization should be light and should scale with your workflow.