New tokenizer scheme cuts token costs for non-English text

Original: Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities

Source: arxiv.org ↗