Perplexity open-sources faster tokenizer cutting CPU usage by 5x

Original: We're open-sourcing the Unigram tokenizer we rebuilt to reduce CPU utilization by 5-6x.

Source: github.com ↗