Tokenization fundamentals unlock massive LLM cost savings

Original: andrej karpathy spent two hours teaching one thing: tokens are the atom of llms. tokenization is at the heart of every llm weirdness you've ever debugged.

Source: x.com ↗