Tagged "tokenization"
What is a token in an LLM?
A token is the unit of text a language model reads and writes. Learn how tokenizers split text, why token counts differ by language, and why it matters for cost.
LLM foundationsBeginner4 min
A token is the unit of text a language model reads and writes. Learn how tokenizers split text, why token counts differ by language, and why it matters for cost.