What is tokenization in LLMs?

Tokenization is the process of converting raw text into tokens, the discrete units that the model processes. Tokens are usually subword units, not full words or characters. The tokenizer assigns each token a numeric ID from a fixed vocabulary, typically containing 30 thousand to 200 thousand entries. Every input must be tokenized before the model can process it.