What is the difference between tokens and words?

Tokens are subword units that may be smaller or larger than words. A common English word like "the" is one token, but "tokenization" might be split into "token" and "ization". Punctuation and spaces are also typically separate tokens. On average, English text tokenizes to roughly 1.3 tokens per word, but this varies widely by content. ---