What is Byte Pair Encoding (BPE)?

Byte Pair Encoding is a tokenization algorithm that starts with individual bytes and iteratively merges the most frequent adjacent pairs to form new tokens. The result is a vocabulary that captures common subwords as single tokens and breaks rare words into multiple pieces. BPE is used by GPT models, Llama, and most modern LLMs because it handles any input text, including unknown words.