deployment
Tokenizer
A component that converts raw text into a sequence of tokens that a language model can process. Tokenizers split text into subword units using learned vocabularies, balancing between character-level and word-level representations.
In practice
The GPT-4 tokenizer splits 'unbelievable' into tokens like 'un', 'believ', 'able' based on frequency patterns.