core concepts
Token
The basic unit of text that language models process. Tokens can be words, parts of words, or individual characters depending on the tokenizer used. Most models process text by converting it into a sequence of token IDs.
In practice
The word 'artificial' might be split into two tokens: 'artific' and 'ial'.
In the index
Tools that mention Token
- SupermavenTrust 79
Matched on each tool’s own description and feature list, highest trust score first.
Briefings