deployment
BPE
Byte Pair Encoding, a subword tokenization algorithm that iteratively merges the most frequent pairs of characters or character sequences. BPE builds a vocabulary of subword units that efficiently represent both common words and rare terms.
In practice
BPE might learn that 'th' and 'e' frequently appear together and merge them into a single 'the' token.