In the end LLM are just predicting what is statistically the most likely follow-up for a sentence, according to what they’ve been trained on. That’s how they work.
For this they cut sentences into bits called tokens. The text they are working on is a sequence of tokens, and they predict the next one.
A token could be a whole word (generally if it’s short and usual), often it’s a syllable, and if you’re trying to use uncommon words/names they might even cut them down letter-by-letter in part to try and fit them into something they can swallow.
Of course tokens would be very language dependent, and some languages just use more characters/longer words and sentences than others on average.
In the end LLM are just predicting what is statistically the most likely follow-up for a sentence, according to what they’ve been trained on. That’s how they work.
For this they cut sentences into bits called tokens. The text they are working on is a sequence of tokens, and they predict the next one.
A token could be a whole word (generally if it’s short and usual), often it’s a syllable, and if you’re trying to use uncommon words/names they might even cut them down letter-by-letter in part to try and fit them into something they can swallow.
Of course tokens would be very language dependent, and some languages just use more characters/longer words and sentences than others on average.