In the end LLM are just predicting what is statistically the most likely follow-up for a sentence, according to what they’ve been trained on. That’s how they work.
For this they cut sentences into bits called tokens. The text they are working on is a sequence of tokens, and they predict the next one.
A token could be a whole word (generally if it’s short and usual), often it’s a syllable, and if you’re trying to use uncommon words/names they might even cut them down letter-by-letter in part to try and fit them into something they can swallow.
Of course tokens would be very language dependent, and some languages just use more characters/longer words and sentences than others on average.
It’s the smallest unit an LLM can process. If you write a long florid prompt, it’s more tokens and therefore costs more for the LLM to interpret. There’s literally a “caveman” mod where it makes the LLM respond in caveman-speak, which means loss tokens and therefore cheaper. What’s odd (did not RTFA) is that some languages are cheaper, IIRC even has a super-caveman mode where it responds in Chinese/Japanese because that costs fewer tokens.
Tf is a token?
In the end LLM are just predicting what is statistically the most likely follow-up for a sentence, according to what they’ve been trained on. That’s how they work.
For this they cut sentences into bits called tokens. The text they are working on is a sequence of tokens, and they predict the next one.
A token could be a whole word (generally if it’s short and usual), often it’s a syllable, and if you’re trying to use uncommon words/names they might even cut them down letter-by-letter in part to try and fit them into something they can swallow.
Of course tokens would be very language dependent, and some languages just use more characters/longer words and sentences than others on average.
It’s the smallest unit an LLM can process. If you write a long florid prompt, it’s more tokens and therefore costs more for the LLM to interpret. There’s literally a “caveman” mod where it makes the LLM respond in caveman-speak, which means loss tokens and therefore cheaper. What’s odd (did not RTFA) is that some languages are cheaper, IIRC even has a super-caveman mode where it responds in Chinese/Japanese because that costs fewer tokens.