This was mostly noticed already in the earliest days of LLM spreading (I think several months since GPT went public), when Russian users noticed it has much worse output in Russian than English. The reason was simply that each letter was treated as a whole distinct token in Russian, leading to more difficulties in outputing words without typos. It is logical that this approach would be more resource-expensive and prone to errors, if the AI has to select each and every letter one by one instead of immediately jumping to selecting words and forming the whole sentence.
English is a fairly analytic language, words barely change, and subtleties in meaning and relationships are expressed with other words. Many other languages encode such information inside the word by various ways of modifying the same base form (inflectional languages). The same way English forms plurals by adding -s, so can other languages add a whole bunch of possible endings to express e.g. that the object is used as a tool, or that it’s the recipient of an action. To analyse and tokenise all that stuff would require decades of manual work. Alternatively I would assume there are some automatised tools that would generally correctly split the words into tokens, at least for the major inflectional languages, such as Russian. But most likely they just split it down to random syllables or even individual letters. Indeed, I just tested Russian, Polish and Croatian in their linked tokenizer, and the results are mostly syllabic or letter-based or just sliced into random sections that don’t correspond to anything linguistically relevant. Not a trace of morphemic analysis (morphemes are meaningful units, e.g. level-head-ed-ness, this style of slicing words down into pieces would probably be the ideal approach here). [Edit: holy shit, it actually seems to treat almost every capital letter as a distinct token.]
Or, to put it most simply, AI bullshits and brute-forces through languages with complicated words. And it’s been known for years. Nice to see additional confirmation, of course, and to learn that they still haven’t improved on it.
This wouldn’t make it to the top 20 issues with AI for me.
The same way as using non-English makes it harder to find any data manually in books/forums/etc(except for strictly local data like laws or similar).
Tf is a token?
In the end LLM are just predicting what is statistically the most likely follow-up for a sentence, according to what they’ve been trained on. That’s how they work.
For this they cut sentences into bits called tokens. The text they are working on is a sequence of tokens, and they predict the next one.
A token could be a whole word (generally if it’s short and usual), often it’s a syllable, and if you’re trying to use uncommon words/names they might even cut them down letter-by-letter in part to try and fit them into something they can swallow.
Of course tokens would be very language dependent, and some languages just use more characters/longer words and sentences than others on average.
It’s the smallest unit an LLM can process. If you write a long florid prompt, it’s more tokens and therefore costs more for the LLM to interpret. There’s literally a “caveman” mod where it makes the LLM respond in caveman-speak, which means loss tokens and therefore cheaper. What’s odd (did not RTFA) is that some languages are cheaper, IIRC even has a super-caveman mode where it responds in Chinese/Japanese because that costs fewer tokens.
When you realize English is such a terrible language that even AI can’t comprehend it.
Sorry, isn’t that the opposite of the point made by this post?
I mean, kind of, yeah. If you just read the headline, then my comment does seem contradictory. ‘Some’ languages are definitely less optimal for LLMs than English.
But this is nothing new, and if you read about it before, you’d know there are many that are more optimal for LLMs, i.e., use fewer tokens for the exact same conversation content. Including languages that have just a fraction of the English language’s native speaker counts.
The article in general isn’t very good, because it provides just one concrete example - English vs Malayam. The rest is pretty much extrapolating that one case to all languages of the world.
It keeps reiterating that ‘Non-English languages’ are at a disadvantage, without any data to back this claim, other than a link to the study it’s about, which is 10x longer than the already lengthy article.
The study itself focuses on the difference between the best and worst case scenarios, which is perfectly fine, but does not exactly present whether LLMs work better with English or not.
It kind of makes sense though given that 55% of the Internet is in English, while every other language is sharing a slice of that minority position. So the majority of training data is in one language.
Anything else is likely to require a translation step into English and then another back to the original input language.
more the reason to use, make sure the more obscure the language the harder it is.
Silicon Valley built a universal intelligence that apparently charges extra for speaking most of the world’s languages. And the surcharge doesn’t even buy better answers: it buys slower responses and less context. Truly revolutionary: exporting the English only customer service experience to the entire planet.
Sure, but the Chinese and other models in the source study were no better than the US ones.
Everyone on the planet learning a different language is luxury/overhead. There are whole economic branches just for translation.
Wondering how much CO2 we could save if the world teached only one unified language.
Probably to their benefit, IMHO. Less usage & dependence.
