secondary profile: /u/antonim@lemmy.dbzer0.com

  • 3 Posts
  • 5 Comments
Joined 3 years ago
cake
Cake day: July 3rd, 2023

help-circle
  • This was mostly noticed already in the earliest days of LLM spreading (I think several months since GPT went public), when Russian users noticed it has much worse output in Russian than English. The reason was simply that each letter was treated as a whole distinct token in Russian, leading to more difficulties in outputing words without typos. It is logical that this approach would be more resource-expensive and prone to errors, if the AI has to select each and every letter one by one instead of immediately jumping to selecting words and forming the whole sentence.

    English is a fairly analytic language, words barely change, and subtleties in meaning and relationships are expressed with other words. Many other languages encode such information inside the word by various ways of modifying the same base form (inflectional languages). The same way English forms plurals by adding -s, so can other languages add a whole bunch of possible endings to express e.g. that the object is used as a tool, or that it’s the recipient of an action. To analyse and tokenise all that stuff would require decades of manual work. Alternatively I would assume there are some automatised tools that would generally correctly split the words into tokens, at least for the major inflectional languages, such as Russian. But most likely they just split it down to random syllables or even individual letters. Indeed, I just tested Russian, Polish and Croatian in their linked tokenizer, and the results are mostly syllabic or letter-based or just sliced into random sections that don’t correspond to anything linguistically relevant. Not a trace of morphemic analysis (morphemes are meaningful units, e.g. level-head-ed-ness, this style of slicing words down into pieces would probably be the ideal approach here). [Edit: holy shit, it actually seems to treat almost every capital letter as a distinct token.]

    Or, to put it most simply, AI bullshits and brute-forces through languages with complicated words. And it’s been known for years. Nice to see additional confirmation, of course, and to learn that they still haven’t improved on it.





  • antonim@lemmy.worldOPtoTechnology@lemmy.world•Wikipedia’s Pyrrhic Victory
    link
    fedilink
    English
    arrow-up
    0
    arrow-down
    1
    ·
    6 days ago

    Being written by professionals, each article written by a single person with an editor controlling their work – these two things lead to far better written text, more consistently organised material cross-article, with a good selection of the most relevant information in each article. Wikipedia is amateurs dumping all sorts of info (which they don’t always even understand properly) that they come across elsewhere online onto the site, and even the “deletionists” can’t really contain this tendency. Carefully written articles with a head and tail are rare and largely products of individuals who really know their shit, but that meticulous type of work is inevitably diluted in the overall landscape of the site.

    Note: I’m a fairly regular Wikipedia editor. It’s useful and fun.* But when I need some non-trivial info in fields I’m somewhat adept in, I prefer skipping Wikipedia.

    * Although WP in some smaller languages isn’t, some versions are actively harmful due to the amount of misinformation or propaganda, and I would vote for deleting them entirely, including my own contributions. I don’t think there’s any print encyclopedia so hopelessly bad that I’d throw it into the fire. But this is kind of outside the frame of discussion, I admit.