cognenza@lemmy.world

1 Post
0 Comments

Joined 8 days ago

Cake day: October 20th, 2026

You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.

OverviewCommentsPosts

cognenza@lemmy.world to

LocalLLaMA@sh.itjust.worksEnglish · 1 day ago

In OpenAI's o200k_base tokenizer, " strawberry" is a single token (101830): a small demo of why letter-counting is awkward for LLMs

0

8

In OpenAI's o200k_base tokenizer, " strawberry" is a single token (101830): a small demo of why letter-counting is awkward for LLMs

cognenza@lemmy.world to

LocalLLaMA@sh.itjust.worksEnglish · 1 day ago

0