Yandex released an open weight 80B-A3B base model about a month or so ago. I’ve heard basically fuck all about it since then, but it’s out there…

First thing that came to mind is Upton Sinclair’s The Jungle which led to, among other things, the creation of the FDA.

That wasn’t his goal though; he’s quoted as saying: “I aimed at the public’s heart, and by accident I hit it in the stomach.”

Great! Any chance you can upstream the fixes to the GitHub version? If not, I’ll try to take a look later in the week and see if I can pull those bits back out myself.

MrKaplan forked it for Lemmy World – https://github.com/Fedihosting-Foundation-Forks/mlmym

I tried taking a stab at adding some fixes to that version, but didn’t get very far. I’ve got some notes about issues I was looking at posted in the tracker there though if anyone wants to pick it up again.

We’ve got it over here on reddthat.

Another 35B-A3B MoE or similar would be nice…

FYI, Yandex released an 80B MoE base model (i.e. not instruction tuned!) recently: https://huggingface.co/yandex/AliceAI-Foundation-80B-A3B-Base – haven’t experimented with it personally though.