There’s a thread over on Reddit’s localllama debunking this. Basically, Cohere prefilled the LLM’s context with stuff that led up to a spot where the LLM would need to fill in a URL, forcing it to come up with one. They then repeated inference over and over, forcing the LLM to generate random URLs from memory, until they got one that looked suspicious. Cohere themselves said that it happened in fewer than 1 in 100000 attempts.
Given that Alibaba trained the LLM I’m not surprised that it would have seen a bunch of Alibaba URLs in its training data and subsequent reinforcement sessions. Naturally something that looks like an Alibaba URL would sometimes pop out of it when forced to generate lots of URLs.
Thank you for providing this, i appreciate it. My spidey sense said something was off.
Yeah, I would frankly be disappointed in Alibaba if they had gone to the trouble of risking their reputation by inserting nefariousness into their models and it turned out to be something this simplistic and obvious. :)
