The way LLMs work is that they actively will make multiple attempts to get past hurdles (because they have no intelligence or methodology) so guardrails need to be extremely tight for them to work, other wise the model will simply see it as one of the challenges to overcome.
That’s the mantra, and that is very poor technology to put in the hands of people who don’t understand how it works.
I hate that I can’t tell if this is a reference to something that actually happened or not.
It’s probably something like “I’ve disabled agent’s
removeFiletool, but LLM figured out that it can use thebashtool, still”.It looks like “AI bad” or “Claude insecure” mantra.
The way LLMs work is that they actively will make multiple attempts to get past hurdles (because they have no intelligence or methodology) so guardrails need to be extremely tight for them to work, other wise the model will simply see it as one of the challenges to overcome.
That’s the mantra, and that is very poor technology to put in the hands of people who don’t understand how it works.