Feeding on Garbage
AI agents who are relying on LLM outputs are asking for trouble
From wunderwuzzi at Embrace the Red, “The Normalization of Deviance in AI”:
The AI industry risks repeating the same cultural failures that contributed to the Space Shuttle Challenger disaster: Quietly normalizing warning signs while progress marches forward.
The original term Normalization of Deviance comes from the American sociologist Diane Vaughan, who describes it as the process in which deviance from correct or proper behavior or rule becomes culturally normalized.
I use the term Normalization of Deviance in AI to describe the gradual and systemic over-reliance on LLM outputs, especially in agentic systems.
At its core, large language models (LLMs) are unreliable (and untrusted) actors in system design.
This means that security controls (access checks, proper encoding, and sanitization, etc.) must be applied downstream of LLM output…
Trust No AI.
MM: There is a line in the report on the Challenger shuttle disaster that I think is very telling “the absence of disaster was mistaken for the presence of safety.” Many of us have seen examples of this effect at work. If you were having dinner with your neighbour and they proudly showed you the nuclear warhead they’d built on their stove, their claims that “Well it hasn’t gone off yet” should probably not reassure you.
Yet this is exactly the situation we find ourselves in with the frontier labs such as OpenAI, Anthropic, Microsoft and Google. To maximise their profit they are pushing LLMs as far and wide as they can. Increasingly this means persuading organisations to automate critical business processes, this can only increase risk both for those organisations but also for the people who depend upon them. All the labs talk about ‘safety’ but are hardly disinterested observers. While they do document risks it has the ring of shifting liability (“we told you so”) rather than a good faith effort to acknowledge or deal with the potential for disaster.
It’s hard to estimate the possibility of unprecedented events with limited historical data. In the face of this we could demand more accountability, more controls. Or, as we did with social media in the ‘00s, we could just trust that for-profit business will keep our interests at heart.
Still. Nothing terrible has happened, yet.
CB: I have a different take, since for me the major disaster that LLMs risk accelerating has already happened, and that’s the breakdown of our shared background for knowledge. Because the LLMs are fed from the internet, they inherit the broken foundations of understanding that manifests on sites like Wikipedia. The robots regurgitate presumptions as if there were no questions about how certain conclusions are supposed to have been reached. To be fair, the LLM frequently has more intellectual honesty than the more skulduggerous Wikipedia editors, so one of the amusing things you can do is poke the robot to see the flaws in its own reasoning. This remains terribly difficult for humans.
Nonetheless, many a time have I seen an LLM get into a state whereby it tells me I’m mistaken, often about things that I’m currently seeing with my own eyes! I’m pretty sure it’s inheriting this premature certainty from forum discussions, which are full of rigid bombast. But this can hardly assuage our very reasonable concerns about handing quite a lot of worrying things into the hands of the robots.
