Last week, an interesting problem with the generative AI system ChatGPT emerged, reported here by Ars Technica:
people discovered that the name “David Mayer” breaks ChatGPT. 404 Media also discovered that the names “Jonathan Zittrain” and “Jonathan Turley” caused ChatGPT to cut conversations short. And we know another name, likely the first, that started the practice last year: Brian Hood. More on that below.
The chat-breaking behavior occurs consistently when users mention these names in any context, and it results from a hard-coded filter that puts the brakes on the AI model’s output before returning it to the user.
Whenever one of these special names was mentioned, ChatGPT always replied: “I am unable to produce a response”. The Ars Technica article explained the background to one of these keywords, and why it had such a dramatic effect on the output of the generative AI system:
We first discovered that ChatGPT choked on the name “Brian Hood” in mid-2023 while writing about his defamation lawsuit. In that lawsuit, the Australian mayor threatened to sue OpenAI after discovering ChatGPT falsely claimed he had been imprisoned for bribery when, in fact, he was a whistleblower who had exposed corporate misconduct.
The case was ultimately resolved in April 2023 when OpenAI agreed to filter out the false statements within Hood’s 28-day ultimatum. That is possibly when the first ChatGPT hard-coded name filter appeared.
Some of the other names that cause the same “choking” behaviour in ChatGPT were similarly involved in lawsuits, while others were simply errors (including “David Mayer”, as it later emerged). What makes this new aspect of generative AI interesting in a copyright context is the following:
Scale AI prompt engineer Riley Goodside discovered how an attacker might interrupt a ChatGPT session using a visual prompt injection of the name “David Mayer” rendered in a light, barely legible font embedded in an image. When ChatGPT sees the image (in this case, a math equation), it stops, but the user might not understand why.
It’s easy to imagine copyright companies adding one or more of these special choking names that are not simply mistakes to Web pages in a suitably invisible form – for example, in a tiny typeface in a very light colour. They might even be placed directly in the text, no matter how irrelevant. There’s a precedent for the latter approach, in the form of fictitious entries, also known as “Mountweasels”, as explained by Wikipedia:
By including a trivial piece of false information in a larger work, it is easier to demonstrate subsequent plagiarism if the fictitious entry is copied along with other material. An admission of this motive appears in the preface to Chambers‘ 1964 mathematical tables: “those [errors] that are known to exist form an uncomfortable trap for any would-be plagiarist”. Similarly, trap streets may be included in a map, or invented phone numbers in a telephone directory.
Modern Mountweasels would not be used to demonstrate plagiarism, but to prevent generative AI systems from performing any operation on that Web page. For copyright companies that are currently seeking to have their material removed from AI models – probably an impossible task given the way that the results from billions of piece of input data are aggregated – this could be an alternative approach that makes it harder for people to use online material with AI services. It wouldn’t be impossible to circumvent, but would be an obstacle that would put off many people.
That might make the copyright companies happy, but Internet users would suffer because of the collateral harm it would cause. It would mean that in addition to the well-attested cases of generative AI systems simply making up answers, there would be swathes of online material that such systems can’t handle, with a resultant impoverishment of access to knowledge.
Although there’s no evidence that this approach is being deployed currently, older Mountweasels have certainly been used. Moreover, it is early days, and it would hardly be a surprise to see modern Mountweasels in the future, especially if the copyright industry loses some of the court cases it has brought against generative AI systems.
Featured image by Stable Diffusion.
