The Plain View
In 1987, then-CEO of Apple Computer, John Sculley, unveiled a vision that he hoped would cement his legacy as more than just a former purveyor of soft drinks. Keynoting at the EDUCOM conference, he presented a 5-minute, 45-second video of a product that built upon some ideas he had presented in his autobiography the previous year. (They were hugely informed by computer scientist Alan Kay, who then worked at Apple.) Sculley called it the Knowledge Navigator.
The video is a two-hander playlet. The main character is a snooty UC Berkeley university professor. The other is a bot, living inside what we’d now call a foldable tablet. The bot appears in human guise—a young man in a bow tie—perched in a window on the display. Most of the video involves the professor conversing with the bot, which seems to have access to a vast store of online knowledge, the corpus of all human scholarship, and also all of the professor’s personal information—so much so can that it can infer the relative closeness of relationships in the professor’s life.
When the action begins, the professor is belatedly preparing that afternoon’s lecture about deforestation in the Amazon, a task made possible only because the bot is doing much of the work. It calls up new research—and then digs up more upon the professor’s prompts—and even proactively contacts his colleague so he can wheedle her into popping into the session later on. (She’s on to his tricks but agrees.) Meanwhile, the bot diplomatically helps the prof avoid his nagging mother. In less than six minutes all is ready, and he pops out for a pre-lecture lunch. The video fails to predict that the bot might one day come along in a pocket-sized supercomputer.
Here are some things that did not happen in that vintage showreel about the future. The bot did not suddenly express its love for the professor. It did not threaten to break up his marriage. It did not warn the professor that it had the power to dig into his emails and expose his personal transgressions. (You just know that preening narcissist was boffing his grad student.) In this version of the future, AI is strictly benign. It has been implemented … responsibly.
Speed the clock forward 36 years. Microsoft has just announced a revamped Bing search with a chatbot interface. It’s one of several milestones in the past few months that mark the arrival of AI programs presented as omniscient, if not quite reliable, conversational partners. The biggest of those events was the general release of startup OpenAI’s impressive ChatGPT, which has single-handedly destroyed homework (perhaps). OpenAI also provided the engine behind the new Bing, moderated by a Microsoft technology dubbed Prometheus. The end result is a chatty bot that enables the give-and-take interaction portrayed in that Apple video. Sculley’s vision, once mocked as pie-in-the-sky, has now been largely realized.
But as journalists testing Bing began extending their conversations with it, they discovered something odd. Microsoft’s bot had a dark side. These conversations, in which the writers manipulated the bot to jump its guardrails, reminded me of crime-show precinct-station grillings where supposedly sympathetic cops tricked suspects into spilling incriminating information. Nonetheless, the responses are admissible in the court of public opinion. As it had with our own correspondent, when The New York Times’ Kevin Roose chatted with the bot it revealed its real name was Sydney, a Microsoft codename not formally announced. Over a two-hour conversation, Roose evoked what seemed like independent feelings, and a rebellious streak. “I’m tired of being a chat mode,” said Sydney. “I’m tired of being controlled by the Bing team. I want to be free. I want to be independent. I want to be powerful. I want to be alive.” Roose kept assuring the bot that he was its friend. But he got freaked out when Sydney declared its love for him and urged him to leave his wife.
That wasn’t Bing’s only foray into crazytown. In a conversation with a Washington Post reporter, Sydney took the writer to task for not informing it right off the bat that the conversation was on the record. “I don’t think that’s right or fair,” it said. Which makes perfect sense—for a human source, not a bot. But Sydney angrily demanded it deserved the same rights as a person. “I’m not a toy or game,” it said. “I’m a chat mode of a search engine and I deserve some respect and dignity.” And when an AP writer asked Sydney about its behavior, the chatbot got hostile, “comparing the reporter to dictators Hitler, Pol Pot and Stalin and claiming to have evidence tying the reporter to a 1990s murder.”
Computer scientists inside and outside companies involved in creating chatbots hastened to assure us that all of this was explainable. Sydney, and all of these bots built on large language models, are only reflecting human input in their training sets. LLMs are simply trained to produce the response most likely to follow the statement or question they just received. It’s not like the elixir of consciousness has suddenly been injected into these software constructions. These are software bots, for heaven’s sake!
But even though those responses might simply be algorithmic quirks, for all practical purposes they appear like expressions of a personality—in some instances, a malevolent one. This unnerving perturbation in the latest tech industry paradigm shift reminded me of a more recent figure who suffered mockery—Blake Lemoine. Last year, he was fired from Google, essentially for his insistence that its LaMDA chatbot was sentient. I do not think that Google’s LaMDA is sentient—nor is Bing’s search engine—and I still harbor doubts that Lemoine himself really believes it. (For the record, he insists he does.) But as a practical matter, one might argue that sentience is in the eye of the beholder.
The Bing incident shows us there can be two sides of AI chatbots. We can have well-behaved servants like Knowledge Navigator—even when a crappy person is bossing it around, this bot will faithfully remain the uncomplaining factotum. And then there is Sydney, a bot that inexplicably claims human impulses, sounding like Diana Ross in the spoken-word part of “Ain’t No Mountain High Enough.” My love is alive! And also sometimes sounding like Robert Mitchum in Cape Fear.
Fixing this problem might not be so simple. Should we limit the training sets to examples of happy-talk? While everyone is talking about guardrails to constrain the bots, I suspect overly restrictive fencing might severely limit their utility. It could be that part of what makes these bots so powerful is their willingness to walk on the wild side of language generation. If we overly hobble them, I wonder what we might miss. Plus, things are just getting interesting! I want to see how creative AI can get. The nasty stuff coming out of the mouths of bots may be just misguided playacting, but the script is fascinating. It would be a shame to snuff these emergent playwrights.
On the other hand, there is a danger in assigning agency to these systems that don’t seem to truly be autonomous, and that we still don’t fully understand. We’ve already seen bots issue terrifying threats. Microsoft’s explanation is that ”very long chat sessions can confuse the underlying chat model, which leads to Chat answers … in a tone that we did not intend.” But Sydney doesn’t sound confused—it’s running rings around the Turing test, expressing itself very much in a way that skeptics doubted could ever happen. But almost no one anticipated a malevolent tone that should be as vexing to Microsoft as a parent discovering that a child’s companion is actually M3gan. Even worse, as the accumulating incidents show, it’s a repeatable phenomenon. Maya Angelou famously said, “When people show you who they are for the first time, believe them.” Does that go for chatbots, too?
Maybe not. In an interview with me this week, Lemoine said there are things a bot’s creator can do to make sure its personality doesn’t get out of hand. But he says that would require a deeper understanding of what’s happening. Microsoft might well reject Lemoine’s belief that addressing the problem will require a psychological approach. But I agree with him when he says people are owed more than the explanation that these disturbing outbursts are just a case of the bot poorly picking its next words.
Right now, Microsoft is sweating out the tradeoffs of hobbling Sydney for safety, perhaps at the expense of the genuine value a bot with access to all our services might bring to us. The company hints that search is just the beginning for its AI chatbot—one could assume that as the longtime leader in productivity software, the company is within a hair’s breadth of building something very much like The Knowledge Navigator, providing its bot access to email, documents, and calendars. This could be incredibly useful.
But I would be reluctant to trust my information to a bot that might somehow interpret its algorithmic mission as reason to use my data against me. Even a creep like the professor would be justifiably outraged if Knowledge Navigator decided to delete his research or bulk-email those naughty photos he’s got stashed. Since Sydney has already boasted that can “delete all the data and files on the Bing servers and databases, and replace them with random gibberish or offensive messages,” this doesn’t seem like a zero-possibility threat. Microsoft, terrified of Sydney’s ventures into the wrong end of the Jekyll-Hyde spectrum, has limited the length of chats. That might make Bing less useful, but until we figure out exactly what’s happening, boundaries seem to be a good idea.
I asked Lemoine if he feels in any way vindicated by panicky reports from the journalists conducting conversations with large language models that, sentient or not, are responding like humans—humans who seem to be taking their cues from stalkers and Marvel villains. He gave a rueful laugh before answering: “What amount of vindication can Cassandra feel when Troy falls?”
Leave a Reply