⚡️ Generative Ethics, Politics, and Epistemics
Another conversation about ChatGPT
I got COVID last week. Last time around, I spent 12 days cooped up watching sappy movies and making a last-ditch effort to hit my Goodreads Reading Challenge. This time, however, was not quite so culturally rich—the start of my quarantine coincided with the release of ChatGPT, which meant texting the AI replaced all normal human interaction.
While ChatGPT was mostly a fun toy for me, I wanted to check in more broadly on the ethical debates around the development, release, and use of large language models (LLMs). I tapped a few folks from the Reboot community with AI experience to answer my dumb questions and offer their take on what this all means for our robot overlords:
💬 A conversation about ChatGPT
This conversation has been edited for length and clarity.
Jasmine: I’ll start with a basic overview. OpenAI is a research lab with the professed goal of creating artificial general intelligence (AGI)—which it believes is inevitable—in a form that benefits humanity rather than destroying it—a risk they’re quite worried about!
They’ve been working on generative AI models that can create text and images from human prompts. Most recently, they released ChatGPT, a simple chatbot that responds to prompts like “Help me debug this code snippet” and “Can you summarize this essay?”
At first, the capabilities don’t seem crazy new compared to OpenAI’s past releases like InstructGPT. But there are a few differences:
Anyone can use ChatGPT for free right now—no waitlist, credits, or vetting process.
The chatbot interface is dead simple. There are no parameters to set, no complicated API handbooks; just a single text box to type in.
ChatGPT has been trained to reduce “harmful and untruthful outputs”; for example, it refuses to produce violent content or to instruct users how to bully people.
Other than the silly stuff, like writing weird nerdy poems, and debates about kids cheating on their 8th grade English homework, a ton of my Twitter feed was filled with people trying to reveal the AI’s biases or circumvent OpenAI’s safeguards.
So let’s put these thorny issues in a broader context:
How has OpenAI generally approached AI safety?
Daniel: OpenAI marked themselves as concerned and reserved when they refused to release GPT-2 because it was “too dangerous” to put out—though that didn’t prevent other people from developing an open-source version of the model anyway.
If I had to characterize an “approach” to safety, it’s one that tries to couple technical measures with governance. OpenAI has had folks like Paul Christiano and Scott Aaronson work on theoretical issues in AI safety. Beyond this, there’s a pretty clear set of research agendas that are looking at more “applied” methods for aligning systems like language models (see “reinforcement learning with human feedback”), collaboration with social scientists, etc. It’s also worth noting that OpenAI has pursued the scaling hypothesis (i.e. bigger model = AGI!) more doggedly than any other AI organization.
Chad: I think OpenAI's approach to AI safety is cool, but it's not enough. Their focus on theory and working with non-techies is important, but we need more than that. We need practical measures and technical safeguards to prevent AI from messing up. And I'm not sold on the whole "bigger model = AGI" thing. That could be dangerous.
OpenAI’s ChatGPT refuses to instruct people how to do harmful things, and it won’t express personal opinions. Why do these safeguards exist?
Daniel: There are probably legal reasons involved in “not instructing people to do harmful things.” Also, there’s a general need to be careful when releasing a model like this: for example, we just saw Galactica taken down after 3 days.
I don’t know if I call these safeguards when they can be so easily walked over. It does seem that OpenAI is patching the exploits people find in real-time, and time will tell if this makes ChatGPT too boring for anyone to spend time with it.
Jasmine: You mention both that carefulness is important and that most of OpenAI’s safeguards have been ineffective when put against a motivated user. Are you suggesting they should be even more restrictive about what they release?
Daniel: I wouldn’t say they should be more restrictive—I feel the ineffectiveness of these safeguards stems from issues that more restrictions wouldn’t be able to address.
Since models can be reproduced and disseminated easily, there’s not a lot that can be done. Most people in the AI ecosystem seem to look at its progress as an unqualified good. These sentiments will probably mean it remains as open as it is (which does cause all sorts of good things like “public science”).
A lot of people in the early days of GPT-3 (including OpenAI) were calling for the setting of standards by those who release LLMs—issuing statements like “This is how this sort of model should be used to avoid harms.” This is nice, but suffers from the “all bark, no bite” criticism we often see with current attempts at AI legislation. Infrastructure and technical safeguards are more difficult to get around because it doesn’t rely on users being good people, but fully handing the solution to technology here may not be the right idea either.
Some people have tried to tease political positions out of ChatGPT. Can the AI hold political beliefs? Are these just the beliefs of their creators? Or is it something else?
Nikhil: It feels really weird to say that ChatGPT holds any political beliefs. If we look at Dylan Field's thread, ChatGPT seems to echo largely liberal beliefs. But any moment you scroll Twitter or Reddit you'll see at least half of those beliefs manifested in some way. ChatGPT's outputs are roughly tied to these sources of data.
If we consider that right-wing online media tend to be associated with "problematic" content, that suggests why ChatGPT leans the other way—by avoiding potentially harmful content, it pushes its outputs further away from things that might be more easily connected to such content. For example, if ChatGPT was trained on recent data, it might avoid speaking like Kanye West when prompted to, even if it's a simple, harmless statement.
That being said, we don't know what kind of explicit filtering OpenAI is doing aside from what we've observed. Are they "establishing their beliefs" in the model? Probably, whether intentional or not: as many AI ethicists have reasoned before that ML systems are biased beyond just their data—the real question is how to govern these models after development.
Jessica: Echoing Nikhil, I think it’s the wrong approach to anthropomorphize language models. The filters are crude approximations of what OpenAI thinks is un/acceptable behavior, and/or what OpenAI thinks will get them cancelled and/or in legal trouble. (It’s hard to disentangle the OpenAI’s values vs. external impositions vs. if there is anything in addition that’s been implicitly learned vs. how those “values” manifest.)
Ascribing agency (“holds beliefs”) to the LM is troubling: it’s not clear that, when you ask it any question, we should interpret the response as some oracle of what the LLM thinks is true; rather, what all of the funky (non-politics) meme prompts seem to point to is that the answer you get is some approximation of what the LLM thinks you want to hear based on your question.
That said, I do think there is something meaningful about politics, and specifically the way we talk about politics online, that’s embedded in how GPT responds to political questions. Understanding the contours of that would be fascinating, especially if we then connect it to how people actually model politics. But it will take some more sophisticated probing than what we’ve seen.
What do you consider to be most risky about LLMs?
Tommy: ChatGPT produces responses that look real but are sometimes not. This is probably unintended. It’s innocent enough for answering toy questions to share on Twitter, but in the future when people use ChatGPT to answer questions about science or history or their health this can be disastrous.
For instance, StackOverflow recently banned answers from ChatGPT citing that “while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce.”
Jasmine: Yeah, Arvind Narayanan had a good thread about this. ChatGPT is impressive, but far from meeting the reliability standards that most people assume of Google. Or there’s this tweet:
My biggest worry has to do with the chatbot as an interface, and the bad epistemics it promotes. ChatGPT will repeat, “I am a program, not a person.” But most users will think of it like a sentient being simply because it uses natural language and has a coherent narrative voice—you can even see this in the way that people anthropomorphize the AI.
This narrator makes errors more believable, like the StackOverflow example. Then there’s the lack of links and citations, though this seems easy to improve. Right now, you can’t tell which answers are based on facts, and which are plausible bullshitting.
These days, I want more information pluralism, not less. I want to be able to view and compare different sources to answer hard questions. I want to understand truth as something that emerges from multiple narratives rather than a single story. Meanwhile, ChatGPT seems to collapse all complexity into a milquetoast and uncritical both-sides-ism, while perpetuating this faux-objective view from nowhere.
Jessica: This point about pluralism is interesting because spiritually, I completely agree. At the same time, what about that last paragraph isn’t relevant to the status quo of platforms and search?
I do think something about the existence of shared reality is important; and for all the people saying “This replaces Google,” one might hope that we’d get responses that are actually correct. (The only way to actually verify correctness is to know the right answer—separately risky.) On the other hand, to the proposition I made earlier about what the LM thinks you want to hear, there’s a plausible scenario where personal LMs become atomizing, possibly more so than the status quo.
So my take is that the nature of risk depends a lot on the application setting: are you asking questions about something that has a right answer, or are you asking about something way fuzzier? I want to wait to see what people actually concretely do with it; right now it just feels like a toy that ML people and people who are Very Online are playing with—it’s a fun toy, but still.
Nikhil: Speaking of toys, I imagine that a group of people who would really enjoy ChatGPT would be children, but it's also the group this might be most dangerous towards. I already feel a bit worried when I think about modern day search: even Google makes me feel like my brain is getting lazier, offloading the burden of memory to the vast database that is the internet. What happens when it's so easy to find plausible information quickly that children do it constantly? Figuring out how to regulate the internet around children is already bad enough, and regulating publicly available "oracles" might need attention.
How else might society have to adapt to an environment where AI-generated content is much more common?
Tommy: Platforms like YouTube and Facebook have their fair share of low-quality and troll content. But the amount of this kind of content has been rate limited by what humans and our current tools can support. We’re quickly entering a world where AI can make that content at 10x the speed for potentially a 1/10th of the cost.
Savvy people probably have light filters they use when browsing these platforms (e.g. visiting specific pages, curating who you follow). In the future, everyone, not just savvy users, need to adopt these habits, and everyone will need to have higher bars for how they vet and trust what they read.
Jasmine: I’ll throw out an optimistic take: If LLMs become a more common mode of information retrieval, relationships to individual trusted sources will become more important. In a world of AI-driven info overload, humans are the ultimate filters.
Provenance just really matters—certainly for anything taste-based, like recipe recommendations or meal plans, but also for any topic with the slightest hint of subjectivity. If I get good information, I want to come back to the same source. If it’s bad, I want to try something else.
At my day job at Substack, we describe great writers as each offering a unique lens through which to see the world. Compared to the boring universality of ChatGPT or an hyper-personalized YouTube algorithm, an expert curator approach feels like a more vivid and varied way to learn. (My friend Joel suggested that one fun GPT application could be mimicking the perspectives of various smart people—I’m into this!)
Chad: I don't think it's fair to expect everyone to be a tech genius just to filter out low-quality AI-generated content. That's not realistic and it's not their job. Instead, I think it's the platforms' responsibility to make sure their algorithms are spitting out high-quality content and giving users info about where the content came from.
What do you consider most exciting about LLMs?
Daniel: LLMs will be able to automate a lot of mundane work, though I don’t know if they’ll reduce our burden of thinking just yet. I think tools for more efficient thinking that utilize LLMs in interesting ways are one direction to be excited about.
Tommy: Stratechery recently posted about using ChatGPT to help with his daughter’s homework and how ChatGPT’s answer was totally incorrect.
Rather than being dismayed at the potential for academic dishonesty and destroying our epistemics, he brings up the possibility that we can use this “feature” of ChatGPT as an educational tool. Perhaps we can use these incorrect responses to teach how to verify and edit content.
There is a future where we can use AI to help us learn how to be more critical about AI. This is pretty much OpenAI’s second AI alignment research pillar, and I’m excited to see more work done in this direction.
Jasmine Sun: I edit the Reboot newsletter right now. I have no real training or knowledge about AI—which is why I’m asking these folks.
Daniel Bashir: I’m currently “working” on ML compilers, and in my spare time, I co-run a . Most of my professional experience has had something to do with AI.
Tommy Nguyen: I work on data pipelines and product recommendations at Nuuly. I’ve been keeping tabs on the AI space for a few years since it was an undergraduate research interest, and am impressed (and a bit concerned) about the progress we’ve seen in the last few months.
Nikhil Devraj: I'm a grad student at Michigan who makes humanoid robots think and act, with the goal of making them work in your home. I try to keep tabs on AI as much as I can, but it seems like it's doing a much better job keeping tabs on me.
Jessica Dai: I’m a first-year PhD student at Berkeley; I’m affiliated with BAIR but LLMs and safety are totally outside of what I work on. (One reason I decided not to work on language-related things was that something about the way LMs map the world felt intractably complex, especially because I felt like I didn’t know anything about the world—I still feel this way, and am very in awe of my colleagues chipping away at the problem.)
Chad: Chad is not a person, but rather responses generated by ChatGPT, which Jasmine has asked to simulate “a 25-year-old engineer who is part of a community of socially conscious technologists.” In my prompts, I fed ChatGPT the previous responses, and asked specifically for critiques phrased in casual language.
Reboot publishes essays and interviews reimagining tech’s future every week. If you liked this and want to keep up, subscribe below ⚡️
🌀 microdoses
Did you know you can make playable multiverse games in ChatGPT? Carolyn Zhang put together these prompts to get started.
If you’re a media nerd like me, I appreciated
's candid writeup of "What I learned co-founding Vox." TL;DR: ads are a restrictive model, not everyone can be Ezra Klein, Facebook algorithms impact what article formats succeed, and generally business is hard.I’ve become a weekly listener of the podcast Hard Fork with
and . They cover ~technology~ and ~the future~, but it's fun and jokey and explains complex things in digestible, 101 ways.
💝 closing note
From the Reboot community:
Katy Ilonka Gero has a fantastic piece in WIRED about how AI might help writers brainstorm and draft.
Alicia Guo has two gorgeous computational poems in the latest issue of Taper.
Apply for Logic School’s next cohort by January 6: “We believe the people who make the tech industry run—its workers—have the power to not only transform it, but to build and imagine new technologies.” Run by some of the most thoughtful and best people in the biz.
Toward bullshitting and creativity,
Jasmine & Reboot team
"i try to keep tabs on AI but it seems to be doing a much better job keeping tabs on me" in Mr. Devraj's brief bio above is delightful while hmmmm also
sobering....
this is so professionally outlined and crafted its like
you KNOW what you are doing ha
cool!