LLMs, Innovation, and Capitalism
LLMs, Innovation, and Capitalism
What happens to innovation when science gets privatized?
For a long time, much of AI research was default-open. OpenAI’s GPT is an innovation on the Transformer model, which Google researchers open-sourced in 2017. Two years later, OpenAI set a new precedent by initially choosing not to release GPT-2 to the public—describing the model as simply “too dangerous.”
Today, both AI research and corporate gatekeeping has accelerated massively. NLP researcher Belinda Li attributes these locked doors not to “safety,” but to capitalism. In today’s guest essay, she makes the democratic case for open science, explaining how the closed model shapes—and constrains—what AI gets built.
P.S. Reboot is bringing on 2-4 managing editors to support our new editorial vision. Learn more and apply here by June 8.
LLMs, Innovation, and Capitalism
By Belinda Li
Originally published on her website
Note: The bulk of this article was written in February 2023. Since then, several open-source replicas of GPT3 have been released, most notably by FAIR and Stanford. While these have improved accessibility to these models, issues mentioned here are still at play: first, these are replicas that are several versions behind the most recent, state-of-the-art models, and second, resource constraints continue to limit who gets to use these models in practice. In the long run, open-source efforts cannot outcompete closed ones, and will always be playing catch-up—the playing field is inherently unequal. Instead, the question is fundamentally about rebalancing political power—and as a community, we should think about pushing back against this trend towards closed science in a more organized way.
Being a graduate student in NLP in 2023 is both an exciting and yet frustrating experience. On one hand, we are faced with a massive amount of hype, attention, and financial investment entering the field—new papers are being written every day at an unprecedented rate. On the other hand, a small number of companies are seizing this opportunity to quickly monopolize this field as their personal pot of gold. They do so by, essentially, closing off access to fundamental research problems. In this new paradigm of research, companies release demos and press releases to draw in credit, clout, and additional funding. They take from the community, building off the backs of decades of open-source research, without offering any of their own information, ideas, or resources in return. Building off of this inequity, these corporations work to concentrate an increasing amount of wealth, resources, and investments in their own hands.
This has immediate consequences for the research community: increasingly, there are insurmountable barriers to entry to certain types of research. Many PhD students have started to ask: what is even possible for us to do? As companies work to purposefully set up these barriers, this has left the rest of the research community having to form their research agenda entirely depending on who and what these corporations elect to grant to them. The rate of innovation is artificially suppressed.
Beyond this, however, there are vast and reaching ethical consequences: Which communities are benefited and harmed by these models, and who gets to decide how important each of these communities are? Whose labor gets used and credited when building these models? Whose voice is privileged in the building of these models, and who gets to dictate the priorities of the field?
The stagnation of innovation
There are two main barriers to accessing large language model (LLM) research: the first being the lack of resources and economic means to train these models, and the second being the lack of access to these models in the first place.
The former is not new in AI research, and certainly not in scientific research as a whole. Though arguably any means of distributing funding privileges certain ideas over others, there are ways to increase accessibility even at this stage, e.g. through sharing and pooling of resources.
We’ll mainly focus on the second barrier, which has emerged in recent years as corporations move increasingly to safeguard and profit off of their scientific ideas. OpenAI, and recently Google AI, have all significantly restricted, or even entirely closed off, access to their LLMs. For example, when GPT3 was released, OpenAI (citing safety concerns) closed off access to almost everything except a (paid) API where users can have limited access to only the top-K predictions. This was a purposeful rejection of the (at the time) predominant practice of open-sourcing code, model weights, data, etc., and has proven to be a lucrative business decision, helping consolidate OpenAI as the leading brand in LLM research.
On the other hand, this has essentially closed off the exploration of fundamental language model research to a small circle of researchers within the direct vicinity of OpenAI. By privileging access to model weights, representations, controls, and data, the type of innovation (on LLMs) that can be produced outside of OpenAI is limited. This is reflected in the type of research that has emerged and been popularized recently, lots of which can either be described as various forms of (or supplements to) prompting, or various ways of using LLM generations for different tasks. In fact, these are the only types of research that can be conducted with access to the OpenAI API. Research that requires access to model logits and weights, such as interpretability studies, model editing, and controlled generation studies (which allow us to understand and control the decisions made by these models), cannot be done except on much smaller-scale language models, which can display very different behavior compared to large language models. Thus, a few powerful entities get to close off entire research avenues for the larger community by deciding what forms of research can be conducted and by whom.
Despite great training, knowledge, and creativity, the rest of the community simply cannot contribute to certain fundamental problems due to these barriers. In fact, a great deal of attention and work gets diverted into simply recreating versions of these systems: GPT3 vs. OPT, ChatGPT vs. BARD. This is duplicate work which could be saved entirely if such systems were accessible and open to all.
To drive this point home, imagine if Google had decided to operate by this same policy and had never released publicly-accessible versions of the Transformer and BERT, instead requiring everyone pay for API access to these models. What innovation would we have missed out on in the past few years? Would researchers at OpenAI even have been able to conceptualize, much less build, GPT2, GPT3, and ChatGPT? Aside from the inherent unfairness of profiting off of years of open science that has brought the field to this state, closing off access to information may also curtail the overall rate of scientific advancement in the future.
Although I’ve spent most of this section pointing the finger at OpenAI, they are just one player in an environment of broader systemic incentives. The problem is, of course, that the principles of open science eventually conflict with the competitive requirements of capitalism. Because of these inherent contradictions, we are at an inflection point where research ideas are increasingly siloed, privatized, and safeguarded by individual corporations. These ideas are made into products to be consumed and marketed to the public, but their recipes are kept secret.
Why does broader access matter?
One counterargument that could be posed: maybe we don’t need any more new architectures, algorithms, and ideas. We’ve already seen that by simply scaling up existing models, we’re able to accomplish new and extraordinary feats. And indeed, this has been the predominant mode of progress in the field in the past few years.
However, as many others have asserted, there are still fundamental problems to solve within AI: the lack of systematicity, interpretability, and factuality, the inability to understand meaning and intent, the inability to deal with real-world environments and low-resource languages, etc. Arguably, some of these may be able to be addressed with scaling to a certain degree. However, it is unclear whether scaling is an absolute solution, and certain hard problems (e.g. of understanding/possessing communicative intent) seem to require fundamentally orthogonal innovations.
I’m also not going to sit here and say that scaling hasn’t been incredibly useful. This is not intended to be an indictment of scaling in and of itself, but rather the incentives driving modern scaling research. Why have we been able to scale so much in the first place? A lot of this can be attributed to the concentration of wealth and resources in the hands of tech companies. In many ways, scaling is just a straightforward application of capitalist growth to NLP, built upon the premise that simple, unfettered, and relatively thoughtless accumulation of resources, data, and labor will somehow solve AI. And as corporations accumulate more wealth, techniques which can take the greatest advantage of this (e.g. large-scale training of connectionist networks) are naturally rewarded.
Rather than suggest we stop research in scaling, I believe we should try and curb the worst elements of capitalism that have weaved its way into the field. We already know that private corporations aren’t incentivized to do so—most evidently, in the ousting of Google AI ethics researchers in 2020. Compare this to OpenAI’s safety policy, or corporate legal teams which flag and review each paper before they are released. It just so happens that the latter group is constructed to protect the interest of these corporations. Risks and ethics are important to consider only to the degree that they bring in profits and evade lawsuits; when it’s no longer profitable to consider them, these things are quickly pushed to the wayside.
It’s worth examining the ethical consequences themselves of the current developments in LLMs. I’ll give only a cursory treatment, as many before me have studied this in great depth. In practice, these LLMs and scaling approaches have worked to serve predominantly the privileged English-speaking population, and other populations well-represented on the internet. Scaling at all costs heavily exploits labor and resources—building ChatGPT entailed using underpaid Kenyan workers to annotate large amounts of toxic data. LLM training also comes at a great environmental cost through energy consumption, and though people have pushed for corporations to recognize the climate consequences of their training methods, scaling research continues to proliferate, while very few corporations conduct serious research (at the same rate and scale) on new, climate-efficient innovations.
Finally, it’s worth thinking about the ethical consequences of the monopolization of these technologies by a few corporations: Whose voices are privileged by the system? Who gets a say in how LLMs are used, built, and deployed? Which communities and issues matter to the model builders and which communities and issues don’t? The increased concentration of resources means corporations themselves currently have unilateral decision-making power about who gets to use these systems and in what capacity. This is not to say that filtration is not necessary: clearly, there are malicious actors who wish to use these systems to enact harm. But are these corporations truly filtering bad actors? Or are they promoting even more harm by closing off access to communities who might benefit from these technologies (and whose labor went into building them)? Policing who has access should not be under the jurisdiction of a single corporation, and furthermore, a blanket closed access policy is far more punitive and distrustful than necessary—the end result is simply, once again, privileging an elite group of paid and institutionally-accredited researchers, which in turn means that these systems get further tailored to their needs.
Possible answers going forward
I think that scaling research can absolutely be conducted in an ethical, thoughtful, and deliberate way. The issue is when it’s privatized and driven by profit. In addition to open science issues suppressing innovation, the profit motive means that corporations are now blindly clamoring on top of each other to train the biggest baddest model without regard for who is served or harmed, and without regard for whether or not this is even the optimal solution to begin with. And because of the lack of open access and resources for those outside of a select few big tech companies, others researchers must accept this at face value. Regardless of how smart and innovative you may be, there is no choice but to be at the whims of whomever has the most resources to train the biggest model.
I believe NLP research works best when driven by a community working collectively for their own needs. In the recent Turkish-Syrian earthquake, a large group of engineers and researchers came together to build technologies to extract locations of survivors and what resources they require. Māori people have led the effort to build technologies for their language, te reo, a low-resource language which has received less attention from mainstream LLM research. Of course, arguments can be made that these efforts came together at a smaller scale and dedicated to very specific issues, and that fundamental NLP advances cannot be made in the same way. However, I believe this is simply because we don’t have the correct societal incentives; people want to work on fundamental problems, but there are insurmountable barriers preventing researchers and communities from doing so.
There are some things we can also do in the immediate short term to rebalance the incentive structure. For example, the trend of closing off LLM access can be discouraged by the community (e.g. through a reproducibility requirement). Moreover, journalists and researchers can avoid hyping up, advertising, and using models that are entirely, essentially, closed to the community (consequently generating more private data for these corporations to use to improve these models). As a community, I believe we should think more critically about what sorts of trends we tout, and whether or not they’re healthy for the research ecosystem as a whole.
In the longer term, we can imagine a future where scaling NLP can operate as a publicly-funded, open-source, international effort, where all contributors of data and labor benefit from the system and are credited. We can and should aim to make LLM development a more democratic effort, with representation from all sectors of society. Ethical guidelines and tradeoffs should also be democratically decided by a representative group of the population, not unilaterally by a single corporation. Imagine the power of a global community of NLP researchers working openly and collectively to advance better technologies for all humans.
Finally, I believe achieving this future is inherently a political question—a question of how we can reshift decision-making power about who gets access to technologies to the democratic majority. There is clearly a lot of discontent (and existential dread) from NLP researchers about the recent trend towards closed science. But other than lamenting on Twitter and in private discussions, very few concrete actions have been taken. One solution, to simply build open-source replicas, is potentially useful in the short term but does not address the root of the issue—corporations always have an upper hand when they get to utilize open-source collective efforts, but get to privatize and profit off of their own ideas. One may also be tempted to simply put forth a sound rhetorical argument for open-sourcing, though it seems difficult to simply persuade a company against pursuing profit in a capitalist society.
I am personally a major proponent of organizing and collective action as a means of building power. The organizing must be directed not just against OpenAI, or even the single issue of closed science, but against the root cause of these problems—the capitalist incentives that currently dictate the priorities of the field, and of society in general.
Belinda Li is a PhD student at MIT researching AI and natural language processing, and a labor organizer for the MIT grad student union. Find her on her website and Twitter.
Acknowledgments: This blog post was inspired by countless conversations with my labmates in the LINGO lab, and fellow MIT GSU organizers. The following people in particular were of instrumental help in the writing of this blog post: Daniel Shen, Alexis Ross, Tianxing He, and Jacob Andreas.
Reboot publishes essays and interviews reimagining tech’s future every week. If you liked this and want more, subscribe below.
🌀 microdoses
Reboot reader Gabriel Nicholas coauthored a new paper on the limits of using multilingual language models for content moderation, emphasizing the importance of involving researchers from different language communities.
In The Atlantic, community members
and argue that TikTok’s risks are based not in “Chinese values,” but in the same exploitative business models pioneered by American tech giants.On a chiller note,
wrote a very lovely piece on “the soft power of Google Doc publishing.”
💝 closing note
When Jessica and I started Reboot, we often quipped about trying to become “the new WIRED.” In the same way that WIRED shaped a generation of technologists’ ambitions during the late 90s and early 00s, we believe that Reboot—in its most ambitious form—can redefine what “success” and “progress” in tech mean altogether.
Yet as I describe in “Does Tech Criticism Still Matter?”, much of tech writing today consists of surface-level takes that do little to change the behavior of tech consumers or creators. Critique abounds, but principled, practical visions are few and far between. Reboot has not been immune from these flaws—but we see a role for our publication to tell positive yet clear-eyed stories about technology and society.
In 2023, we’re doubling down on our editorial work in service of our mission to reimagine techno-optimism for a better collective future. We plan to publish Kernel Magazine twice as often, under specific themes, and to increase the quality, rigor, and relevance of our newsletter and event series.
To support this next chapter, we’re reconfiguring our core contributor team as an editorial board, and seeking 2-4 people to join as managing editors.
I’m especially excited about having individual editors’ voices drive more of our work. While I’m proud of being a “big tent” organization, the unified Reboot umbrella—and concentrating editorial responsibility in me and Jessica—has hampered our range and creativity. We want to build a diverse, pluralistic team that takes strong intellectual stances, even at the cost of misunderstanding, conflict, or failure. We want to celebrate conviction: having the courage to give a shit.
Within the editorial board, we hope to create an environment like a combination newsroom, groupchat, and debate club; filled with reading recs, memes, good-faith intellectual sparring, and deep care for the Reboot mission and community.
Learn more about the role and responsibilities.
Apply here by June 8.
Join us?
Jasmine & Reboot team
 | A guest post by
|