The Future is Constructed, Not Predicted
Shortly after the AI 2027 report was released, my friend Saffron Huang posted a tweet/mini-blog in response:
Looking to “accurately” predict AI outcomes is… highly counterproductive for good outcomes.
The researchers aim for predictive accuracy and make a big deal of their credentials in forecasting and research. (Although they obscure the actual research, wrapping this up with lots of very specific narrative.) This creates an intended illusion, especially for the majority of people who haven’t thought much about AI, that the near term scenarios are basically inevitable--they claim they are so objective, and good at forecasting!
Why implicitly frame it as inevitable if they explicitly say (buried in a footnote in the “What is this?” info box) that they hope that this scenario does not come to pass? Why not draw attention to points of leverage for human agency in this future, if they *actually* want this scenario to not come to pass?
I, too, was somewhat confused about the report, to put it lightly, and wanted to talk through it together. We try to understand the report and forecasting in general, and our conversation turns out to be less of an AI 2027 hate train than we initially thought!
By the end, we end up coalescing around these three ideas: (1) the very act of prediction has an impact on the future; (2) forecasts should be empowering, not demoralizing; (3) evaluating forecasts is messy business, so the intentions of the forecasters matter.
— Jessica
JD Do you want to start by talking a bit about your tweet?
SH When AI 2027 came out I remember reading it on the Caltrain, and just feeling like — this feels like so much work done for something that feels not very useful for most people.
I think it’s such a shame because they could be communicating the research in a way where people could actually do something about it — saying, hey, we did all this research and it looks like these are the main drivers of AI progress and here’s what we could do if we wanted to change the direction of that; here’s what we learned about geopolitics, and so on. But instead, we have this very neat narrative where the exponentials are smooth and the story is coherent. This specific forecast is very hard to evaluate because there are all of these narrative details, it’s hard to understand what exactly they throw their weight behind. But if you try to disentangle it, somehow, a lot of the dynamics that drive progress turn out to be the exact things that AI safety researchers specifically are already worried about.
Also, one of my key points was that I’m very against making people feel like the future happens to them instead of recognizing that, actually, we all have a hand in the future — especially the kinds of people who are writing and reading these kinds of things. You should be making people notice their agency and galvanizing them to action, instead of making them feel hopeless.
JD One thing that really struck me about the actual report is how much it’s written as though it’s literally a novel — these events just spool out in front of you into the future, like you’re watching a film. Something about that narrative I personally found made it really hard to keep reading.
But another consequence of the style, which you allude to in your post, is how hard it is to actually evaluate the claims being made. Maybe I spent too much time in high school doing activities that involve this category of speculation with this specific type of causal narrative (we called it “internal link chains” in policy debate) — this thing will happen and therefore this other thing will happen and therefore we will all die in the end. I find it very difficult to take that narrative seriously. In a debate we’d say, if you don’t pass this law, we’re all going to die. The other team would say, if you do pass this law, we’re all going to die.
And we weren’t just pulling stuff out of thin air — we did a ton of research, academic research, news and current events, and so on, to justify everything. All of our “internal links” were “research backed,” but of course it was patently untrue that passing or not passing a particular law would have explicitly caused extinction. But I think that experience has made it really difficult for me to then take something like AI 2027 seriously, which is maybe a personal psychological problem.
SH If you’re trying to read this critically, as a person who is not necessarily already convinced, you might come to it and ask questions like, how are they going to convince me? How did they put the narrative together? Which parts of this are more trustworthy and which parts of this are not?
It’s actually very hard to get a foothold in this because it’s just a smooth narrative. They did research, separately, but it’s really hard to find where specific claims are tied to specific pieces of research. I don’t think it’s very convincing.
JD I guess the question becomes: what was this supposed to accomplish?
SH Yeah, who is the audience for this? Because if you’re a person who already works in AI and has had lots of debates about AI, and you’re asking, “How is this going to be useful to me? Where do my decisions come in?” The report isn’t really speaking on the level of human decision making. It just says the train progresses forward like it’s an inevitable thing, and it’s really hard to engage with that. I mean, I work on the societal impacts of AI. I don’t really get anything from this in terms of what I can do.
JD It’s not just that they don’t write with individual agency in mind. It’s also not clear what the authors think should be done. If they had ultimate power, what would they change about the world? And I think that’s something that’s really hard for me to find here.
SH If the goal were to persuade people, as I mentioned, it’s hard to decide whether I should be persuaded because it’s hard to understand the cruxes here. If it’s persuading normies, or people who aren’t already convinced, it’s not really... It’s probably too sci-fi, especially near the end, to convince people.
JD It’s compelling, though, that’s the thing. Maybe if you’re not like the two of us, and you haven’t spent a million years already reading about these things, then maybe the story of what’s happening could actually be influential. But then the question of what you do with that….
SH This reads really sci-fi to me, and I think people like stories. But I don’t know that they always believe stories, and especially not sci-fi stories. I mean, look at this concluding paragraph.
The rockets start launching. People terraform and settle the solar system, and prepare to go beyond. AIs running at thousands of times subjective human speed reflect on the meaning of existence, exchanging findings with each other, and shaping the values it will bring to the stars. A new age dawns, one that is unimaginably amazing in almost every way but more familiar in some.
JD And just before it, they have the CCP capitulating to widespread pro-democracy protests and a world government that’s actually controlled by the U.S.
SH Yeah, so obviously this is the yay scenario. And then the other one is like, everyone dies. It flattens the future into some very extreme binaries. I think it is just a bit too sci-fi. On the one hand, it’s hard for the normies to stay with you and be convinced that this is plausible, because you’re really stretching it. If you’re already working in AI, it’s also not that useful. And so maybe it’s targeted at someone weirdly in between who works in tech, and is open to something like this, but they’re not concretely convinced about how exactly it might happen. It seems like a very specific slice of person.
Going back to something I was saying before — I personally have a strong preference to view the future as constructed rather than predicted, and I wish as many people as possible felt this way. Especially when we have such a high variance technology as AI, I think we could do great things if we try and put more of our brain power into doing that. So I’m personally sensitive to whether a specific narrative about the future makes space for human agency or not, and especially a high profile narrative by high profile people who have credentials who are able to spread this meme and shape how we view the future.
I actually had a friend who works in AI safety reach out to me and they were like, wow, this is really depressing. They felt discouraged by it. Is that useful? If you scroll to the very bottom, the report says “reminder that this scenario is a forecast, not recommendation,” and so on. In lots of places they say, we’re not saying we want this. But what do they want instead?
JD I think what you’re getting at is — the very act of prediction has an impact, and so when you make predictions about the world, you are also affecting what the future looks like. I really like this “constructed not predicted” framing because you can predict the world, but your prediction is actually shaping what happens in the future.
SH You’re part of the world.
JD And, as you said, a very high profile part of this world.
SH And in that sense you can’t just “observe” the world.
JD Or look at it just “scientifically.”
I’m trying to think about other places where people “forecast” and whether they affect outcomes.... People were making predictions about the Pope, but obviously nobody’s predictions had any bearing on the outcome. They individually could not have taken any action to induce any particular outcome. So that’s maybe one extreme.
On the other hand, climate predictions are often written in a way that’s structured as, “we are scientists and truth-seeking, but the science also suggests that this particular action needs to be taken.” Maybe there is a way that this could have been analogized to AI predictions, but AI 2027 is like if a climate scientist had written Ministry for the Future and then that was it. This is what the future will look like because global warming is happening.
SH Well, Daniel, one of the co-authors, responded to my tweet and we had a pretty nice back and forth about this. He did say, yeah, we don’t want to create a self fulfilling prophecy, but we don’t think we’re going to influence them that much; we’re going to present a positive vision and policy recommendations later, but it will make them more sense to people if they understand where we’re headed by default. Also, we have a general heuristic of there are no adults in the room, so we should just have a broad public conversation rather than trying to get a handful of powerful people to pull strings.
That makes sense, but I also think that people like him, probably he underrates his influence. Also maybe it would have been good to delay this until the policy recommendations were ready, release them together. Why not?
JD Maybe they will do this when they release those policy recommendations, but it would have been nice to have had something like, “this was a critical tipping point in our scenario, the corresponding action from policy or companies or some other actor that would have changed outcomes.”
SH And then on the public conversation point, you have to be thoughtful about how you conduct the public conversation and how much you make it useful versus making it—everyone’s panicking and that’s kind of it.
Something else about these forecasts — there’s maybe a pattern of underrating coordination and underrating human psychology. I’m curious what their policy recommendations will be and how much stock they would put on the power of international coordination or institutions, what their models of people’s incentives are.
There are a couple of reasons I think that they maybe don’t have a very good model of this. In the scenario, it turns out that the AI is better at AI safety research than the AI safety researchers. So they just run a bajillion copies of the AI to do AI safety research for them. But… these are AI safety researchers. Why would they do that with insufficient checks if they are the exact people who are terrified of AI being misaligned or taking over?
There are just some strange things, like around deployment in the economy, where the story is that “this just happens.” And, well, the economy is made up of human decision-makers, and they don’t just replace humans with AI en masse as soon as some research suggests that capabilities are at a certain bar.
Near the end, where they say that we should remember that this scenario is a forecast, not a recommendation, they also say,
The slowdown ending scenario represents our best guess about how we could successfully muddle through with a combination of luck, rude awakenings, pivots, intense technical alignment effort, and virtuous people winning power struggles.
There’s very little coordination here. There’s a strong technical bent of technical stuff being the thing that matters, and rude awakenings, which I think is what AI 2027 itself is supposed to be or something. And I’m just... oh, this is an interesting hint at what their theory of how stuff works in the world is. It’s... virtuous people have to win power struggles.
In contrast with climate, where climate has coordination much more at the center because you have to coordinate to have everybody reduce their emissions. And for whatever reason it’s played out as much more around these kinds of institutions and incentives and all of these things that computer scientists may not pay a lot of attention to.
JD I was just thinking about when you were talking about their narrative of, oh, adoption will just happen. I had dinner with a friend last night, and we were talking about how people are so weird — most people are really weird along some axis, and that does affect the substantive reality of what products get built, or what features get built, or what kinds of policies do people aim for. So much of this is downstream of personality quirks in a way that seems all just smoothed away.
SH It’s hard to think about that like weirdness at a civilizational scale, which is maybe why it’s glossed over here or something. The narrative is too coherent. There are coherent entities that make decisions — there’s the US and there’s China and there’s the President and there’s the CEO of OpenBrain, and there’s the public as this one unitary mass. Ok, that’s not true. I’m overstating this a little bit. I’m being a little bit unfair. But the general vibe is that people fall in these neat categories.
Meanwhile, the ending hinges on this oversight committee — this public private committee that’s set up to make decisions about AI. It’s interesting that the decisive pivotal point is whether the oversight committee votes to slow down and reassess or not, and it’s... a very “neat” understanding of power. It’s really odd that the question comes down to “is it 6-4 or is it 4-6?” — that’s the thing that either leads to human extinction or human flourishing and expansion across the galaxy.
It’s kind of a weird message that, man, it’s just these two random people who determine the fate of the universe. So in terms of constructing the future, I think that’s pretty bleak.
JD Now as I’m thinking more about this out loud… Maybe they don’t necessarily go so far saying this, but part of me wonders if they really don’t want this to be taken “seriously,” for some definition of “seriously” — this becomes more clear once you look at the two versions of the endings. But also, they have all these charts, it’s a beige microsite, so it has to be “right.”
SH I just don’t like the mixed signals. Either I should take you seriously, and I should put time and effort into evaluating your claims, or I should not. On the one hand, there’s a bunch of clear signalling around “we are really good forecasters, we did all this research behind it.” But then on the other hand, at the same time, they’re like, “we’re not wedded to this specific scenario. Anything could happen.”
JD So then it’s like — so what? What parts should I pay attention to? What am I supposed to do?
SH I think it’s also funny because, somewhere in one of the “research forecasts” on this microsite, they have a plot where they say, here’s a line, and we think the thing is going to go along this line, but our error bars are really large. But then they don’t necessarily show their error bars.
JD How could you, though?
SH The error bars would just be the entire plot, right?
JD Back in the day, error bars meant something statistical, like you had actual numbers to compute statistics with. What would that even be in this case?
SH Yeah…. But even if the error bars are metaphorical rather than actually statistical, if the metaphorical error bars are so large, why anchor us on this line? Why not draw a completely different line? I just find that kind of sus. It’s the same mixed messages thing — you anchor people to this very specific line, but your way out of it is like ah, but my error margins are wide. But you’ve already anchored people. It’s the sleight of hand that I don’t like.
JD On the one hand, there is so much fuzziness that you can make anything sound plausible. You can post-hoc rationalize anything that happened as having been consistent with your prior predictions. But at the same time, your forecast did actually affect what happened. Your forecast at time t = 0 affects what’s going to happen in the world at time t = 1 or t = 5 or t = 50. If you’re doing this unconsciously then at worst this is an ideological thing that you’re trying to steer people towards; but then, even with the best of intent, it’s not clear what to do with it or how to evaluate it.
SH Can you say more about the thing about “you can post-hoc rationalize anything”? I think one perspective of the forecasters is, wait, no, but I’m trying to provide falsifiable events, I’m sticking my neck out there.
JD I think the way that they’ve written this narrative is that specific events will happen, but it feels like what they actually want to communicate is the vibe of the trend. If, for example, a particular thing that was predicted to happen in November 2025 didn’t actually happen, the response wouldn’t be, “oh great, the forecast has been falsified.” I’m sure it would instead be something like, “oh, well, the general trend was still consistent.”
I saw this tweet that was like, with these new data points, the scaling is consistent with a super-exponential trend. It’s a plot where you can see that the “error bars” would have been so large as to have been consistent with a lot of things — it’s consistent with a super-exponential trend but it’s also consistent with a bunch of other things. There’s so little data that consistency doesn’t really mean much, so then, like you were saying, why anchor on this?
The question you actually asked me was around post-hoc rationalization. One other place where forecasts are really common outside of AI is in national security settings — they do these war games, playing out these scenarios with different actors and so on. There is also this broader rhetorical thing that happens, that’s been studied, where if you are starting from the mindset of “country X is an adversary and wants to escalate,” then any action they take is contextualized into that narrative post-hoc. “Obviously they did that because they were hoping to escalate” — that confirms your prior hypothesis, and because of that, you feel compelled to take actions that are downstream of that hypothesis, but that in turn actually drives future escalation.
I don’t think I’m saying anything especially deep here. I mean, this is kind of how the Cold War spiraled, early on; there’s also just pretty canonical psych concepts, confirmation bias and so on, at play here. I do think there is something real around like, once you anchor yourself to this broad narrative, then you’ll be tempted to make each new piece of information stay consistent with that narrative.
To be clear, though, quote-unquote “both sides” do this too. I think this is happening less now, but there was definitely a contingent of AI critics who always, constantly, were saying “oh, AI sucks and will always suck,” and every time a new model releases they’d say “oh, yeah, look at these mistakes, it’s still bad, what’s new?” But, obviously, there was something meaningfully changing as capabilities developed. So I don’t mean this as a specific indictment of, e.g., people who believe in AI x-risk.
SH It makes a lot of sense when you think about it as predicting a general vibe or a general trend. That’s vague and fuzzy enough that, if you drive enough of the conversation or the hype, you really can make it happen. Venture capital is a really good example of this: we’ll put a ton of money into X thing because there’s a ton of hype, and because of the ton of money that came from the hype, X thing then wins the market.
People have this model of the world or this thing that they want to be true — a hypothesis, but they also believe it, and their predictions are all in line with it. Then the predictions become the thing that they fit future events (and actions) to, instead of trying to fit their predictions to future events, if that makes sense.
JD I remember talking to someone and they were really attached to this idea that the authors were “superforecasters” — they were right about so many things in 2020 that turned out to have been true — and thus should be paid attention to.
Evaluating forecasts in general is a super interesting problem. There’s actually some work in CS theory that’s asking this question, how would you evaluate a forecaster? It turns out that oftentime evaluations of forecasters are super unreliable, because you can get past a mistake you made in a prior forecast by adjusting your future forecast to cancel it out, even without predicting what you think is true, if what you want to do is be right on average. And it actually matters whether people are going to be making decisions using your forecasts, too, because there’s an interplay between the forecaster, who’s providing the signal, and the decisionmaker, who has to decide how to use the signal.
SH It’s interesting because it gets at the incentives of the forecasters, which is not to necessarily just to get any individual forecast right, but to have a high batting average. And then you can see how people decide to do different things based on that, which incentivizes some different behavior.
JD Also, this is no longer really mathematical, but: if you just made a lot of predictions, some of them will be right. Most people just don’t make that many predictions, and maybe people should be making more — but if you’re just making a lot of them, you’re bound to be right on some. And you can argue that those are the ones that mattered.
SH It’s also interesting because prediction is not explanation. In classical statistics there’s often a tradeoff between modeling a phenomenon and being able to predict it. I think this is the divide between machine learning and regular statistics, where machine learning is... forget it, we don’t need to understand the phenomenon, we’re just gonna predict, right? Whereas normal statistics is…
JD Very careful about how we model things.
SH Yeah, exactly. You care about what the coefficient on the regression is.
JD The coefficient actually means something. To go to even more disciplines, there’s “theory” in social science as well, that asks, like, “how does the world work?” To evaluate whether these theories are good, you look at whether they are both explanatory and predictive. And then obviously there’s discussions about, no, it only needs to be explanatory, or have been predictive, or no, it has to be both.
But maybe AI is an example of where the thing you’re trying to model is so complex that you can’t hope to simplify it down to something tractable. AI 2027 is like the deep learning approach to modeling the world, as opposed to a social-scientific way where you isolate three variables and make an argument about how they all interact. Which I think is actually really closely related — to have some notion of correctness, like social science research hopes to have, the scope of your prediction needs to be small enough.
SH I think this is why trying to predict something like AGI is really difficult because it’s complex in a different way that social science is complex. “AGI” is underdefined, and very litigated, and just a very stretchy term. We don’t think necessarily that there is going to be a single AGI, but at the same time, people still do try to predict this on Metaculus, which is a prediction aggregator thing. They don’t call themselves a prediction market; I don’t know why.
On Metaculus, one of the most participated in markets is one on whether we will have human-machine parity by 2040. And to resolve this market, you have to decide on a definition of human-machine parity, and the definition that they choose is: can an AI answer exam questions on physics, computer science and mathematics? So, in the domains of physics, mathematics and computer science, which obviously span all of human intelligence, you pick exam questions. If the AI can answer them as well or better than three graduate students (from top-25 institutions), then we have human-machine parity.
Why is this your benchmark for human intelligence? Grad students aren’t encyclopedias; their brains are different from APIs. Again, it goes back to this bias towards the quantitative, with the quantitative forecasts really steering something like AI 2027, while underrating more squishy human things that, maybe, sci-fi is better at capturing.
JD I mean, it’s funny, first of all, like three random grad students is so much variance already. But also — I’m not saying that this is the justification people have for saying yes on that market, or that this is an action that people would actually take — you could also very easily imagine a scenario where humans are just “worse” over time. Humans already no longer perform the calculations that we know a calculator can do; a more depressing version is that students can’t read or write anymore, allegedly. If this is actually the case, then, yeah, in 2040 AI will be very superhuman. But that is also a pathway towards “AI beating human as benchmark.”
SH This gets back to the prediction versus explanation thing for me — these kinds of predictions feel a bit meaningless without any explanatory theory behind it. If you’re just predicting it, what is it telling you about the world? Is it telling you that AIs got really, really good, or is it telling you that humans got much worse, or is it telling you that AIs are specifically well suited to doing physics and mathematics and computer science?
I think this is why I’m maybe not as interested in prediction markets and forecasting, because I would rather have a good explanatory model of things than be really accurate about the specific year in which something happened, or the specific amount that compute costs.
JD But people do find forecasts really exciting and motivating.
SH Why is it fun? Yeah, I don’t want to be unfair.
JD I think it would be fun. And that’s fine, if that’s all they’re trying to do; like the pope — people are maybe betting for fun, or because they’re trying to gamble.
SH It’s plausible that prediction markets just become another form of gambling. In which case... ok, I’m not going to get into that but people could be gambling. That’s fine or not.
JD Separately.
SH Ok, so that’s why people would like to do it, themselves. But then there’s also the question of why do people like it when others make forecasts. I guess my first answer here is just that people want someone to have made the future less uncertain.
JD A psychological thing.
SH Yeah, maybe this sense of, The future is forecastable if you put in enough research. And thank goodness somebody has done that, and even if it’s not perfect, we can better understand the future. There’s something that feels kind of nice and safe about that. If we couldn’t do predictions, maybe that would be scarier.
JD Well, it is the case that research buys you something in terms of, if you know more things about the world, you are probably better at guessing what happens in the future than if you lived under a rock and didn’t know what was happening.
I think it is a psychological thing… it just feels better to know that someone has “done the research,” so to speak. But maybe the difference is, for any unit of research, there are many worlds of forecasts, but we only see the one that the forecaster shows. Maybe the answer is just that more people should be doing the research and making their own forecasts.
SH But then the prediction market version of it essentially has to be a yes-or-no thing. I don’t see any of the work that goes into a prediction market — the mental models that people are working with, for example. They’re a very information-free way of communicating about the future.
JD Now that you’re saying this about binary prediction markets, now I’m thinking that maybe AI 2027, as written, is the less bad version of that. At least they tell you a pathway. Maybe the prediction markets are literally just people having fun, like you said, with this really information-free dynamic. The longform style at least has the possibility of capturing multiple dynamics at play.
SH Yeah, we haven’t really talked about the version of forecasting that is good and valuable.
JD Yes, what would we have wanted instead? Is there a way to think about forecasting that is productive?
SH My first thought about this question is that I think that forecasting is much more useful for very narrow, well defined outcomes than it is for fuzzy diffuse stuff that is hard to measure. And especially outcomes where there’s a lot of public information or interesting things to be learned from doing the forecasting.
As an example, I’m thinking about how a lot of people bet on who the next president is going to be. That’s a pretty well defined thing; it has very clear implications for the world. Whether the president is Trump or whether it’s Harris, that is just a clear pivotal event for the world. It would be good to know the odds of that. There’s some amount of public information, there’s polling. Maybe by trying to predict it, you can learn a bunch of things about how people feel about stuff. So that feels valuable.
But then, AI forecasts are already by themselves very fuzzy. There are much fewer players, and people’s roles aren’t clearly defined. It’s an entirely new field within very new companies with internal dynamics that are not well known. We have entirely new technologies.
JD Right, like in war games or tabletop simulations for geopolitical actions, the predictions are about actions that known-ish entities can take. Maybe we have better or worse models about different countries, but at least the space of prediction is somewhat well-defined in the sense that it’s limited to what this entity could or could not do.
SH Yeah, it’s much more constrained. There’s history. There’s known action spaces. There’s a lot of well developed theory around it — we have military historians, game theory. There’s enough stable elements that it’s kind of worth doing, whereas I think for AI it’s really tricky.
JD I know we’re supposed to be in the positive section, but I did want to poke at the voting example. One of the things that, allegedly, people were saying is that one of the reasons Hillary lost is because the forecasts were so strongly in her favor that maybe it disincentivized people from actually going to vote because “she was going to win anyway.”
SH That would be a very strong example of predictions actually impacting the future.
JD Again, this is a post hoc justification — nobody is going to be the person who says, yeah, I personally stayed home because of this, nobody is going to volunteer that information — but it is something that could have happened. So maybe to narrow down your criteria more, it has something to do with whether your prediction is something that other people can take action on, including in a way that could be counterproductive. Maybe in the best-case scenario for AI forecasting, there is some prescriptive thing for individuals to do, and moreover that aligns with someone’s natural response to the prediction. For example, maybe a natural response to AI 2027 is that the median reader says, oh shit, this is really bad. I’m going to write a letter to my Congressperson. That’s an outcome induced by the existence of a forecast that is consistent with the good outcome, but for polling, I guess there’s just too much of a potential for the impact to go the other way.
SH Yeah, that’s the problem with viewing prediction markets, or viewing yourself, as just an observer, and trying to predict the world as if you’re not a part of it. People don’t really think about the second-order effects of prediction, but I can easily see how perverse incentives can come into play. When people say, we should do more things on the basis of prediction markets, I’m just like, do people think about the second order effects?
JD Your earlier point about being actors, not observers — maybe one case for forecasting is that, well, you can forecast the outcomes of your own actions. I just… personally don’t call it forecasting, but it is — the questions sound like, what would my life look like if I made this choice or this other choice?
To its credit, there are some hidden decision points in AI 2027, but instead of forecasting the different outcomes, there’s just one branch that they explore. But maybe there is a way to think through the “vibes” of points at which various actors could have made different choices.
SH Yeah, I think there’s a way you could write this story with more refocus on the leverage points.
JD Or a “choose your own adventure” thing where, maybe, you can see the multiplicity of ways that the future can play out — either, in all 10,000 worlds, everything sucks, or, there are actually 10,000 worlds that are completely divergent depending on the actions that people take.
Not to fix forecasting by just saying “ohh, but just produce a distribution,” but now that I’m saying it out loud, it feels like a distribution could hold a lot of information. [Editor’s note: no, this would not be a distribution in a formal mathematical sense, don’t @ me.] As in, here are different chunks of possible outcomes, here are the types of actions that would correspond to different chunks. I think that would be cool to see, and clearly they have an awesome web developer who would totally be able to implement that functionality.
SH Yeah, like you can explore this causal pathway. I think that would be really cool. And in fact — AI could help! Maybe you can specify lots of the causal chain and generate a bunch of scenarios; I’m sure there’s some way to present this information that would be useful, that does call attention to the causality.
They said they based a lot of this on war games, but those war games are done with participants who are taking actions and immediately seeing the outcomes of their actions; they are intimately familiar with that causal chain because they participate in the game. That’s just so different from AI 2027, where that path is obscured from people. So if there’s a way to simulate that interplay, that would be interesting. Or, when organizations do scenario planning, they do that to figure out multiple possible versions of the future, then use that information to concretely determine how to plan for the scenario.
To me, that feels a lot more proactive and useful as a way of framing something that’s similar to forecasting.
JD To be clear, they still have the epistemological problem of predicting the future, but somehow it feels more tied to something you could have actually done.
SH Yes, the consumers of the scenario are more empowered, and their role is clearer.
All my points are basically repeats of each other: I want to understand causality; show me the causation, and then it’s more useful.
JD And my repeated point is just: what do you actually want to happen? But maybe this will be resolved when they publish their policy recommendations.
SH Yeah, maybe they’ll just figure all of this out.
JD And we will automatically be conquering the stars.
SH Hopefully, they’ll tell us exactly how we can do that. Let’s see... “A new age dawns, one that is unimaginably amazing in almost every way, but more familiar in some.” I’m confused about the “more familiar in some” part.
JD That’s our utopia of 2030.
SH 2030. All right.
JD It’s kind of crazy. It’s close enough that you could ask when to have kids... like, maybe I’ll wait for utopia, you know?
Saffron Huang is a researcher in the societal impacts team at Anthropic. She also co-founded the Collective Intelligence Project (working on AI & democracy), worked at the UK AI Safety Institute, and was on the founding team of Reboot’s Kernel Magazine.
Reboot publishes essays on tech, humanity, and power every week. If you want to keep up with the community, subscribe below ⚡️
closing note
Relevant links!!!
Beige microsite macrodose from the one and only
Blog post of some CS theory research on evaluating forecasters. Full disclosure - Eric is a friend, labmate, and collaborator :)
In a prior conversation I had with Ben Recht, he makes the case that the allure of “uncertainty quantification” is 0% scientific and 100% psychological
— Jessica & Reboot Team
Share this post