metaforecast/data/xrisk-questions.csv

53 KiB
Raw Blame History

1titleurlplatformdescriptionoptionsnumforecastsnumforecastersstars
2Total existential risk by 2120https://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: ~17% (~1 in 6) Ord writes: "Dont take these numbers to be completely objective. [...] And dont take the estimates to be precise. Their purpose is to show the right order of magnitude, rather than a more precise probability." This estimate already incorporates Ord's expectation that people will start taking these risks more seriously in future. For his "business as usual" estimate, see the conditional estimates sheet.[{"name":"Yes","probability":0.17,"type":"PROBABILITY"},{"name":"No","probability":0.83,"type":"PROBABILITY"}]2
3Overall risk of extinction prior to 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.19,"type":"PROBABILITY"},{"name":"No","probability":0.81,"type":"PROBABILITY"}]2
4Existential risk in the 21st centuryhttps://80000hours.org/podcast/episodes/will-macaskill-paralysis-and-hinge-of-history/#transcriptX-risk estimates[{"name":"Yes","probability":0.01,"type":"PROBABILITY"},{"name":"No","probability":0.99,"type":"PROBABILITY"}]2
5Extinction risk in the next centuryhttps://80000hours.org/articles/extinction-risk/X-risk estimatesActual estimate: Probably at or above 3% [{"name":"Yes","probability":0.03,"type":"PROBABILITY"},{"name":"No","probability":0.97,"type":"PROBABILITY"}]2
6Risk of extinction over the next five centurieshttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: At or above 30% The probability of the human race avoiding extinction for the next five centuries is encouragingly high, perhaps as high as 70 percent”[{"name":"Yes","probability":0.3,"type":"PROBABILITY"},{"name":"No","probability":0.7,"type":"PROBABILITY"}]2
7Our present civilization on earth will survive to the end of the present centuryhttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: ≤50% ("no better than fifty-fifty") [{"name":"Yes","probability":0.5,"type":"PROBABILITY"},{"name":"No","probability":0.5,"type":"PROBABILITY"}]2
8There be zero living humans on planet earth on January 1, 2100https://www.metaculus.com/questions/578/human-extinction-by-2100/X-risk estimatesActual estimate: Median: 1%. Mean: 7%. While the general feeling of most people, especially now that the cold war is (mostly) over, is that the risk of human extinction is extremely small, experts have assigned a significantly higher probability to the event. In 2008 an informal poll at the Global Catastrophic Risk Conference at the University of Oxford yielded a median probability of human extinction by 2100 of 19%. Yet, one might want to be cautious when using this result as a good estimate of the true probability of human extinction, as there may be a powerful selection effect at play. Only those who assign a high probability to human extinction are likely to attend the Global Catastrophic Risk Conference in the first place, meaning that the survey was effectively sampling opinions from one extreme tail of the opinion distribution on the subject. Indeed, the conference report itself stated that the findings should be taken 'with a grain of salt'.. Therefore, it is asked: will there be zero living humans on planet earth on January 1, 2100?. For these purposes we'll define humans as biological creatures who have as their ancestors via a chain of live births from mothers circa 2000 humans OR who could mate with circa 2000 humans to produce viable offspring. (So AIs, ems, genetically engineered beings of a different species brought up in artificial wombs, etc. would not count.). N.B. Even though it is obviously the case that if human extinction occurs Metaculus points won't be very valuable anymore and that it will be practically impossible to check for true human extinction (zero humans left), I would like to ask people not to let this fact influence their prediction and to predict in good faith.[{"name":"Yes","probability":0.08,"type":"PROBABILITY"},{"name":"No","probability":0.92,"type":"PROBABILITY"}]2
9Existential disaster will do us inhttps://www.nickbostrom.com/existential/risks.htmlX-risk estimatesActual estimate: Probably at or above 25% [{"name":"Yes","probability":0.25,"type":"PROBABILITY"},{"name":"No","probability":0.75,"type":"PROBABILITY"}]2
10Humanity will cease to exist before 5,100 years or thrive beyond 7.8 million yearshttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimates[{"name":"Yes","probability":0.05,"type":"PROBABILITY"},{"name":"No","probability":0.95,"type":"PROBABILITY"}]2
11Annual probability as of 2009 of extinctionhttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: 0.3-0.4% [{"name":"Yes","probability":0.0035,"type":"PROBABILITY"},{"name":"No","probability":0.9965,"type":"PROBABILITY"}]2
12Global catastrophic risk per year.https://arxiv.org/abs/1611.03072X-risk estimatesActual estimate: 0.2% Beard et al. seem to imply this is about extinction, but the quote suggests it's about "global catastrophic risk".[{"name":"Yes","probability":0.002,"type":"PROBABILITY"},{"name":"No","probability":0.998,"type":"PROBABILITY"}]2
13Humanity avoids every existential catastrophe and eventually fulfils its potential: achieving something close to the best future open to ushttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: 50% (~1 in 2) [{"name":"Yes","probability":0.5,"type":"PROBABILITY"},{"name":"No","probability":0.5,"type":"PROBABILITY"}]2
14Sentient life will survive for at least billions of yearshttps://forum.effectivealtruism.org/posts/MSYhEatxkEfg46j3D/the-case-of-the-missing-cause-prioritisation-research?commentId=iWkoScDxocaAJE4JgX-risk estimatesActual estimate: >20% I think it's fairly likely(>20%) that sentient life will survive for at least billions of years; and that there may be a fair amount of lock-in, so changing the trajectory of things could be great.[{"name":"Yes","probability":0.2,"type":"PROBABILITY"},{"name":"No","probability":0.8,"type":"PROBABILITY"}]2
15Existential catastrophe by 2120 as a result of unaligned AIhttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: ~10% [{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
16Human extinction by 2100 as a result of superintelligent AIhttps://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. Beard et al.'s appendix says "Note that for these predictions no time frame was given." I think that that's incorrect, based on phrasings in the original source, but I'm not certain.[{"name":"Yes","probability":0.05,"type":"PROBABILITY"},{"name":"No","probability":0.95,"type":"PROBABILITY"}]2
17Extremely bad (e.g. extinction) long-run impact on humanity from “high-level machine intelligencehttps://arxiv.org/abs/1705.08807X-risk estimatesThe report's authors discuss potential concerns around non-response bias and the fact that “NIPS and ICML authors are representative of machine learning but not of the field of artificial intelligence as a whole”. There was also evidence of apparent inconsistencies in estimates of AI timelines as a result of small changes to how questions were asked, providing further reason to wonder how meaningful these experts predictions were. https://web.archive.org/web/20171030220008/https://aiimpacts.org/some-survey-results/[{"name":"Yes","probability":0.05,"type":"PROBABILITY"},{"name":"No","probability":0.95,"type":"PROBABILITY"}]2
18A state where civilization collapses and does not recover, or a situation where all human life ends, due to AIhttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: 0-10% [{"name":"Yes","probability":0.05,"type":"PROBABILITY"},{"name":"No","probability":0.95,"type":"PROBABILITY"}]2
19AI causing an existential catastrophe in the next centuryhttps://forum.effectivealtruism.org/posts/7gxtXrMeqw78ZZeY9/ama-or-discuss-my-80k-podcast-episode-ben-garfinkel-fhi?commentId=uxiKooRc6d7JpjMSgX-risk estimatesActual estimate: ~0.1-1% Garfinkel was asked for his estimate during an AMA, and replied "I currently give it something in the .1%-1% range.[{"name":"Yes","probability":0.055,"type":"PROBABILITY"},{"name":"No","probability":0.945,"type":"PROBABILITY"}]2
20Chance that AI, through adversarial optimization against humans only, will cause existential catastrophehttps://www.lesswrong.com/posts/TdwpN484eTbPSvZkm/rohin-shah-on-reasons-for-ai-optimismX-risk estimatesActual estimate: ~5% This is my interpretation of some comments that may not have been meant to be taken very literally. Elsewhere, Rohin noted that this was “[his] opinion before updating on other people's views": https://forum.effectivealtruism.org/posts/tugs9KQyNqi4yRTsb/does-80-000-hours-focus-too-much-on-ai-risk#ZmtPji3pQaZK7Y4FF I think he updated this in 2020 to ~9%, due to pessimism about discontinuous scenarios: https://www.lesswrong.com/posts/TdwpN484eTbPSvZkm/rohin-shah-on-reasons-for-ai-optimism?commentId=n577gwGB3vRpwkBmj Rohin also discusses his estimates here: https://futureoflife.org/2020/04/15/an-overview-of-technical-ai-alignment-in-2018-and-2019-with-buck-shlegeris-and-rohin-shah/[{"name":"Yes","probability":0.05,"type":"PROBABILITY"},{"name":"No","probability":0.95,"type":"PROBABILITY"}]2
21AI-induced existential catastrophehttps://futureoflife.org/2020/04/15/an-overview-of-technical-ai-alignment-in-2018-and-2019-with-buck-shlegeris-and-rohin-shah/X-risk estimates[{"name":"Yes","probability":0.5,"type":"PROBABILITY"},{"name":"No","probability":0.5,"type":"PROBABILITY"}]2
22Existential risk from unaligned AI over the coming 100 yearshttps://forum.effectivealtruism.org/posts/2sMR7n32FSvLCoJLQ/critical-review-of-the-precipice-a-reassessment-of-the-risksX-risk estimatesActual estimate: 0.05% This was a direct response to Ord's estimate. It focuses on one pathway to x-risk from AI, not all pathways (e.g., not AI misuse or risks from competition between powerful AIs). "These estimates should not be taken very seriously. I do not believe we have enough information to make sensible quantitative estimates about these eventualities. Nevertheless, I present my estimates largely in order to illustrate the extent of my disagreement with Ords estimates, and to illustrate the key considerations I examine in order to arrive at an estimate." In comments on the source, Rohin Shah critiques some of the inputs to this estimate, and provides his own, substantially higher estimates.[{"name":"Yes","probability":0.0005,"type":"PROBABILITY"},{"name":"No","probability":0.9995,"type":"PROBABILITY"}]2
23Existential risk from AIhttps://youtu.be/WLXuZtWoRcE?t=1229X-risk estimatesActual estimate: 5-30% I put the probability that [AI/AGI] is an existential risk roughly in the 30% to 5% range, depending on how the problem is phrased." I assume he means the probability of existential catastrophe from AI/AGI, not the probability that AI/AGI poses an existential risk.[{"name":"Yes","probability":0.175,"type":"PROBABILITY"},{"name":"No","probability":0.825,"type":"PROBABILITY"}]2
24Chance of humanity not surviving AIhttps://www.youtube.com/watch?v=i4LjoJGpqIY& (from 39:40)X-risk estimatesActual estimate: 50, 40, or 33% Stated verbally during an interview. Not totally clear precisely what was being estimated (e.g. just extinction, or existential catastrophe more broadly?). He noted "This number fluctuates a lot". He indicated he thought we had a 2/3 chance of surviving, then said he'd adjust to 50%, which is his number for an "actually superintelligent" AI, whereas for "AI in general" it'd be 60%. This is notably higher than his 2020 estimate, implying either that he updated towards somewhat more "optimism" between 2014 and 2020, or that one or both of these estimates don't reflect stable beliefs.[{"name":"Yes","probability":0.4,"type":"PROBABILITY"},{"name":"No","probability":0.6,"type":"PROBABILITY"}]2
25Existential catastrophe happening this century (maybe just from AI?)https://youtu.be/aFAI8itZCGk?t=854X-risk estimatesActual estimate: 33-50% This comes from a verbal interview (from the 14:14 mark). The interview was focused on AI, and this estimate may have been as well. Tallinn said he's not very confident, but is fairly confident his estimate would be in double-digits, and then said "two obvious Schelling points" are 33% or 50%, so he'd guess somewhere in between those. Other comments during the interview seem to imply Tallinn is either just talking about extinction risk or thinks existential risk happens to be dominated by extinction risk.[{"name":"Yes","probability":0.415,"type":"PROBABILITY"},{"name":"No","probability":0.585,"type":"PROBABILITY"}]2
26Existential catastrophe from engineered pandemics by 2120https://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: ~3% (~1 in 30) [{"name":"Yes","probability":0.03,"type":"PROBABILITY"},{"name":"No","probability":0.97,"type":"PROBABILITY"}]2
27Human extinction by 2100 as a result of the single biggest natural pandemichttps://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesActual estimate: 0.05% This is the median. Beard et al.'s appendix says "Note that for these predictions no time frame was given." I think that that's incorrect, based on phrasings in the original source, but I'm not certain.[{"name":"Yes","probability":0.0005,"type":"PROBABILITY"},{"name":"No","probability":0.9995,"type":"PROBABILITY"}]2
28Existential catastrophe from naturally arising pandemics by 2120https://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: ~0.01% (~1 in 10,000) [{"name":"Yes","probability":0.0001,"type":"PROBABILITY"},{"name":"No","probability":0.9999,"type":"PROBABILITY"}]2
29Human extinction by 2100 as a result of single biggest engineered pandemichttps://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. Beard et al.'s appendix says "Note that for these predictions no time frame was given." I think that that's incorrect, based on phrasings in the original source, but I'm not certain.[{"name":"Yes","probability":0.02,"type":"PROBABILITY"},{"name":"No","probability":0.98,"type":"PROBABILITY"}]2
30Annual probability of an existential catastrophe arising from a global pandemichttps://www.liebertpub.com/doi/10.1089/hs.2017.0028X-risk estimatesActual estimate: 0.008% to 0.0000016% (between 8 x 10-5 and 1.6 x 10-8) The fact that there's a separate estimate from the same source for biowarfare and bioterrorism suggests to me that this is meant to be an estimate of the risk from a natural pandemic only. But I'm not sure. This might also include "accidental" release of a bioengineered pathogen.[{"name":"Yes","probability":0.00004,"type":"PROBABILITY"},{"name":"No","probability":0.99996,"type":"PROBABILITY"}]2
31Annual probability of an existential catastrophe arising from biowarfare or bioterrorismhttps://www.liebertpub.com/doi/10.1089/hs.2017.0028X-risk estimatesActual estimate: 0.00019% (0.0000019) [{"name":"Yes","probability":0.0000019,"type":"PROBABILITY"},{"name":"No","probability":0.9999981,"type":"PROBABILITY"}]2
32Civilization collapses and does not recover, or a situation where all human life ends due to a global pandemichttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: 0.0001% The fact that there's a separate estimate from the same source for "synthetic biology" suggests to me that this is meant to be an estimate of the risk from a natural pandemic only.[{"name":"Yes","probability":0.000001,"type":"PROBABILITY"},{"name":"No","probability":0.999999,"type":"PROBABILITY"}]2
33Civilization collapses and does not recover, or a situation where all human life ends, due to synthetic biologyhttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: 0.0001% [{"name":"Yes","probability":0.000001,"type":"PROBABILITY"},{"name":"No","probability":0.999999,"type":"PROBABILITY"}]2
34Extinction risk from engineered pandemics over the coming 100 yearshttps://forum.effectivealtruism.org/posts/2sMR7n32FSvLCoJLQ/critical-review-of-the-precipice-a-reassessment-of-the-risksX-risk estimatesActual estimate: 0.0002% This was a direct response to Ord's estimate, although this estimate is of extinction risk rather than existential risk. "These estimates should not be taken very seriously. I do not believe we have enough information to make sensible quantitative estimates about these eventualities. Nevertheless, I present my estimates largely in order to illustrate the extent of my disagreement with Ords estimates, and to illustrate the key considerations I examine in order to arrive at an estimate." In comments on the source, Will Bradshaw critiques some of the inputs to this estimate.[{"name":"Yes","probability":0.000002,"type":"PROBABILITY"},{"name":"No","probability":0.999998,"type":"PROBABILITY"}]2
35Human extinction by 2100 as a result of molecular nanotech weaponshttps://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. Beard et al.'s appendix says "Note that for these predictions no time frame was given." I think that that's incorrect, based on phrasings in the original source, but I'm not certain.[{"name":"Yes","probability":0.05,"type":"PROBABILITY"},{"name":"No","probability":0.95,"type":"PROBABILITY"}]2
36Human extinction by 2100 as a result of the single biggest nanotech accidenthttps://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesActual estimate: 0.5% This is the median. Beard et al.'s appendix says "Note that for these predictions no time frame was given." I think that that's incorrect, based on phrasings in the original source, but I'm not certain.[{"name":"Yes","probability":0.005,"type":"PROBABILITY"},{"name":"No","probability":0.995,"type":"PROBABILITY"}]2
37Civilization collapses and does not recover, or a situation where all human life ends due to nanotechnologyhttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: 0.0100% [{"name":"Yes","probability":0.0001,"type":"PROBABILITY"},{"name":"No","probability":0.9999,"type":"PROBABILITY"}]2
38Existential catastrophe from other anthropogenic risks (which includes but is not limited to nanotechnology) by 2120https://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=0X-risk estimatesActual estimate: ~2% (~1 in 50) See this post for some commentary: [Some thoughts on Toby Ords existential risk estimates](https://forum.effectivealtruism.org/posts/Z5KZ2cui8WDjyF6gJ/my-thoughts-on-toby-ord-s-existential-risk-estimates#_Unforeseen__and__other__anthropogenic_risks__Surprisingly_risky_)[{"name":"Yes","probability":0.02,"type":"PROBABILITY"},{"name":"No","probability":0.98,"type":"PROBABILITY"}]2
39Total existential risk by 2120 if we just carry on as we are, with business as usual (which Ord doesn't expect us to do)https://80000hours.org/podcast/episodes/toby-ord-the-precipice-existential-risk-future-humanity/#estimates-for-specific-x-risks-000810X-risk estimatesActual estimate: ~33% ("about one in three") Ord: "one in six is my best guess as to the chance [an existential catastrophe] happens [by 2120]. Thats not a business as usual estimate. Whereas I think often people are assuming that estimates like this are, if we just carry on as we are, whats the chance that something will happen? My best guess for that is actually about one in three this century. If we carry on mostly ignoring these risks with humanitys escalating power during the century and some of these threats being very serious. But I think that theres a good chance that we will rise to these challenges and do something about them. So you could think of my overall estimate as being something like Russian roulette, but my initial business as usual estimate being theres something like two bullets in the chamber of the gun, but then well probably remove one and that if we really got our act together, we could basically remove both of them. And so, in some sense, maybe the headline figure should be one in three being the difference between the business as usual risk and how much of that we could eliminate if we really got our act together."" Arden Koehler replies ""Okay. So business as usual means doing what we are approximately doing now extrapolated into the future but we dont put much more effort into it as opposed to doing nothing at all?"" Ord replies: ""Thats right, and it turns out to be quite hard to define business as usual. Thats the reason why, for my key estimate, that I make it… In some sense, its difficult to define estimates where they take into account whether or not people follow the advice that youre giving; that introduces its own challenges. But at least thats just what a probability normally means. It means that your best guess of the chance something happens, whereas a best guess that something happens conditional upon certain trends either staying at the same level or continuing on the same trajectory or something is just quite a bit more unclear as to what youre even talking about.""[{"name":"Yes","probability":0.33,"type":"PROBABILITY"},{"name":"No","probability":0.6699999999999999,"type":"PROBABILITY"}]2
40The probability that the long-run overall impact on humanity of human level machine intelligence will be Extremely bad (existential catastrophe), assuming Human Level Machine Intelligence will at some point exist.https://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=511918904X-risk estimatesThis is the mean. According to Beard et al., the question was "4. Assume for the purpose of this question that such Human Level Machine Intelligence (HLMI) will at some point exist. How positive or negative would be overall impact on humanity, in the long run?[{"name":"Yes","probability":0.18,"type":"PROBABILITY"},{"name":"No","probability":0.8200000000000001,"type":"PROBABILITY"}]2
41Chance that AI, through “adversarial optimization against humans only”, will cause existential catastrophe, conditional on there not being “additional intervention by longtermists” (or perhaps “no intervention from longtermists”)https://www.lesswrong.com/posts/TdwpN484eTbPSvZkm/rohin-shah-on-reasons-for-ai-optimismX-risk estimatesActual estimate: ~10% This is my interpretation of some comments that may not have been meant to be taken very literally. I think he updated this in 2020 to ~15%, due to pessimism about discontinuous scenarios: https://www.lesswrong.com/posts/TdwpN484eTbPSvZkm/rohin-shah-on-reasons-for-ai-optimism?commentId=n577gwGB3vRpwkBmj Rohin also discusses his estimates here: https://futureoflife.org/2020/04/15/an-overview-of-technical-ai-alignment-in-2018-and-2019-with-buck-shlegeris-and-rohin-shah/[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
42Chance that AI, through “adversarial optimization against humans only”, will cause existential catastrophe, conditional on “discontinuous takeoff”https://www.lesswrong.com/posts/TdwpN484eTbPSvZkm/rohin-shah-on-reasons-for-ai-optimismX-risk estimatesActual estimate: ~70% (but with “way more uncertainty” than his other estimates) [{"name":"Yes","probability":0.7,"type":"PROBABILITY"},{"name":"No","probability":0.30000000000000004,"type":"PROBABILITY"}]2
43Chance that we don't manage to survive that transition [to there being something that's more intelligent than humanity], being in charge of our future.https://80000hours.org/podcast/episodes/toby-ord-the-precipice-existential-risk-future-humanity/#transcriptX-risk estimatesActual estimate: ~20% This may have been specifically if the transition happens in the net 100 years; it's possible Ord would estimate we'd have a different chance if this transition happened at a later time. "Basically, you can look at my [estimate that the existential risk from AI in the next 100 years is] 10% as, theres about a 50% chance that we create something thats more intelligent than humanity this century. And then theres only an 80% chance that we manage to survive that transition, being in charge of our future. If you put that together, you get a 10% chance thats the time where we lost control of the future in a negative way. [For people who would disagree, a question] is why would they think that we have much higher than an 80% chance of surviving this passing this baton to these other entities, but still retaining control of our future or making sure that they build a future that is excellent, surpassingly good by our own perspective? I think that the very people who are working on trying to actually make sure that artificial intelligence would be aligned with our values are finding it extremely difficult. Theyre not that hopeful about it. So it seems hard to think theres more than 80% chance, based on what we know, to get through that.[{"name":"Yes","probability":0.2,"type":"PROBABILITY"},{"name":"No","probability":0.8,"type":"PROBABILITY"}]2
44Chance that a full-scale nuclear war in the next century would be the end of human potentialhttps://80000hours.org/podcast/episodes/toby-ord-the-precipice-existential-risk-future-humanity/#transcriptX-risk estimatesActual estimate: ~2% I give existential risk over the next century from nuclear war at about one in a thousand. I initially thought it would be higher than that. Thats actually something that while researching the book, thought was a lower risk than I had initially thought. And how Id break it down is to something like a 5% chance of a full-scale nuclear war in the next century and a 2% chance that that would be the end of human potential." Ord discusses his reasoning more both in that interview and in The Precipice.[{"name":"Yes","probability":0.02,"type":"PROBABILITY"},{"name":"No","probability":0.98,"type":"PROBABILITY"}]2
45Global human population of zero resulting from the 150 Tg of black carbon scenario in our 2007 paperhttp://www.overcomingbias.com/2012/11/nuclear-winter-and-human-extinction-qa-with-luke-oman.htmlX-risk estimatesActual estimate: 0.001-0.01% (“in the range of 1 in 10,000 to 1 in 100,000”) I think that this is Omans estimate of the chance that extinction would occur if that black carbon scenario occurred, rather than an estimate that also takes into account the low probability that that black carbon scenario occurs. I.e., I think that this estimate was conditional on a particular type of nuclear war occurring. But Im not sure about that, and the full context doesnt make it much clearer.[{"name":"Yes","probability":0.000055,"type":"PROBABILITY"},{"name":"No","probability":0.999945,"type":"PROBABILITY"}]2
46Full-scale collapse of society, perhaps due to very, very widespread famine, if there's 2 degrees of warminghttps://80000hours.org/podcast/episodes/mark-lynas-climate-change-nuclear-energy/X-risk estimatesArden Koehler: "...do you have a guess at what degree of warming we would need to reach for the full-scale collapse of society, perhaps due to very, very widespread famine to have say a 10% chance of happening? Mark Lynas: "Oh, I think… You want to put me on the spot. I would say it has a 30 to 40% chance of happening at three degrees, and a 60% chance of happening at four degrees, and 90% at five degrees, and 97% at six degrees. [...] Maybe 10% at two degrees.[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
47Full-scale collapse of society, perhaps due to very, very widespread famine, if there's 3 degrees of warminghttps://80000hours.org/podcast/episodes/mark-lynas-climate-change-nuclear-energy/X-risk estimatesActual estimate: 30-40% Arden Koehler: "...do you have a guess at what degree of warming we would need to reach for the full-scale collapse of society, perhaps due to very, very widespread famine to have say a 10% chance of happening? Mark Lynas: "Oh, I think… You want to put me on the spot. I would say it has a 30 to 40% chance of happening at three degrees, and a 60% chance of happening at four degrees, and 90% at five degrees, and 97% at six degrees. [...] Maybe 10% at two degrees.[{"name":"Yes","probability":0.35,"type":"PROBABILITY"},{"name":"No","probability":0.65,"type":"PROBABILITY"}]2
48Full-scale collapse of society, perhaps due to very, very widespread famine, if there's 4 degrees of warminghttps://80000hours.org/podcast/episodes/mark-lynas-climate-change-nuclear-energy/X-risk estimatesArden Koehler: "...do you have a guess at what degree of warming we would need to reach for the full-scale collapse of society, perhaps due to very, very widespread famine to have say a 10% chance of happening? Mark Lynas: "Oh, I think… You want to put me on the spot. I would say it has a 30 to 40% chance of happening at three degrees, and a 60% chance of happening at four degrees, and 90% at five degrees, and 97% at six degrees. [...] Maybe 10% at two degrees.[{"name":"Yes","probability":0.6,"type":"PROBABILITY"},{"name":"No","probability":0.4,"type":"PROBABILITY"}]2
49Full-scale collapse of society, perhaps due to very, very widespread famine, if there's 5 degrees of warminghttps://80000hours.org/podcast/episodes/mark-lynas-climate-change-nuclear-energy/X-risk estimatesArden Koehler: "...do you have a guess at what degree of warming we would need to reach for the full-scale collapse of society, perhaps due to very, very widespread famine to have say a 10% chance of happening? Mark Lynas: "Oh, I think… You want to put me on the spot. I would say it has a 30 to 40% chance of happening at three degrees, and a 60% chance of happening at four degrees, and 90% at five degrees, and 97% at six degrees. [...] Maybe 10% at two degrees.[{"name":"Yes","probability":0.9,"type":"PROBABILITY"},{"name":"No","probability":0.09999999999999998,"type":"PROBABILITY"}]2
50Full-scale collapse of society, perhaps due to very, very widespread famine, if there's 6 degrees of warminghttps://80000hours.org/podcast/episodes/mark-lynas-climate-change-nuclear-energy/X-risk estimatesArden Koehler: "...do you have a guess at what degree of warming we would need to reach for the full-scale collapse of society, perhaps due to very, very widespread famine to have say a 10% chance of happening? Mark Lynas: "Oh, I think… You want to put me on the spot. I would say it has a 30 to 40% chance of happening at three degrees, and a 60% chance of happening at four degrees, and 90% at five degrees, and 97% at six degrees. [...] Maybe 10% at two degrees.[{"name":"Yes","probability":0.97,"type":"PROBABILITY"},{"name":"No","probability":0.030000000000000027,"type":"PROBABILITY"}]2
51A world totalitarian government will emerge during the next one thousand years and last for a thousand years or more, conditional on genetic screening for personality traits becom[ing] cheap and accurate, but the principle of reproductive freedom prevail[ing]https://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=511918904X-risk estimatesReduced from his 5% unconditional probability[{"name":"Yes","probability":0.03,"type":"PROBABILITY"},{"name":"No","probability":0.97,"type":"PROBABILITY"}]2
52A world totalitarian government will emerge during the next one thousand years and last for a thousand years or more, conditional on genetic screening for personality traits becom[ing] cheap and accurate and extensive government regulationhttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=511918905X-risk estimatesIncreased from his 5% unconditional probability[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
53A world totalitarian government will emerge during the next one thousand years and last for a thousand years or more, conditional on the number of independent countries on earth [not decreasing] during the next thousand yearshttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=511918906X-risk estimatesActual estimate: 0.1% Reduced from his 5% unconditional probability[{"name":"Yes","probability":0.001,"type":"PROBABILITY"},{"name":"No","probability":0.999,"type":"PROBABILITY"}]2
54A world totalitarian government will emerge during the next one thousand years and last for a thousand years or more, conditional on the number of independent countries on earth [falling to 1] during the next thousand yearshttps://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid=511918907X-risk estimatesIncreased from his 5% unconditional probability[{"name":"Yes","probability":0.25,"type":"PROBABILITY"},{"name":"No","probability":0.75,"type":"PROBABILITY"}]2
55At least 1 million dead as a result of superintelligent AI before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
56At least 1 billion dead as a result of superintelligent AI before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A. Interestingly, this is the same as the estimate from this source of the chance of human as a result of superintelligent AI by 2100.[{"name":"Yes","probability":0.05,"type":"PROBABILITY"},{"name":"No","probability":0.95,"type":"PROBABILITY"}]2
57AI safety is as hard as a (caricature of) MIRI suggestshttps://aiimpacts.org/conversation-with-adam-gleave/X-risk estimatesActual estimate: ~10% So, decent chance I think I put a reasonable probability, like 10% probability, on the hard-mode MIRI version of the world being true. In which case, I think theres probably nothing we can do.[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
58AI safety basically [doesn't need] to be solved, well just solve it by default unless were completely completely carelesshttps://aiimpacts.org/conversation-with-adam-gleave/X-risk estimatesActual estimate: ~20-30% [{"name":"Yes","probability":0.25,"type":"PROBABILITY"},{"name":"No","probability":0.75,"type":"PROBABILITY"}]2
59The first thing we try just works and we dont even need to solve any sort of alignment problemhttps://futureoflife.org/2020/04/15/an-overview-of-technical-ai-alignment-in-2018-and-2019-with-buck-shlegeris-and-rohin-shah/X-risk estimatesActual estimate: ~30% Theres some chance that the first thing we try just works and we dont even need to solve any sort of alignment problem. It might just be fine. This is not implausible to me. Maybe thats 30% or something.[{"name":"Yes","probability":0.3,"type":"PROBABILITY"},{"name":"No","probability":0.7,"type":"PROBABILITY"}]2
60We have good competitive alignment techniques by the time that its importanthttps://futureoflife.org/2020/04/15/an-overview-of-technical-ai-alignment-in-2018-and-2019-with-buck-shlegeris-and-rohin-shah/X-risk estimatesActual estimate: ~30% I havent actually written down these numbers since I last changed my mind about a lot of the inputs to them, so maybe Im being really dumb. I guess, it feels to me that in fast takeoff worlds, we are very sad unless we have competitive alignment techniques, and so then were just only okay if we have these competitive alignment techniques. I guess I would say that Im something like 30% on us having good competitive alignment techniques by the time that its important, which incidentally is higher than Rohin I think. [...] So Im like 30% that we can just solve the AI alignment problem in this excellent way, such that anyone who wants to can have a little extra cost and then make AI systems that are aligned. I feel like in worlds where we did that, its pretty likely that things are reasonably okay.[{"name":"Yes","probability":0.3,"type":"PROBABILITY"},{"name":"No","probability":0.7,"type":"PROBABILITY"}]2
61We create something thats more intelligent than humanity in the next 100 yearshttps://80000hours.org/podcast/episodes/toby-ord-the-precipice-existential-risk-future-humanity/#transcriptX-risk estimatesActual estimate: ~50% Basically, you can look at my [estimate that the existential risk from AI in the next 100 years is] 10% as, theres about a 50% chance that we create something thats more intelligent than humanity this century. And then theres only an 80% chance that we manage to survive that transition, being in charge of our future. If you put that together, you get a 10% chance thats the time where we lost control of the future in a negative way. Toby Ord: With that number, Ive spent a lot of time thinking about this. Actually, my first degree was in computer science, and Ive been involved in artificial intelligence for a long time, although its not what I did my PhD on. But, if you ask the typical AI experts view of the chance that we develop smarter than human AGI, artificial general intelligence, this century is about 50%. If you survey the public, which has been done, its about 50%. So, my 50% is both based on the information I know actually about whats going on in AI, and also is in line with all of the relevant outside views. It feels difficult to have a wildly different number on that. The onus would be on the other person.[{"name":"Yes","probability":0.5,"type":"PROBABILITY"},{"name":"No","probability":0.5,"type":"PROBABILITY"}]2
62Soft AGI takeoffhttps://reducing-suffering.org/summary-beliefs-values-big-questions/X-risk estimates[{"name":"Yes","probability":0.7,"type":"PROBABILITY"},{"name":"No","probability":0.30000000000000004,"type":"PROBABILITY"}]2
63By at least 10 years before human-level AGI is built, debate about AGI risk will be as mainstream as global warming is in 2015https://reducing-suffering.org/summary-beliefs-values-big-questions/X-risk estimates[{"name":"Yes","probability":0.67,"type":"PROBABILITY"},{"name":"No","probability":0.32999999999999996,"type":"PROBABILITY"}]2
64A government will build the first human-level AGI, assuming humans build one at allhttps://reducing-suffering.org/summary-beliefs-values-big-questions/X-risk estimates[{"name":"Yes","probability":0.62,"type":"PROBABILITY"},{"name":"No","probability":0.38,"type":"PROBABILITY"}]2
65A government will build the first human-level AGI, assuming humans build one at allhttp://www.stafforini.com/blog/what_i_believe/X-risk estimates[{"name":"Yes","probability":0.6,"type":"PROBABILITY"},{"name":"No","probability":0.4,"type":"PROBABILITY"}]2
66Human-controlled AGI in expectation would result in less suffering than uncontrolledhttps://reducing-suffering.org/summary-beliefs-values-big-questions/X-risk estimates[{"name":"Yes","probability":0.52,"type":"PROBABILITY"},{"name":"No","probability":0.48,"type":"PROBABILITY"}]2
67A design very close to CEV will be implemented in humanity's AGI, conditional on AGI being built (excluding other value-learning approaches and other machine-ethics proposals)https://reducing-suffering.org/summary-beliefs-values-big-questions/X-risk estimatesActual estimate: 0.5% [{"name":"Yes","probability":0.005,"type":"PROBABILITY"},{"name":"No","probability":0.995,"type":"PROBABILITY"}]2
68A design very close to CEV will be implemented in humanity's AGI, conditional on AGI being built (excluding other value-learning approaches and other machine-ethics proposals)http://www.stafforini.com/blog/what_i_believe/X-risk estimates[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
69At least 1 million dead as a result of the single biggest engineered pandemic before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.3,"type":"PROBABILITY"},{"name":"No","probability":0.7,"type":"PROBABILITY"}]2
70At least 1 billion dead as a result of the single biggest engineered pandemic before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
71At least 1 million dead as a result of the single biggest natural pandemic before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.6,"type":"PROBABILITY"},{"name":"No","probability":0.4,"type":"PROBABILITY"}]2
72At least 1 billion dead as a result of the single biggest natural pandemic before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.05,"type":"PROBABILITY"},{"name":"No","probability":0.95,"type":"PROBABILITY"}]2
73At least 1 million dead as a result of molecular nanotech weapons before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.25,"type":"PROBABILITY"},{"name":"No","probability":0.75,"type":"PROBABILITY"}]2
74At least 1 billion dead as a result of molecular nanotech weapons before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
75At least 1 million dead as a result of the single biggest nanotech accident before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.05,"type":"PROBABILITY"},{"name":"No","probability":0.95,"type":"PROBABILITY"}]2
76At least 1 billion dead as a result of the single biggest nanotech accident before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.01,"type":"PROBABILITY"},{"name":"No","probability":0.99,"type":"PROBABILITY"}]2
77At least 1 million dead as a result of all nuclear wars before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.3,"type":"PROBABILITY"},{"name":"No","probability":0.7,"type":"PROBABILITY"}]2
78At least 1 billion dead as a result of all nuclear wars before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
79At least 1 million dead as a result of all acts of nuclear terrorism before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.3,"type":"PROBABILITY"},{"name":"No","probability":0.7,"type":"PROBABILITY"}]2
80At least 1 billion dead as a result of all acts of nuclear terrorism before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimatesThis is the median. The report about these estimates also plots the results for each question “with individual response distributions visible” in Appendix A.[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2
81chance of a full-scale nuclear war in the next centuryhttps://80000hours.org/podcast/episodes/toby-ord-the-precipice-existential-risk-future-humanity/#transcriptX-risk estimatesActual estimate: ~5% I give existential risk over the next century from nuclear war at about one in a thousand. I initially thought it would be higher than that. Thats actually something that while researching the book, thought was a lower risk than I had initially thought. And how Id break it down is to something like a 5% chance of a full-scale nuclear war in the next century and a 2% chance that that would be the end of human potential." Ord discusses his reasoning more both in that interview and in The Precipice.[{"name":"Yes","probability":0.05,"type":"PROBABILITY"},{"name":"No","probability":0.95,"type":"PROBABILITY"}]2
82Per year chance of nuclear warhttps://forum.effectivealtruism.org/posts/PAYa6on5gJKwAywrF/how-likely-is-a-nuclear-exchange-between-the-us-and-russia-1X-risk estimatesActual estimate: 1.10% In this post, I get a rough sense of how probable a nuclear war might be by looking at historical evidence, the views of experts, and predictions made by forecasters. I find that, if we aggregate those perspectives, theres about a 1.1% chance of nuclear war each year, and that the chances of a nuclear war between the US and Russia, in particular, are around 0.38% per year." This is not presented as Luisa's own credence; this may not be the number she herself would give. Readers may also be interested in the estimates implied by each of the perspectives Luisa aggregates.[{"name":"Yes","probability":0.011,"type":"PROBABILITY"},{"name":"No","probability":0.989,"type":"PROBABILITY"}]2
83Per year chance of nuclear war between the US and Russiahttps://forum.effectivealtruism.org/posts/PAYa6on5gJKwAywrF/how-likely-is-a-nuclear-exchange-between-the-us-and-russia-1X-risk estimatesActual estimate: 0.38% In this post, I get a rough sense of how probable a nuclear war might be by looking at historical evidence, the views of experts, and predictions made by forecasters. I find that, if we aggregate those perspectives, theres about a 1.1% chance of nuclear war each year, and that the chances of a nuclear war between the US and Russia, in particular, are around 0.38% per year." This is not presented as Luisa's own credence; this may not be the number she herself would give. Readers may also be interested in the estimates implied by each of the perspectives Luisa aggregates.[{"name":"Yes","probability":0.0038,"type":"PROBABILITY"},{"name":"No","probability":0.9962,"type":"PROBABILITY"}]2
84Climate change will cause more suffering than it preventshttps://reducing-suffering.org/summary-beliefs-values-big-questions/X-risk estimates[{"name":"Yes","probability":0.5,"type":"PROBABILITY"},{"name":"No","probability":0.5,"type":"PROBABILITY"}]2
85At least 1 million dead as a result of all wars (including civil wars) before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimates[{"name":"Yes","probability":0.98,"type":"PROBABILITY"},{"name":"No","probability":0.020000000000000018,"type":"PROBABILITY"}]2
86At least 1 billion dead as a result of all wars (including civil wars) before 2100https://www.fhi.ox.ac.uk/reports/2008-1.pdfX-risk estimates[{"name":"Yes","probability":0.3,"type":"PROBABILITY"},{"name":"No","probability":0.7,"type":"PROBABILITY"}]2
87Human-inspired colonization of space will cause more suffering than it prevents if it happenshttps://reducing-suffering.org/summary-beliefs-values-big-questions/X-risk estimates[{"name":"Yes","probability":0.72,"type":"PROBABILITY"},{"name":"No","probability":0.28,"type":"PROBABILITY"}]2
88Earth will eventually be controlled by a singleton of some sorthttps://reducing-suffering.org/summary-beliefs-values-big-questions/X-risk estimates[{"name":"Yes","probability":0.72,"type":"PROBABILITY"},{"name":"No","probability":0.28,"type":"PROBABILITY"}]2
89Earth will eventually be controlled by a singleton of some sorthttp://www.stafforini.com/blog/what_i_believe/X-risk estimates[{"name":"Yes","probability":0.7,"type":"PROBABILITY"},{"name":"No","probability":0.30000000000000004,"type":"PROBABILITY"}]2
90Earth-originating intelligence will colonize the entire galaxy (ignoring anthropic arguments)https://reducing-suffering.org/summary-beliefs-values-big-questions/X-risk estimates[{"name":"Yes","probability":0.5,"type":"PROBABILITY"},{"name":"No","probability":0.5,"type":"PROBABILITY"}]2
91Earth-originating intelligence will colonize the entire galaxy (ignoring anthropic arguments)http://www.stafforini.com/blog/what_i_believe/X-risk estimates[{"name":"Yes","probability":0.1,"type":"PROBABILITY"},{"name":"No","probability":0.9,"type":"PROBABILITY"}]2