Analysis of some predictions about the 2018 EA Survey

Introduction.

Some effective altruists made predictions about the 2018 EA Survey: a survey which aims to reach most people within the effective altruism movement. Here, I present the set up for the prediction making, the questions, and explain some judgement calls I made when judging the answers. Everything is written such that you can play along.

Set up

For every question, try to come up with an interval such that you're 80% confident the answer lies in it. If you use a search engine, the surveys from previous years are fair game.

Judgement call

In some cases, people didn't answer the question. For example, under the is.veg variable, you can have TRUE, FALSE, or NA: Not Available. If their number is respectively x, y and z, it might be a good first order approximation to estimate the actual proportion of vegetarians/vegans as x/(x+y).

However, I've decided to be extremely anal about it, and choose to define the actual proportion of people who define as vegan as x/(x+y+z). To do otherwise would be to replace questions. This doesn't make much of a difference in the case of plant eating, but it does in the identity politics questions. Curiously, doing so raises the average number of questions participants got right, but not by much.

Questions

How many people do you think will take the EA survey?
What percent of people will say they got involved in EA in 2017?
What percent of people will say they first heard about EA from a personal contact?
What percent of people will say they first heard about EA from LessWrong?
What percent of people will say they are involved in a local EA group?
What percent of people will say they have taken the Giving What We Can pledge?
What percent of people will say they find the EA community either welcoming or very welcoming?
What percent of people will say they find the EA community either unwelcoming or very unwelcoming?
What percent of people will say that global poverty should be the top or near top priority?
What percent of people will say that cause prioritisation should be the top or near top priority?
What percent of people will say that reducing risks from AI should be the top or near top priority?
What percent of people will say that animal welfare/rights should be the top or near top priority?
What percent of people will say that meta charities should be the top or near top priority?
What percent of people who took the survey will be between 20 and 35 years of age?
What percent of people will identify as male?
What percent of people will identify as white?
What percent of people will say they live in the US or UK?
What percent of people will say they live in continental Europe?
What percent of people will say they are atheist, agnostic or non-religious?
What percent of people will say they are vegan or vegetarian?
What percent of people will say they are politically on the left or centre left?
What percent of people will say they are politically on the centre?
What percent of people will say they are politically on the right or centre right?
What percent of people will say that they are single?
What percent of people will say that they are employed?
What percent of people will say that they are a student?

Answers

22.20943613
33.17990027
11.73762946
15.53509781
25.77675489
59.37859609
4.75642501
61.33486766
41.46528577
44.1503644
35.28960491
34.06214039
NA
59.76217875
78.17414653
52.35903337
NA
72.0751822
38.43498274
60.98964327
8.630609896
2.685078634
35.82662064
52.5508247
26.50556195

Results

For the 35 people who took part in the original prediction making, their results can be seen in the following graphics:

The average accuracy is 55.12%, that is, the average participant got 13.22 out of 24 questions right. If it had been reached, a target credence of 80% would imply an average of 19.2 correct answers. In other words, in this limited domain, when these people say 80%, the thing happens 55% of the time. If they bet, they'd be replacing ~1:1 bets with 1:4 bets.

4.4 KiB Raw Blame History