nunosempere.github.io/ESPR-Evaluation/4-Measurements.md

Measurements

Difficulties

The changes which through ESPR could be induced in the students are, in some sense, fuzzy and soft. There is some tension between measuring what is easiest to measure and measuring what we're actually interested in, and we firmly choose the second kind. For example, when measuring openness, we don't care about questions such as:

I see Myself as Someone Who...

• Is original, comes up with new ideas
• Is curious about many different things
• Is ingenious, a deep thinker
• Has an active imagination
• Is inventive
• Values artistic, aesthetic experiences
• Prefers work that is routine
• Likes to reflect, play with ideas
• Has few artistic interests
• Is sophisticated in art, music, or literature

From John, O. P., & Srivastava, S. (1999). The Big-Five trait taxonomy: History, measurement, and theoretical perspectives. In L. A. Pervin & O. P. John (Eds.), Handbook of personality: Theory and research (Vol. 2, pp. 102–138). New York: Guilford Press.

But instead want to ask things such as:

• What was the last time you tried out something new?
• How often do you try something new?
• How much have you explored vs specialized in the last year?
• What was the last time you did something which you thought had a <=5% chance of succeeding?

Note that the 2015 CFAR Longitudinal study takes a different approach:

"We relied heavily on existing measures which have been validated and used by psychology researchers, especially in the areas of well-being and personality. These measures typically are not a perfect match for what we care about, but we expected them to be sufficiently correlated with what we care about for them to be worth using"

For example, they used the questions written above, but they'd be insufficient to capture the effects of CoZE, one of the highest impact activities in a CFAR Workshop.

Things we want to measure.

- and ways to measure them.

Recommend a song which lasts roughly as long as it should take to complete the survey.

Every time you lie or exaggerate, a kitten dies. By answering this survey, you help make the world a better place.

If you find yourself fatigued by the length of the survey feel free to take a break and come back. It is also preferable to just go to the end and turn in what you have. Some questions are explicitly marked 'bonus' or 'optional' meaning they are especially skippable.

1. Demographic information: Ask for consent for aggregation / doing a study on this. ✓ Can we include your survey data in a public dataset? ✓ Ask for the email. Followup survey. ✓ Ikea: birthdate: dd.mm.yyyy + initials + first letter of the country you were born with. Age / Sex assigned at birth / Gender / Country (if many, the one you most identify with) / Ethnic group (most identify) / sexual orientation ✓

2. Choices influenced by espr.

• Average prestigiousness of the universities to which the apply / to which they get in.
• % people who are not going to university.
• Do you feel like you've made a life-changing choice in the last year? If you have: Write a brief tweet.
• Do you feel like your life has significantly changed in the last year? If you have: Write a brief tweet.
• Do you feel like the course of your life has significantly changed in the last year? If you have: Write a brief tweet.

1. Self-Confidence/ Modern Survival Skills.

• I think I could do pretty well in a Zombie Apocalypse. ✓
• It wouldn't bother me excessively if I woke up in a random city in the world with nothing but my clothes on. ✓

1. Decisiveness.

• To what extent do you agree with the statement: I am a decisive person.
• To what extent would your friends agree with the statement: You are a decisive person.
• What was the last time you did something which you thought had <=5% of succeeding? Why did you attempt it? Also, describe it. ✓
• To what extent do you struggle with doing things you've decided to do? ✓

1. Openess to new experiences?

• What was the last time you tried out something new? ✓
• How often do you try something new?

1. People, connections.

• What is the approximate number of people who you interacted with in the past week? ✓
• What is the approximate number of people you'd be willing to confide in about something personal? ✓
• What is the approximate number of people who would let you crash at their place if you needed somewhere to stay? ✓ Their numerical responses were then capped at 300, log transformed, and averaged into a single measure of social support.

1. Attitudes towards EA. There is no right or wrong answers. Our philosophical positions are very diverse, and even include nietzschean philosophy.

• Do you know what Effective Altruism is
  • Yes / No but I've heard of it / No.✓
• Do you self-identify as an Effective Altruist?✓
• Has Effective Altruism caused you to make donations you otherwise wouldn't?✓
• Do you expect Effective Altruism to cause you to make donations in the future which you otherwise won't?✓
• If that is the case, what % of your earnings do you expect to donate to ea charities (like Against Malaria Foundation, Malaria Consortium, Schistosomiasis Control Initiative, Evidence Action's Deworm the World Initiative, GiveWell, 80.000 hours, etc) over your life?
  • Ask this privately
• What's your overall opinion of Effective Altruism? ✓
• If you had to distribute 1 billion dollars to different charities, on the basis of which criteria would you do it?

1. Attitudes towards existential risk.

• Are you familiar with the term "existential risk"?
• Without searching the internet, looking at Wikipedia, etc., how would you describe the concept in a short tweet?
• If you had heard about it, how much of a threat do you think it poses?
• In percentage points, how likely do you judge it that your career will in some way be related to existential risk? And directly related?

1. Attitudes towards AI Safety

• Are you familiar with the field of AI Safety?
• Have you read any papers related to the field?
• If you knew about it beforehand, how much of a threat do you think it poses?
• How would you describe the concept in a short tweet? ✓
• In percentage points, how likely do you judge it that your career will in some way be related to AI Safety? And directly related? ✓

Sofware upgrade

1. Introspective power. Internal Design. Habits.

• I undestand myself ✓
• I have fiddled with the different parts of myself.
• I work to change the parts of myself which I don't like. ✓
• I purposefully create habits. ✓
• When was the last time you did this? ✓

1. Position towards emotions.

• Emotions as your allies. ✓
• To what extent do you agree with the following:
• Emotions are my allies, ✓
• Emotions often give me useful information. ✓
• Emotions often hinder me, ✓
• I would prefer to feel less. ✓
• I often ignore my emotions. ✓
• I am in touch with my emotions. ✓

1. Life optimization

• I have in place mechanisms for constant, iterated improvement of my life.✓
• Write a short tweet about it.
• Units of exchange: I often explicitly consider the tradeoffs between money, time, prestige, etc., when making decisions.
• When was the last time you've done that (if ever)
• Write a short tweet about it.
• Think about your current set of skills, your habits, the things you spend your time on, how you interact with other people, the intellectual questions that you find engaging, the goals you’re aiming towards, and the challenges that you’re currently facing going forward. Next, think about how you were one year ago on each of these dimensions. How different are you now from how you were one year ago?
• Not at all different / Slightly different / Somewhat different / Very different / Extremely different. ✓
• [optional] In about one tweet, what is one difference that stands out as being particularly large or significant? ✓
• Can you think of any changes that you’ve made in the past month to your daily routines or habits in order to make things go better? These can be tiny changes (e.g., adjusted the curtains on my bedroom window so that less light comes in while I’m sleeping) or large ones. Spend about 60 seconds recalling as many examples of these kinds of changes as you can and listing them here. (If you want to skip this question, leave it blank. If you spend the 60 seconds and no specific examples come to mind, write "none.") ✓

1. Mental illness. I actually don't care about the "Post-espr depression". While having a mental illness sucks, there is no right or wrong answer. Some of the best people I know face depression, aspergers, etc.

• Have you been diagnosed with a mental illness?
• Do you think you have a mental illness?
• If so, which?

1. Goal clarity With regards to my goals,

• I know what my goals are. ✓
• I feel that different parts of myself are aligned. ✓
• I feel that the different parts of myself are more aligned than 1y ago. ✓
• when an internal conflict arises, I have adequate tools to resolve it. ✓ |N: I copied the first person from somewhere else.

1. Communication

• I can nonviolently communicate with the people I care about. Too abstract.
• When I talk to people, they perceive that I'm speaking in good faith. ✓
• I successfully assert my needs to others. ✓
• The last time I had a discussion, it was resolved gracefully. ✓
• When I debate with people, there is often a satisfying conclusion.

1. Stupid mistakes.

• How often do you make stupid mistakes?
• When was the last stupid mistake you made? ✓
• Write a short tweet about it.
• Did you implement any measures to avoid making that specific stupid mistake in the future? ✓
• If so, write a short tweet about it. ✓

1. Life satisfaction

• How satisfied are you with your life as a whole?
• To what extent do you agree with the statement: I am winning at life? ✓
• Stuckness: I feel like my life is stuck

1. Effective Approaches to Working on Projects When I decide that I want to do something (like doing a project, developing a new regular practice, or changing some part of my lifestyle), I … ✓

• plan out what specific tasks I will need to do to accomplish it.
• try to think in advance about what obstacles I might face, and how I can get past them.
• seek out information about other people who have attempted similar projects to learn about what they did.
• end up getting it done. The four items were averaged into a single measure of effective approaches to projects.

1. Probabilities / Calibration.

• Are you comfortable using probabilities? Do you use them in your daily life / When was the last time you explicitly assigned a probability to something?

• To what extent do you agree with the following: Thinking in terms of probabilities is a valuable tool in my skill repertoire.

• When was the last time you explicitly assigned a probability to something? Write a short tweet about it.

• "Calibration" is the practice of knowing how certain you are, even when you're not certain. For example, a bookie who says they're 90% certain of the outcome of each of a hundred horse races, and who is right about ninety out of those hundred horse races - is perfectly calibrated.

In these questions, you will be asked a question and then asked to give a calibration percent. The percent represents your probability that the answer is right. Suppose the question is "What country is the city of Paris located in?" and you are absolutely sure it is France. In that case, your calibration percent is 100 - you are 100% sure it is France.

But suppose you think there's a fifty-fifty chance it's either France or Germany. In that case, you might still answer France, but your calibration percent is only 50 - you are only 50% sure it's France.

Or suppose you have no idea, so you pick a country totally at random. In that case, you might think that if there are about one hundred possible countries, and it could be any of them, there's only about a 1% chance you're right. Therefore, you would put down a calibration percent of 1. Please answer on a scale from 0% (definitely false) to 100% (definitely true)

• Are you smiling right now?
• After each question: Without checking a source, estimate your subjective probability that the answer you just gave is correct.-
• Which is heavier, a virus or a prion?
• I'm thinking of a number between one and ten, what is it?
• What year was the fast food chain "Dairy Queen" founded? (Within five years)
• Alexander Hamilton appears on how many distinct denominations of US Currency?
• Without counting, how many keys on a standard IBM keyboard released after 1986, within ten?
  • Too easy to cheat.
• What's the diameter of a standard soccerball, in cm within 2?
• How many calories in a reese's peanut butter cup within 20?
• What is the probability that supernatural events (including God, ghosts, magic, etc) have occurred since the beginning of the universe?
• What is the probability that there is a god, defined as a supernatural intelligent entity who created the universe?
• What is the probability that any of humankind's revealed religions is more or less correct?

Attribution

I cribbed some questions from the 2016 LessWrong Diaspora Survey and the CFAR 2015 longitudinal study

CFAR's rationality checklist http://www.rationality.org/resources/rationality-checklist