From 1b2a1a8608d967bd8dac1ee0b47c40d0a8914427 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nu=C3=B1o=20Sempere?= Date: Sat, 18 May 2019 13:31:07 +0200 Subject: [PATCH] Update p-hacking.md --- rat/eamentalhealth/p-hacking.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rat/eamentalhealth/p-hacking.md b/rat/eamentalhealth/p-hacking.md index 6f6a820..1389399 100644 --- a/rat/eamentalhealth/p-hacking.md +++ b/rat/eamentalhealth/p-hacking.md @@ -23,8 +23,8 @@ Thus, we can regress the first four variables on the fifth and sixth. ## Results If we choose only one among the 8 comparisons, the results are *not* whatever we want them to be, because the data is extremely suggestive of one interpretation. But we can massage them, concluding either: -a) If we only report B ~ X (the only regression which did not reach significance), we find no significant effect, smallish effect which could be due to chance, because p>0.1. -b) If we only report A ~ Y, we find a huge effect; whereas male or female EAs have a mean of 0.76 mental ilnesses, gender nonconforming EAs have a mean of 1.6 mental ilnesses, p < 0.001. If we bother to [calculate the exact p-value](https://www.wolframalpha.com/input/?i=N(mean%3D0,+standard+deviation+%3D+0.23271)+>+0.84798), it's [~0.0003649317](https://www.wolframalpha.com/input/?i=((2476491678888003+e)%2F18446744073709551616)). Additionally, "the most conservative method, which is free of dependence and distributional assumptions, is the Bonferroni correction" [Wikipedia](https://en.wikipedia.org/wiki/Multiple_comparisons_problem). If we harshly apply it to correct for having tested 8 hypothesis, we get p = 0.0003649317\*8 = 0.002919452 ~ 0.003, which is still ridiculously low. +- If we only report B ~ X (the only regression which did not reach significance), we find no significant effect, smallish effect which could be due to chance, because p>0.1. +- If we only report A ~ Y, we find a huge effect; whereas male or female EAs have a mean of 0.76 mental ilnesses, gender nonconforming EAs have a mean of 1.6 mental ilnesses, p < 0.001. If we bother to [calculate the exact p-value](https://www.wolframalpha.com/input/?i=N(mean%3D0,+standard+deviation+%3D+0.23271)+>+0.84798), it's [~0.0003649317](https://www.wolframalpha.com/input/?i=((2476491678888003+e)%2F18446744073709551616)). Additionally, "the most conservative method, which is free of dependence and distributional assumptions, is the Bonferroni correction". [Wikipedia dixit](https://en.wikipedia.org/wiki/Multiple_comparisons_problem). If we harshly apply it to correct for having tested 8 hypothesis, we get p = 0.0003649317\*8 = 0.002919452 ~ 0.003, which is still ridiculously low. ## A note on regressions and frequentist probability. If you have 303 values for the variable A: {A1, A2, A3, ..., A303} and 303 values for the variable B. {B1, B2, B3, ..., B303}, you consider lines of the form A = I + C\*B, and look at their associated points {(I+C\*B1,B1),(I+C\*B2,B2), (I+C\*B3,B3), ..., (I+C\*B303,B303)}. They are separated from the points {(A1,B1),(A2,B2),(A3,B3),..., (A303,B303)} by whatever distance.