From 1e9f35fda09da9642d6d1d3360d9716fedde9b8f Mon Sep 17 00:00:00 2001 From: Nuno Sempere Date: Sun, 18 Sep 2022 11:16:24 +0000 Subject: [PATCH] feat: big update --- .blog/_werc/config | 2 + _werc/lib/footer.inc | 11 ++ _werc/lib/{top_bar.inc => top_bar_old.inc} | 0 index.md | 3 +- media-mentions/index.md | 23 ++++ projects/index.md | 19 +++ sitemap.gz | Bin 168 -> 191 bytes sitemap.txt | 4 +- track-record/index.md | 129 +++++++++++++++++++++ 9 files changed, 189 insertions(+), 2 deletions(-) create mode 100755 .blog/_werc/config create mode 100755 _werc/lib/footer.inc rename _werc/lib/{top_bar.inc => top_bar_old.inc} (100%) create mode 100644 media-mentions/index.md create mode 100644 projects/index.md mode change 100755 => 100644 sitemap.txt create mode 100644 track-record/index.md diff --git a/.blog/_werc/config b/.blog/_werc/config new file mode 100755 index 0000000..2aab8e5 --- /dev/null +++ b/.blog/_werc/config @@ -0,0 +1,2 @@ +# conf_enable_comments -n +conf_enable_blog diff --git a/_werc/lib/footer.inc b/_werc/lib/footer.inc new file mode 100755 index 0000000..d968723 --- /dev/null +++ b/_werc/lib/footer.inc @@ -0,0 +1,11 @@ +
+ +
Powered by werc, Alpine Linux and nginx
+ + diff --git a/_werc/lib/top_bar.inc b/_werc/lib/top_bar_old.inc similarity index 100% rename from _werc/lib/top_bar.inc rename to _werc/lib/top_bar_old.inc diff --git a/index.md b/index.md index 2f8c764..1f37ac3 100644 --- a/index.md +++ b/index.md @@ -10,4 +10,5 @@ Scott Alexander [describes us as](https://astralcodexten.substack.com/p/mantic-m > Enter Samotsvety Forecasts. This is a team of some of the best superforecasters in the world. They won the CSET-Foretell forecasting competition by an absolutely obscene margin, “around twice as good as the next-best team in terms of the relative Brier score”. If the point of forecasting tournaments is to figure out who you can trust, the science has spoken, and the answer is “these guys”. -We are open to forecasting consulting requests, and can be reached out at [info@samotsvety.org](mailto:info@samotsvety.org). +We are open to forecasting consulting requests, and can be reached out at [info@samotsvety.org](mailto:info@samotsvety.org). Readers might also want to view our [track record](./track-record), browse our [projects](./projects) or read our [media mentions](./media-mentions). + diff --git a/media-mentions/index.md b/media-mentions/index.md new file mode 100644 index 0000000..2cc0122 --- /dev/null +++ b/media-mentions/index.md @@ -0,0 +1,23 @@ +## Media mentions + +N.B.: We are open to media mentions or collaborations, and can be reached at [info@samotsvety.org](mailto:info@samotsvety.org). + +[CNN: How to assess the risk of nuclear war without freaking out](https://edition.cnn.com/2022/06/28/opinions/nuclear-war-likelihood-probability-russia-us-scoblic-mandel/index.html) ([a](https://web.archive.org/web/20220628081603/https://edition.cnn.com/2022/06/28/opinions/nuclear-war-likelihood-probability-russia-us-scoblic-mandel/index.html)): + +> One group of [highly regarded forecasters](https://forum.effectivealtruism.org/posts/KRFXjCqqfGQAYirm5/samotsvety-nuclear-risk-forecasts-march-2022#_blank) put the probability of Russia using a nuclear weapon against London before February 2023 at 0.8%[^1] + +[El País: Is it possible to predict the future of the war in Ukraine? Online forecasting communities think so](Is it possible to predict the future of the war in Ukraine? Online forecasting communities think so) ([a](https://web.archive.org/web/20220401061002/https://english.elpais.com/science-tech/2022-03-26/is-it-possible-to-predict-the-future-of-the-war-in-ukraine-online-forecasting-communities-think-so.html), [es](https://elpais.com/tecnologia/2022-03-24/se-puede-predecir-el-futuro-de-la-guerra-en-ucrania-es-lo-que-intenta-una-comunidad-de-pronosticadores-en-internet.html)): + +> What risk is there for an alert citizen in London of dying next month due to a nuclear explosion? Some 24 micro deaths, 24 options in a million, according to the group of renowned forecasters of which Nuño Sempere, from Madrid, is a part. + +[Astral Codex Ten: Mantic Monday](https://astralcodexten.substack.com/p/mantic-monday-31422) ([a](https://web.archive.org/web/20220401013834/https://astralcodexten.substack.com/p/mantic-monday-31422)): + +> Enter Samotsvety Forecasts. This is a team of some of the best superforecasters in the world. They won the CSET-Foretell forecasting competition by [an absolutely obscene margin](https://www.lesswrong.com/posts/EGHtomuh55375u7RT/forecasting-newsletter-march-2021), “around twice as good as the next-best team in terms of the relative Brier score”. If the point of forecasting tournaments is to figure out who you can trust, the science has spoken, and the answer is “these guys”. + +[INFER: The Making of a Top Forecaster: Techniques to Boost Accuracy](https://www.infer-pub.com/blog/top-forecaster-techniques) ([a](https://web.archive.org/web/20220221032529/https://www.infer-pub.com/blog/top-forecaster-techniques)): + +> The Good Judgment Project’s research also showed the value of working in teams. Teams who debated their forecasts (usually virtually) performed better than individuals working alone. Perhaps that’s yet another technique benefitting top forecaster Sempere (@Loki), who meets regularly with his two teammates, @yagudin (also in the top 1%) and @elifland, to discuss and evaluate questions from different angles. + +
+ +[^1]: We gave a 0.8% _annualized_ probability to a Russia/NATO nuclear exchange killing at least one person in the first month of the conflict. diff --git a/projects/index.md b/projects/index.md new file mode 100644 index 0000000..603700a --- /dev/null +++ b/projects/index.md @@ -0,0 +1,19 @@ +## Done as _Samotsvety_: + +- [Samotsvety Nuclear Risk Forecasts — March 2022](https://forum.effectivealtruism.org/posts/KRFXjCqqfGQAYirm5/samotsvety-nuclear-risk-forecasts-march-2022) +- [Samotsvety's AI risk forecasts](https://forum.effectivealtruism.org/posts/EG9xDM8YRz4JN4wMN/samotsvety-s-ai-risk-forecasts) +- [Prediction Markets in the Corporate Setting](https://forum.effectivealtruism.org/posts/dQhjwHA7LhfE8YpYF/prediction-markets-in-the-corporate-setting) +- [GJO Calibration app](https://share.streamlit.io/yagudin/gjo-calibration/main/strmlt.py) +- [Micro-grants for forecasting](https://forum.effectivealtruism.org/posts/oqFa8obfyEmvD79Jn/we-are-giving-usd10k-as-forecasting-micro-grants) +- [Bottlenecks to more impactful crowd forecasting](https://forum.effectivealtruism.org/posts/S2vfrZsFHn7Wy4ocm/bottlenecks-to-more-impactful-crowd-forecasting-2) + +## With other hats + +- [Forecasting Newsletter](https://forecasting.substack.com/), by Nuño +- [Metaforecast](https://metaforecast.org/), by [QURI](https://quantifieduncertainty.org/) +- [Squiggle](https://www.squiggle-language.com/), by [QURI](https://quantifieduncertainty.org/) +- [Impactful Forecasting Prize](https://forum.effectivealtruism.org/posts/HDoMrQFG76QtkdrZJ/impactful-forecasting-prize-for-forecast-writeups-on-curated), by [Sage](https://forum.effectivealtruism.org/users/sage) +- [Pastcasting](http://pastcasting.com/), by [Sage](https://forum.effectivealtruism.org/users/sage) +- [Better scoring rules](https://github.com/SamotsvetyForecasting/optimal-scoring), by Nuño +- [Alignment Problems With Current Forecasting Platforms](https://arxiv.org/abs/2106.11248), by Nuño and Alex Lawsen +- [Comparing top forecasters and domain experts](https://forum.effectivealtruism.org/posts/qZqvBLvR5hX9sEkjR/comparing-top-forecasters-and-domain-experts), by [Arb](https://arbresearch.com/) diff --git a/sitemap.gz b/sitemap.gz index 914838925e57a20bae44493d36142b0975079110..cc3c8d227712146acda46083335becc4d44a9c13 100755 GIT binary patch literal 191 zcmV;w06_mAiwFQxq9tPh1Feuv4uUWcM)#b8bYBM8D5jL1C(r}b(g3kPq%(rv-lFb} ziOc!sdoP(0-QC%IF%hTWTUHe%%Z=BepZw6W^XbU^=RQA}ik!U^utm|6f zM8>HZix3AuJ*ENS6GB-u>+5bcp>gUU>gcW>5bvCy;Of008DHUCaOg literal 168 zcmV;Z09XGXiwFQ~Wh`O<11*k84udcZMfW)c%APn33yMP6^#tt!5JZ5K0bv)~+c#-h zef{BCR{h)A>@Yj7!FRk6l4r(`VOss%@#o{7H@w$Oy>^R+vN-nG@r7t>0nYOjxRP +![](https://i.imgur.com/4hayqba.png) +
+
Note that we are a bit more than twice as good as the next best team, in terms of the relative Brier score
+ + +As of 2022-09-15, we are also 4th on the nascent [Insight Prediction leaderboard](https://insightprediction.com/leaderboards) as a result of one (1) large bet when we correctly foresaw the Russian invasion of Ukraine and put our money where our mouth was: + +![](https://i.imgur.com/ob5Cla4.png) + +## Individuals + +Individually, Samotsvety members are, as of 2022-09-15, occupying all top 4 spots on INFER's [overall leaderboard](https://www.infer-pub.com/leaderboards/overall): + +![](https://i.imgur.com/ygMxoht.png) + +All scores mentioned below are as of 2022-09-15: + +#### Misha Yagudin +- Runs a research consultancy [Arb Research](https://arbresearch.com/) +- [GJOpen](https://www.gjopen.com/forecaster/yagudin) Brier score 0.191 vs. 0.283 median, ratio 0.67 +- 7th and 3rd place forecaster in the first two seasons of CSET-Foretell/INFER ([I](https://www.infer-pub.com/leaderboards/seasons/1), [II](https://www.infer-pub.com/leaderboards/seasons/2)); [3rd](https://www.infer-pub.com/forecaster/yagudin/performance?alltime=true) all-time (as of 2022/9/15). + +#### Nuño Sempere + +- Researcher at the [Quantified Uncertainty Research Institute](https://quantifieduncertainty.org/) +- [GJOpen](https://www.gjopen.com/forecaster/LokiOdinevich) Brier score 0.209 vs. 0.294 median, ratio 0.71 +- Top 5 forecaster in the first two seasons of CSET-Foretell/INFER ([I](https://www.infer-pub.com/leaderboards/seasons/1),[II](https://www.infer-pub.com/leaderboards/seasons/2)); [4th](https://www.infer-pub.com/forecaster/loki/performance?alltime=true) all-time (as of 2022/9/15) + +#### Alex Lyzhov + +- ML PhD student at NYU +- Participates primarily in AI-related forecasts due to expertise +- Active Metaculus [contributor](https://www.metaculus.com/accounts/profile/103447/) + +#### Eli Lifland +- Figuring out what's up with AI alignment (and sometimes other stuff), writing [Foxy Scout](https://www.foxy-scout.com/). +- [GJOpen](https://www.gjopen.com/forecaster/elifland) Brier score 0.23 vs. 0.301 median, ratio 0.76 +- [1st](https://www.infer-pub.com/forecaster/elifland/performance?alltime=true) forecaster of all time on CSET-Foretell/INFER (as of 2022/9/15) +- 2nd in Metaculus [Economist 2021 tournament](https://www.metaculus.com/questions/11438/economist-2021-tournament-winners/), 1st in [Salk Tournament](https://pandemic.metaculus.com/contests/?selected=salk-tournament) (as of 2022/9/10). +- Track record is described in detail [here](https://www.lesswrong.com/posts/XAkhqkNQByEaT8MED/personal-forecasting-retrospective-2020-2022) + +#### Jonathan Mann + +- Professional in the financial sector +- [GJOpen](https://www.gjopen.com/forecaster/JonathanMann) Brier score 0.155 vs. 0.247 median, ratio 0.63 +- Pro forecaster on [INFER](https://www.infer-pub.com/forecaster/JonathanMann/performance) +- [About + Github](https://jonathanmann.github.io/about/) + +#### Juan Cambeiro + +- MPH student in epidemiology at Columbia University +- Biosecurity fellow at the Institute for Progress +- [Good Judgment](https://www.gjopen.com/) Superforecaster +- Analyst for [Metaculus](https://www.metaculus.com/questions/) +- [GJOpen](https://www.gjopen.com/forecaster/juancambeiro) Brier score 0.25 vs. 0.317 median, ratio 0.79 +- First in [IARPA's FOCUS](https://www.iarpa.gov/research-programs/focus-2) Tournament on COVID-19 forecasting +- First in [Good Judgment Open's US Election 2020 challenge](https://www.gjopen.com/challenges/42-us-election-2020/scores) +- Second in [Good Judgment Open's In the News 2020 challenge](https://www.gjopen.com/challenges/36-in-the-news-2020/scores) +- First in the time active on [Good Judgment Open's Coronavirus Outbreak challenge](https://www.gjopen.com/challenges/43-coronavirus-outbreak/scores), now third + +#### Sam Glover + +- Writes about social science at [samstack.io](https://www.samstack.io) +- [Good Judgment](https://www.gjopen.com/) Superforecaster +- GJOpen [Brier score](https://www.gjopen.com/forecaster/samglover97) 0.228vs 0.309 median, ratio 0.74, increased significantly since becoming inactive in late 2021 + +#### Molly Hickman + +- Data scientist at [nLine, Inc.](https://nline.io/), and previously worked on crowd-sourced prediction projects on the evaluation side. +- [Top 2](https://www.infer-pub.com/forecaster/mollygh) forecaster of all time on [INFER](https://www.infer-pub.com/) (as of 2022/9/15); [2nd place forecaster in season 2021](https://www.infer-pub.com/leaderboards/seasons/2) (-1.684 relative Brier score); [8th place forecaster in season 2020](https://www.infer-pub.com/leaderboards/seasons/1) (-0.529 relative Brier score) + +#### Gregory Lewis + +- Ex public health doctor, now working in biosecurity +- [Good Judgment](https://www.gjopen.com/) Superforecaster (since 2021) +- [GJOpen](https://www.gjopen.com/forecaster/Gjlewis) Brier score 0.226 vs. 0.313 median, ratio 0.72 + + +#### Pablo Stafforini + +- Philosopher by training, currently editor of the *Global Priorities Encyclopedia*. +- 1st place in Road to Recovery tournament +- 3rd place in Li Wenliang Forecasting Tournament I tournament +- 1st place in Li Wenliang Forecasting Tournament II tournament +- See [my Metaculus profile](https://www.metaculus.com/accounts/profile/101341/) for additional information. + +#### Jared Leibowich + +- Data Scientist +- Forecaster for the [Swift Centre](https://www.swiftcentre.org/) +- [GJO](https://www.gjopen.com/forecaster/jleibowich) Brier score 0.22 vs. 0.303 median, ratio 0.73 +- Placed 1st out of 7,000 ["In the News 2021" Good Judgment competition](https://www.gjopen.com/leaderboards/challenges/47-in-the-news-2021) +- Currently ranked 1st out of 3,000 ["In the News 2022" Good Judgment competition](https://www.gjopen.com/leaderboards/challenges/57-in-the-news-2022) + +#### Greg Justice + +- Analyst in the healthcare space +- [GJOpen](https://www.gjopen.com/forecaster/slapthepancake) Brier score 0.187 vs. 0.251 median, ratio 0.75 +- 1st place in [Adam Grant's Think Again challenge on GJ Open](https://adamgrant.bulletin.com/how-to-be-less-wrong) + (as of 2022-09-11) +- 4th place - Olivier Sibony's Noise challenge on GJ Open (as of 2022-09-11) +- 7th place - Good Judgment Project 2.0, in initial accuracy (as of 2022-09-11) + +#### @belikewater + +- MA in Economics, PhD in Neuroscience +- [GJOpen](https://www.gjopen.com/forecaster/belikewater) Brier score 0.215 vs. 0.279 median, ratio 0.77 +- Recently, also forecasting at INFER, Metaculus, Manifold Markets + +#### Tolga Bilge + +- Mathematics student +- GJI Superforecaster +- Forecaster for the [Swift Centre](https://swiftcentre.org) +- [GJOpen](https://www.gjopen.com/forecaster/Tolga) Brier score 0.164 vs. 0.254 median, ratio 0.65 +- Forecaster on [INFER](https://www.infer-pub.com/forecaster/Tolga) + +[^1]: Says Pablo.