personal/nunosempere.github.io
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Forecasting Newsletter for June - Draft

Browse Source
This commit is contained in:
Nuno Sempere 2020-06-28 20:06:18 +02:00
parent 8408e5be4c
commit 75b994926f
1 changed files with 13 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
ea/ForecastingNewsletter/June2020.md
View File

@ -12,24 +12,27 @@ Whatever happened to forecasting? June 2020
Ordered in subjective order of importance: Ordered in subjective order of importance:
- Foretell, a forecasting tournament by the Center for Security and Emerging Technology, is now [open](www.cset-foretell.com). I find the thought heartening that this might end up influencing actual politicians. - Foretell, a forecasting tournament by the Center for Security and Emerging Technology, is now [open](www.cset-foretell.com). I find the thought heartening that this might end up influencing bona-fide politicians.
- Metaculus - Metaculus
- posted [A Preliminary Look at Metaculus and Expert Forecasts](https://www.metaculus.com/news/2020/06/02/LRT/): Metaculus forecasters do better, and the piece is a nice reference point. - posted [A Preliminary Look at Metaculus and Expert Forecasts](https://www.metaculus.com/news/2020/06/02/LRT/): Metaculus forecasters do better, and the piece is a nice reference point.
- was featured in [Forbes](https://www.forbes.com/sites/erikbirkeneder/2020/06/01/do-crowdsourced-predictions-show-the-wisdom-of-humans/#743b7e106d9d). - was featured in [Forbes](https://www.forbes.com/sites/erikbirkeneder/2020/06/01/do-crowdsourced-predictions-show-the-wisdom-of-humans/#743b7e106d9d).
- anounced their [Metaculus Summer Academy](https://www.metaculus.com/questions/4566/announcing-a-metaculus-academy-summer-series-for-new-forecasters/): "an introduction to forecasting for those who are relatively new to the activity and are looking for a fresh intellectual pursuit this summer" - anounced their [Metaculus Summer Academy](https://www.metaculus.com/questions/4566/announcing-a-metaculus-academy-summer-series-for-new-forecasters/): "an introduction to forecasting for those who are relatively new to the activity and are looking for a fresh intellectual pursuit this summer"
- [Replication Markets](https://predict.replicationmarkets.com/) might add a new round with social and behavioral science claims related to COVID-19, and a preprint market, which would ask participants to forecast items like publication or citation. Replication Markets is also asking for more participants, with the catchline "If they are knowledgeable and opinionated, Replication Markets is the place to be to make your opinions really count." - [Replication Markets](https://predict.replicationmarkets.com/) might add a new round with social and behavioral science claims related to COVID-19, and a preprint market, which would ask participants to forecast items like publication or citation. Replication Markets is also asking for more participants, with the catchline "If they are knowledgeable and opinionated, Replication Markets is the place to be to make your opinions really count."
- Good Judgement family - Good Judgement family
- [Good Judgement Open](https://www.gjopen.com/): Superforecasters were able to detect that Russia and the USA would in fact undertake some (albeit limited) form of negotiation, and do so much earlier than the general public, even while posting their reasons in full view. One thread to follow is [this one](https://www.gjopen.com/comments/1039968). - [Good Judgement Open](https://www.gjopen.com/): Superforecasters were able to detect that Russia and the USA would in fact undertake some (albeit limited) form of negotiation, and do so much earlier than the general public, even while posting their reasons in full view. One thread to follow is [this one](https://www.gjopen.com/comments/1039968).
- Good Judgement Analytics continues to provide their [covid dashboard](https://goodjudgment.com/covidrecovery/). - Good Judgement Analytics continues to provide their [covid dashboard](https://goodjudgment.com/covidrecovery/).
- [PredictIt](https://www.predictit.org/) & [Election Betting Odds](http://electionbettingodds.com/). I stumbled upon an old [538 piece](https://fivethirtyeight.com/features/fake-polls-are-a-real-problem/) on fake polls: some may have been conducted by PredictIt traders in order to mislead or troll other PredictIt traders. - [PredictIt](https://www.predictit.org/) & [Election Betting Odds](http://electionbettingodds.com/). I stumbled upon an old [538 piece](https://fivethirtyeight.com/features/fake-polls-are-a-real-problem/) on fake polls: some may have been conducted by PredictIt traders in order to mislead or troll other PredictIt traders.
- Augur: - Augur:
- [An overview of the platform and of v2 modifications](https://bravenewcoin.com/insights/augur-price-analysis-v2-release-scheuled-for-june-12th). - [An overview of the platform and of v2 modifications](https://bravenewcoin.com/insights/augur-price-analysis-v2-release-scheuled-for-june-12th).
- Augur also happens to have a [blog](https://augur.substack.com/archive) with some interesting tidbits, such as the extremely clickbaity [How One Trader Turned $400 into $400k with Political Futures](https://augur.substack.com/p/how-one-trader-turned-400-into-400k) ("I find high volume markets...like the Democratic Nominee market or the 2020 Presidential Winner market... and what I’m doing is I’m just getting in line at the ‘buy’ price and waiting my turn until my orders get filled. Then when those orders get filled I just sell them for 1c more.") - Augur also happens to have a [blog](https://augur.substack.com/archive) with some interesting tidbits, such as the extremely clickbaity [How One Trader Turned $400 into $400k with Political Futures](https://augur.substack.com/p/how-one-trader-turned-400-into-400k) ("I find high volume markets...like the Democratic Nominee market or the 2020 Presidential Winner market... and what I’m doing is I’m just getting in line at the ‘buy’ price and waiting my turn until my orders get filled. Then when those orders get filled I just sell them for 1c more.")
- [Coronavirus Information Markets](https://coronainformationmarkets.com/) is down to ca. $12000 in trading volume; it seems like they didn't take off. - [Coronavirus Information Markets](https://coronainformationmarkets.com/) is down to ca. $12000 in trading volume; it seems like they didn't take off.
@ -62,6 +65,7 @@ Ordered in subjective order of importance:
## Negative examples. ## Negative examples.
- World powers to converge on strategies for presenting COVID-19 information to make forecasters' jobs more interesting: - World powers to converge on strategies for presenting COVID-19 information to make forecasters' jobs more interesting:
- [Brazil stops releasing Covid-19 death toll and wipes data from official site](https://www.theguardian.com/world/2020/jun/07/brazil-stops-releasing-covid-19-death-toll-and-wipes-data-from-official-site). - [Brazil stops releasing Covid-19 death toll and wipes data from official site](https://www.theguardian.com/world/2020/jun/07/brazil-stops-releasing-covid-19-death-toll-and-wipes-data-from-official-site).
- Meanwhile in Russia, [St Petersburg issues 1,552 more death certificates in May than last year, but Covid-19 toll was 171](https://www.theguardian.com/world/2020/jun/04/st-petersburg-death-tally-casts-doubt-on-russian-coronavirus-figures). - Meanwhile in Russia, [St Petersburg issues 1,552 more death certificates in May than last year, but Covid-19 toll was 171](https://www.theguardian.com/world/2020/jun/04/st-petersburg-death-tally-casts-doubt-on-russian-coronavirus-figures).
@ -110,7 +114,10 @@ Ordered in subjective order of importance:
> On the other hand, a well-calibrated mechanistic model that accounts for all relevant dynamic factors and external influences could, in principle, have been used to predict the behaviour of the epidemic reliably and precisely. Yet, lack of detailed data on transmission routes and risk factors precluded the parameterisation of such a model and are likely to do so again in future epidemics in resource-poor settings. > On the other hand, a well-calibrated mechanistic model that accounts for all relevant dynamic factors and external influences could, in principle, have been used to predict the behaviour of the epidemic reliably and precisely. Yet, lack of detailed data on transmission routes and risk factors precluded the parameterisation of such a model and are likely to do so again in future epidemics in resource-poor settings.
- [Calibration Scoring Rules for Practical Prediction Training](https://arxiv.org/abs/1808.07501). I found it most interesting when considering how Brier and log rules didn't have all the pedagogic desiderata. - [Calibration Scoring Rules for Practical Prediction Training](https://arxiv.org/abs/1808.07501). I found it most interesting when considering how Brier and log rules didn't have all the pedagogic desiderata.
- I also found the following derivation of the logarithmic scoring rule interesting. Consider: If you assign a probability to n events, then the combined probability of these events is p1 x p2 x p3 x ... pn. Taking logarithms, this is log(p1 x p2 x p3 x ... x pn) = Σ log(pn), i.e., the logarithmic scoring rule.
- [Binary Scoring Rules that Incentivize Precision](https://arxiv.org/abs/2002.10669). The results (the closed-form of scoring rules which minimize the a given forecasting error) are interesting, but the journey to get there is kind of a drag, and ultimately the logarithmic scoring rule ends up being pretty decent according to their measure of error. - I also found the following derivation of the logarithmic scoring rule interesting. Consider: If you assign a probability to n events, then the combined probability of these events is p1 x p2 x p3 x ... pn. Taking logarithms, this is log(p1 x p2 x p3 x ... x pn) = Σ log(pn), i.e., the logarithmic scoring rule.
- [Binary Scoring Rules that Incentivize Precision](https://arxiv.org/abs/2002.10669). The results (the closed-form of scoring rules which minimize the a given forecasting error) are interesting, but the journey to get there is kind of a drag, and ultimately the logarithmic scoring rule ends up being pretty decent according to their measure of error.
- Opinion: I'm not sure whether their results are going to be useful for things I'm interested in (like human forecasting tournaments, rather than kaggle data analysis competitions). In practice, what I might do if I wanted to incentivize precision is to ask myself if this is a question where the answer is going to be closer to 50%, or closer to either of 0% or 100%, and then use either the Brier or the logarithmic scoring rules. That is, I don't want to minimize an l-norm of the error over [0,1], I want to minimize an l-norm over the region I think the answer is going to be in, and the paper falls short of addressing that. - Opinion: I'm not sure whether their results are going to be useful for things I'm interested in (like human forecasting tournaments, rather than kaggle data analysis competitions). In practice, what I might do if I wanted to incentivize precision is to ask myself if this is a question where the answer is going to be closer to 50%, or closer to either of 0% or 100%, and then use either the Brier or the logarithmic scoring rules. That is, I don't want to minimize an l-norm of the error over [0,1], I want to minimize an l-norm over the region I think the answer is going to be in, and the paper falls short of addressing that.