Categories:
Economic Methods
I just learned that blogger Scott Alexander starts ends the year by scoring the accuracy of his numerous annual predictions: here are the results for 2014, 2015, and 2016. Since he assigns probabilities to each prediction, Scott can graph his ex ante subjective probabilities against the ex post objective probabilities. Here’s his graph for 2016. The red line shows perfect calibration; the blue line shows Scott’s actual calibration.
Contrary to Tyler, this is not “just another kind of religion,” but this intensity of scrupulous rationality does approach the divine. Or as a wise man once said, “People who say it cannot be done shouldn’t interrupt those who are doing it.”
READER COMMENTS
richard
Apr 6 2017 at 3:09pm
It looks like he’s just learning to make fewer predictions, which might be a kind of wisdom
“Of 50% predictions, I got 8 right and 5 wrong, for a score of 62%
Of 60% predictions, I got 12 right and 9 wrong, for a score of 57%
Of 70% predictions, I got 13 right and 3 wrong, for a score of 81%
Of 80% predictions, I got 13 right and 3 wrong, for a score of 81%
Of 90% predictions, I got 16 right and 1 wrong, for a score of 94%
For 95% predictions, I got 9 right and 0 wrong, for a score of 100%
For 99% predictions, I got 3 right and 0 wrong, for a score of 100%”
brokilo
Apr 6 2017 at 3:40pm
@richard
2014: 59 predictions
2015: 35 predictions
2016: 99 predictions
You mean less >90% confident predictions?
robc
Apr 6 2017 at 3:42pm
It looks like he is underestimating the probabilities.
With the exception of the 60% category, he overshot his category mark each time.
Which means he is either bad at predicting probabilities or he was sandbagging. Or the sample size was just small.
brokilo
Apr 6 2017 at 3:51pm
@robc
He’s not perfect at predicting probabilities. The whole point is that he calibrates himself as time goes on.
He reviews how much he over/undershot his probabilities and factors that into next years predictions. Looks like he needs to mark his confidence up by about 5-10 percent across the board this year.
I doubt I could do much better if I measured myself in this fashion.
Garrett M. Petersen
Apr 6 2017 at 4:00pm
I made a similar set of predictions for 2016. Turns out I was pretty overconfident.
Michael Keenan
Apr 6 2017 at 4:31pm
Richard, I think you might have misinterpreted it; he’s increased the number of predictions he made over time, not decreased.
He made 59 predictions in 2014, 55 in 2015, 99 in 2016, and (though Bryan didn’t link to it), 105 in 2017.
Ryan Murphy
Apr 6 2017 at 6:00pm
Ye whose purported subjective probabilities are poorly calibrated, confess to the community at the Towne Square!!
Richard
Apr 7 2017 at 12:05am
My mistake, I was skimming, thanks for pointing it out.
RPLong
Apr 10 2017 at 3:30pm
Another way to look at it is that he is now only making predictions about things he is sure of, regardless of what “subjective probability” he happens to attach to the prediction.
That would be a special version of robc’s “sandbagging” comment, but in this version of it, it’s impossible to self-detect because it would certainly be a subconscious act.
This brings me to the whole problem with the “rationalist community.” We presume that it’s “rational” to build a predictive model in anticipation of a particular outcome. But decisions have to be made about what goes in the model and what doesn’t. Ie., it’s subjectivity dressed up in the language of objectivity. Relevant variables might be omitted from the model for erroneous, subjective reasons. Irrelevant variables might be included in the model for erroneous, subjective reasons.
To my knowledge, rationalists aren’t doing single-variable analysis to determine which of the included variables really are statistically relevant. Or, when they are, their subconscious mind makes sure to pad the model with various other factors that are sure to make the important variables look better and the offensive variables look worse.
They call this rationalism, but it’s just a mental game we play to make our subjective beliefs look more credible.
Vera
Apr 18 2017 at 10:28pm
Do a controlled study with more than one individual and I’ll pay attention.
Comments are closed.