by Last week we received feedback in the form of some comments underneath the preview of one of the last races of the 2020-21 season. One of our readers indicated that even if he liked our previews and our race simulations, he was unimpressed by the accuracy of our race predictions:
“I have enjoyed checking your previews this season. Your website is the only I know who is doing something like that for biathlon. I think it is creative and good looking, however I am not impressed with the accuracy of your predictions. I think your simulation which makes those predictions is not very good. But thanks for launching these previews anyway. Wish you to improve before the next season.”
As we value feedback highly, we naturally engaged in a discussion about the quality of our predictions. The original poster followed up by highlighting that:
“Bob Voulgaris who is well known for computer simulations of basketball, once said something along the lines of:
“Simulations not capable of beating the betting markets are not very valuable.”
I am confident that your biathlon simulations are incapable of beating the betting markets. You should not get too disheartened by this, even a huge website like 538 who probably have a lot more sophisticated simulations than what you have made, have been shown to not even be close to beating the betting markets.”
Knowing how much work has gone into building the model which is running our simulations we were surprised by the criticism. However being able to outperform the betting markets is a very tall order. To show an ability to profitably exploit the betting markets is a bar few models, used for previewing sport events on a public platform, will ever be able to clear.
Based on the above it is easier to understand where the original poster is coming from. Furthermore he is right about FiveThirtyEight. It is a website with truly a lot of traffic. Services like SEMrush and Similarweb indicates monthly traffic of some ten million and for some months even a lot more.
Considering the resources at FiveThirtyEight’s disposal I suspect that our reader can be correct and that their models can be more sophisticated than our biathlon model. It has been clearly established multiple times that FiveThirtyEight’s forecast are both well calibrated and able to achieve high Brier scores, which proves that their predictions are far superior to unskilled guesses. Even so FiveThirtyEight’s forecasts are still far from able of beating the betting markets.
The below paragraphs from the above article is informative:
“Combining an historical Pinnacle closing odds database with the implied probabilities made available by FiveThirtyEight, I put together a sample of 16,635 matches from European soccer leagues played between 12th August 12, 2016 and March 31, 2019, for a total of 49,905 home/draw/away odds pairs.
On 20,093 occasions, Pinnacle’s closing odds were longer than those implied by FiveThirtyEight’s probability forecasts. The average superiority of those odds (average 4.12) was 16.2%, implying that had we bet those odds to level stakes we should have made a profit of about 16.2% assuming FiveThirtyEight’s odds, on average, to be an accurate or efficient reflection of the ‘true’ odds. In fact, they showed a loss of -6.0%, worse than the -4.3% loss betting all 49,905 prices (although not statistically significantly so).”
Betting what FiveThirtyEight’s models indicated as value bets, did not only fail to win. It lost even more than betting all outcomes for every match would have done. In all likelihood betting random outcomes would have lost you a lot less than betting what FiveThirtyEight’s models indicated were value bets.
Still I think few people would have the audacity to claim that FiveThirtyEight’s models are crap, I think the models are good, just that the task, of showing a profit betting, is monumental.
The next article in this series we will cover how we obtained the odds needed to assess the performance of our model during the 2020-21 biathlon season.
