.png){kind=small}
Prediction is hard
2020 has been a tough year for predictors. Epidemiologists spent months modelling the impact of COVID-19, only to see their forecasts undermined by the President of the United States. Political pundits had a torrid time predicting the US Election (again), receiving a lot of criticism during a marathon election night (week), which saw Biden start heavy favourite before wildly swinging to outsider, then back to favourite within 24 hours. Simply put, predicting events is hard. One of the predictors who came under scrutiny during the election was Nate Silver, founder of fivethirtyeight.com (538), a politics, economics and sports blog rooted in statistical analysis.
After back-to-back elections where polls and most political predictors have been wrong, a lot of people have lost faith in the predictive power of statistical analysis. In the end 538 did predict a Biden victory (even if it did get some states wrong along the way).
During election season, everyone has a prediction and 538 always gets a lot of attention. But, did you know that every year they also create predictions for the Premier League? The top of the table is the closest it’s ever been at this point in the season (see our recent post). We have just passed the quarter-way mark of the season, and no team has established themselves as a clear favourite.
Chelsea are coming along nicely after an average start, but they have so many new players on this team, who knows if they can keep it together for a full season
Son and Kane are in career best form for Tottenham, but can they keep it up for the whole season?
The champions Liverpool are playing well despite a large injury list that could hurt them in a year with a condensed fixture
Both Manchester clubs have had an inconsistent start to the season, but are still very much in contention if they find some consistent form
Here at Tenner, we love predictions, so thought it would be interesting to analyse 538’s past Premier League predictions and see how accurate they were, and if they have any insights into who will win the 2020/21 season.
To make their predictions, 538 assign an offensive (and defensive) rating to each team that represents the number of goals it would expect to score (or concede) against an average team on a neutral field. Given these ratings, 538 can project the result of every match in a season and then put together an expected table. To form these ratings, 538 takes into account thousands of data points from past games as well as using play-by-play data. It all sounds very complex. But how good is it? And how accurately can it predict Premier League outcomes?
Another source of predictions for the Premier League, albeit indirect, are FIFA (the video game) ratings. FIFA uses a collection of individual player ratings to form team ratings. Indirectly, FIFA ranks teams but their rankings are not inherently trying to predict table positions for the season. Thus, it will be interesting to see whether FIFA, who are not inherently trying to predict the table, can do a better job than 538’s model that is set up exclusively for this purpose. For this analysis, we have assumed the FIFA team with the highest rating is predicted to finish first in the season and the team with the lowest ranking last.
Comparing 538 predictions to FIFA ratings
This analysis will look at the predictive power of 538 and FIFA ratings over the last four years (538 only started making predictions for the Premier League in 2016/17).
To evaluate which source is more accurate, we have plotted every team’s prediction on a matrix. On the x-axis is the actual finishing position. On the y-axis is the predicted finishing position according to 538 and FIFA. To evaluate which prediction is better we calculate the difference between the prediction and the actual finishing position and assign them into a category based on the value.
Perfect prediction: the prediction is equal to the actual position
Good prediction: the prediction is within two positions from actual position
Average prediction: the prediction is three of four positions away from actual position
Bad prediction: the prediction is five or six positions away from actual position
Not even close prediction: the prediction is seven or more positions away from actual position
Because table positions are discrete i.e. can only be whole numbers, in order to see if there are overlapping values, the data points are slightly jittered so that points don’t perfectly overlap.
Analysis
Overview
From just looking at the graph, it is hard to discern which source has better predictive power in regards to the Premier League - but it looks very close. The closer the values are to the diagonal dotted line, the more accurate.
Each point on the graph represents a team's position from the past four seasons. Hence there are 160 data points - 80 FIFA ratings and 80 538 predictions.
Of the 160 predictions, 83 (51.88%) were accurate to two positions, and only 22 (13.75%) predictions were deemed not even close (7 or more positions different to actual).
Below is a table comparing the two results of 538 and FIFA:
From the above table, it would suggest that 538 just slightly has the edge over FIFA ratings as it outperforms FIFA in the perfect and good prediction categories.
Segmenting on table position
As expected, it is much easier to predict the top 6 than other positions on the table. This is because, from year to year, it tends to be the same teams who finish in the top 6. In the last four years, FIFA and 538 have predicted 81.25% of the teams in the top 6 within two positions. However, things look quite different in the middle and end of the table. In the middle of the table only 45.31% of the teams are predicted within two of their actual position, while almost a third of these teams have predictions 5 or more away from their actual results. Interestingly, both sources struggle the most to predict the bottom six. 22.91% of the predictions were not even close meaning the actual result was 7 or more positions away from the predicted. Considering that the bottom of the ladder is bounded (by the 20th position), the predictions seem to be heavily overvaluing some teams.
Sum of difference between actual and predicted
From the above graphs, we can infer that 538 is a slightly better predictor than FIFA. However, the above graphs fail to give a good understanding of the magnitude of difference between 538 and FIFA. To get a more granular understanding of the magnitude we can take the sum of all the absolute differences between predicted position and actual position and see which is lower. This will give more representation to individual data points and better highlights any outliers or nuances in the dataset.
This means that over the last four years, 538 were better predictors than FIFA ratings by 24 positions. On average a 538 prediction was 2.95 positions away from the actual position whereas FIFA was 3.25. This is actually rather high considering there are only 20 positions on the table.
However, this analysis does not take into consideration the fact that we expect (and know) that predicting the top 6 of the table is a lot easier than predicting the middle and bottom of the table. This fact should be incorporated into the analysis. The weighting of the predictions for the top 6 teams should not be the same as the weighting of the predictions for teams positioned 7th to 20th. From the above analysis, we know that a good/perfect prediction was about 50% more likely to occur in the top 6 than teams from 7th to 20th. Therefore, the differences for teams that finished in the top 6 will be multiplied by a factor of 1.5 to account for the relative ease of making these predictions. The differences for the teams that finished 7th to 20th, will be skewed down by a factor of 1/1.5 due to the relative difficulty of predicting these outcomes. With these adjustments, it heavily penalises incorrect predictions for the top 6, but is more lenient from position 7 onwards. Using the new adjustment, the results are:
Adjusting for the difficulty of making predictions later on, 538 still remains the better predictor. However, now, on average a 538 prediction was 2.33 positions away from the actual position whereas FIFA was 2.63.
Interesting data points
The biggest outlier
One of the teams that both FIFA and 538 got very wrong was Wolverhampton’s performance in 2018/19. Both sources had predicted Wolves to be near the relegation zone, but in their first season in the Premier League, Wolves were great and finished seventh - the best result from a newly promoted team since Sunderland finished seventh in 1999/00.
The unanimous winner
In 2018/19, both 538 and FIFA ratings had Manchester City as the winner. It was the only year where they both predicted the same winner and indeed Manchester City did win. From the four years analysed, it was also the only year that either predictor correctly chose the winner.
The biggest difference
Although in most cases, the difference between the predicted position of 538 and FIFA is rather small there were some rather big differences. The biggest being West Ham in 2018/19 where FIFA had them as the 8th best ranked team but 538 had them finishing 18th and being relegated. In the end, West Ham finished 10th. 538 got this wrong, but it was not their worst....
The worst prediction
Strangely enough, the worst prediction for FIFA and 538 was the same team. Last season both FIFA and 538 agreed that Sheffield United would be last. But in a stunning season, Sheffield, in their first year in the Premier League finished 9th. Sadly things are not looking so good in 2020/21 as they sit bottom on the table after 13 games with 12 losses and 1 draw.
So who is going to win the Premier League in 2020/21?
As it stands, Liverpool sit at top the league by 3 points after getting a big three points against Tottenham courtesy of a 90th minute winner. At the start of the year, 538 had predicted Manchester City and FIFA had Liverpool as the top ranked team. As the season progresses, 538 update their projections and FIFA update their ratings. As it stands (as of December 18th), this is how both predictors are predicting the top of the table:
Even with a below average start to the year and City lagging leading Liverpool by eight points, 538’s live predictions still have City winning the league. However at the start of the year 538 gave City a 57% chance of winning, this has been downgraded to 46% while Liverpool’s chance of winning have been upgraded from 24% to 29%.
Who do you think is going to win the Premier League this year?
Come make your predictions with Tenner! Tenner runs completely free-to-play prediction games for the Football as well as a heap of other sports where if you are able to get every question right, you can win big cash prizes! Download the Tenner app today and start making your predictions now!
Comments