FiveThirtyEight is Dead; Long Live Public Soccer Projection Models

The preeminent public projection model is no longer. Where do we nerds go from here?

As the calendar turns to August and I gear up to declare my annual Premier League season predictions — coming next week in this here newsletter — the quest for optimal preseason predictions must begin in a new place for the 2023-24 season. Alas, our beloved soccer projection model king and what my phone tells me is my single most visited website is dead. Long live public projection models, or at least tools that allow the general public to have informed probabilistic perspectives on potential soccer outcomes.

In this post, I will explore a range of topics that may seem wholly uninteresting to the average citizen of the world but is weirdly very important to me and, in my mind, should be to the public more broadly: the good/bad/basics of FiveThirtyEight’s projection models, why I am eulogizing rather than utilizing them, existing alternatives and (crucially) why there are so few of them, and where I see the future of public projection models going.

FiveThirtyEight’s model: what it is, how it works, its strengths, its flaws

My assumption is some readers are similarly obsessed with FiveThirtyEight’s model and experiencing the same withdrawal symptoms I am, while other readers hardly know what I am talking about. Let’s start by bringing the latter group up to speed on the basics of the model.

FiveThirtyEight is a media platform, existing primarily in website form, that is owned by ABC, which is owned by The Walt Disney Company. The site was launched by Nate Silver in 2008 as essentially a personal blog. Silver is perhaps the most famous person working in the projection/probability/modeling business and deserves substantial credit for making such a “business” exist in a public-relevant way. The Michigan native previously worked at Baseball Prospectus, one of the original sports statistics nerd conglomerates, where he designed their seminal PECOTA projection models. FiveThirtyEight — referring to the number of electors in the United States electoral college — has primarily served as a platform for Silver’s US elections models, functioning first as an independent site, then under The New York Times, then ESPN, and now ABC. Silver and FiveThirtyEight have been a major part of American political coverage growing increasingly driven by data, polling, and evidence. Simultaneously, particularly under its ESPN ownership, the site expanded its coverage to all major sports, both with written content and, of course, projection models.

FiveThirtyEight launched its soccer projection models in January 2017. I could spend several paragraphs explaining the model, but, being practical here, the best starting point is simply reading FiveThirtyEight’s very clear explanation of how its model works. In summary (using present tense here because the model technically still exists; it’s just not outputting publicly): The model gives each team a score based on its on-pitch performance on a game-by-game level, primarily grounded in an adjusted goals calculation, shot-based expected goals, and non-shot-based expected goals. Pre- and early-season projections are aided by pinning down underlying squad value using Transfermarkt player values. Every single game has projected outcomes — available publicly on the site — using Poisson distributions. The model then scales up those game projections to a season level using Monte Carlo simulations. From there, the model outputs probabilities for each team at each finishing position. The model covers a wide breadth of major club leagues and UEFA cup competitions, plus World Cups. For those less familiar with the model, I recommend pulling up the site and clicking around.

The utility of the model should be pretty clear. When most people pull up the league table after a weekend of league action, I often instead clicked my FiveThirtyEight Premier League bookmark to check out how the results affected the projections. To me, getting an informed probability of key outcomes — such as Arsenal’s chance to win the league or Liverpool’s chance to finish in the top four — offers far more value than simply comparing point totals. Most fans when looking at a league table essentially use it as a basis to make projections in their heads: “We’re only three points off top four but are a notch better than our rivals, play them once more, and have an easier fixture list — I reckon we are slightly more likely than not to finish in the top four.” Yet, that mental process can be not only taxing, but is liable to miscalibrated calculations at several steps in that process. The most difficult part is pinning a reliable probability on potential outcomes, which particularly comes into play in cup competitions, where any given team’s chance at winning the competition is far lower than many fans realize. (Even entering the Champions League knockout stage last season, FiveThirtyEight gave Manchester City just a 21% chance of winning the competition, with only one other team (Bayern Munich) having a >8% chance.)

That brings us to FiveThirtyEight’s core value to the public: taking the same inputs (plus, crucially, some fancy advanced metrics) and doing that projection process for us, yielding informed probabilities for every data point we could want. The data exist to back-test FiveThirtyEight’s model accuracy at a decent sample, but my salary of $0 from this newsletter does not quite justify doing all that work to likely confirm the view I have developed from years of obsessively tracking this stuff: the model is much better at projecting the probability of outcomes than you, me, or any of us hopelessly flawed humans are capable of. It also came in a reasonably slick interface, updated practically the minute every match ended, and had match-specific projections that theoretically allowed fans to realize their club actually only had a ~50% chance of winning at the City Ground rather than 90%.

Beyond the value provided at an individual fan level, it is worth considering how strong public projection models bring value at a community, societal level. Over the past fifteen or so years, Silver’s election forecasts have led a shift in media coverage and public discourse around elections to being far more data-native. Polls and projections are the starting point of many American election conversations, with there being functionally a burden of proof on anyone disputing polling consensus. While soccer outcomes are surely lower stakes than election outcomes — actually, are we sure about that? — and soccer projection models have achieved far from the level of ubiquity than that of election projection models, greater proliferation of such models can help build more data fluency in the general public. Interest areas provide strong entry points for people to learn about important topics, and I am hopeful that projection models can help bring stronger understanding not only of what drives outcomes in soccer clubs’ seasons, but also more broadly of how to consider probabilistic outcomes. Thus far, FiveThirtyEight’s model quality and public awareness has given us the best foundation in the soccer space thus far, with plenty of room to grow.

Espousing the strong value of FiveThirtyEight’s model is not at all to say the model is perfect. The FiveThirtyEight model firmly falls into British statistician George Box’s famous maxim, “All models are wrong, but some are useful.” The most basic flaws are understandable ones: incorporating injuries, manager changes, fixture congestion and any other exogenous factors that match data and competition-specific fixture lists cannot capture. The model does not know if a team is facing an injury crisis until the performance loss meaningfully manifests itself in the match data, which could take a long time, probably even a longer duration than the relevant injuries. The model faces a similar flaw with being unaware of manager changes, though evidence suggests the identity of the man in the technical area affects performance less than conventional wisdom seems to think. The model also lacks awareness of a club’s broader fixture list. If it knew that one club gunning for top four is deep into Champions/Europa League and FA Cup runs but a competitive club is only playing Premier League matches, the model would bump the probability of the latter club. Each of these issues is perhaps solvable in some capacity, but it would take a massive data analysis task to get even in the range of knowing how much these factors should impact projections.

Another potential flaw is more complex and perhaps more a unique model choice than an outright issue. To my eye, the model’s sensitivity — that is, the extent to which its projections adjust following a particular result — is well-calibrated but may be a notch too sticky. For example, at the turn of the calendar this past season, Liverpool were in sixth place, five points off fourth place and just one ahead of Chelsea’s disastrous campaign. Many Liverpool fans were quite upset at Liverpool’s performance and dismissed the team as dramatically underperforming. Yet, FiveThirtyEight still considered Liverpool to be the second-best team in the league, with the third-best odds for top four. Now, I think this example actually primarily demonstrates a key value of projection models: it is much truer to actual team quality and probabilities than us overly emotional, irrational fans. (One of my favorite criticisms of FiveThirtyEight that I have seen is an Arsenal fan in spring 2022 amid the club’s ultra-tight top four race complaining that the projections adjust too much following results. I then scrolled through his Twitter feed to see that his view on Arsenal’s quality and odds seemed to swing much more frequently and extremely than FiveThirtyEight’s!) The Liverpool example specifically showed the model’s value, as the model effectively saw the club was just a handful of months removed from a nearly-perfect season with an extremely talented squad. The model’s optimism proceeded to arguably be vindicated by Liverpool finishing with a closer shout at top four than many fans expected throughout the season.

However, FiveThirtyEight seemed to stubbornly cling to its perception of intrinsic squad quality just a little too much for just a little too long — certainly more than other models did as I compared throughout the season. Overall, determining whether the model’s sensitivity is accurate would take significant samples of back-testing, and any (good) model like this would be typically less sensitive to results than most fans’ perceptions. Yet, if somebody wanted to complain that FiveThirtyEight’s model is a notch or two too sticky on squad quality, they would have a point.

Of course, we as humans can correct for these flaws. Rather than saying, “FiveThirtyEight doesn’t know Marcus Rashford is hurt — it’s rubbish!” and dismissing it entirely, we can embrace the usefulness of the model and make these adjustments ourselves. We can use the model outputs — say, the model giving Manchester United a 73% chance of finishing in the top four — then in turn consider these exogenous factors to make some personal adjustments to the projections. For instance, if I think the model overstates Manchester United’s chances because it does not recognize the club’s injury and fixture congestion situation relative to their competitors, I would scale down Manchester United’s chance to something like 65%. We can accept the model’s imperfections while feeling quite happy with its usefulness.

Now that we have established a strong understanding of the FiveThirtyEight model soccer projection models writ large, let’s discuss its swift demise and where we go from here.

Our king meets its fate

On April 25th, Silver announced he would be leaving FiveThirtyEight imminently. As part of the massive cost-cutting directives seen across the media industry in 2023, Disney has enacted widespread layoffs across its subsidiaries. For whatever reason, Disney did not perceive Silver as worth keeping on staff. Curiously, FiveThirtyEight continues to exist, with G. Elliott Morris (formerly of The Economist and Silver’s angry Twitter mentions) stepping into essentially Silver’s role.

The key here for our purposes is Silver’s prudent choice to own most of FiveThirtyEight’s models himself and license the use of them to FiveThirtyEight/Disney, as Silver explains on his Substack. It has never been formally announced which models will persist and which have faced a quiet death, but simply clicking through the site indicates all its sports models stopped updating in June. In fact, the journalistic side of the site, which once featured multiple sports posts per day, has been left barren since April. In prior eras, FiveThirtyEight would have created a Women’s World Cup model, but that opportunity has come and gone. Clearly, the soccer projection model is dead, with no mention of future usage plans.

It is, however, worth considering the same model may return elsewhere. Silver said, “I still own these models, and can license or sell them elsewhere.” It remains to be seen what Silver will do next beyond his current newsletter and poker book, but one can imagine several possibilities for bringing the sports models back. I would advise not to hold our breaths, as there is little indication the soccer model is a priority for Silver, and I suspect his next venture to have an altered scope from what FiveThirtyEight has executed over the past decade.

Alternatives are surprisingly sparse — for a key reason

It may not seem clear to a reasonable cynic why the death of a model that could be described as basic, unremarkable, or flawed is so disappointing — reference header image meme — until we realize just how few alternatives there are. For publicly available club soccer projections, we are stuck with this very finite group of alternatives:

• Opta Analyst: Opta (now a subsidiary of increasingly monopolistic Stats Perform) is the official date provider of the Premier League and many other global leagues. It has basically all the data a projection modeler could want. The company has a public-facing content group called The Analyst, which has solid analytics-y content. Opta Analyst has FiveThirtyEight-like projections you can occasionally find on its website and Twitter account. It also shares projections for competitions like the Women’s World Cup and English Championship. Yet, as far as I can tell, there is no information on how the models work — except for branding it as coming from the “Opta Supercomputer,” a word that immediately has me skeptical — and no persistent interface. Potential exists here, but it clearly falls short from what FiveThirtyEight offered the public.

• Twenty First Group: Twenty First Group is a B2B sports intelligence company that essentially functions as a consultancy for sports rightsholders and related parties. TFG has access to key data feeds and possesses a strong degree of internal analytics capabilities, with one of its outputs coming in the form of projection models for the Premier League and other key competitions. TFG periodically shares their models on its Twitter account and on that of data scientist Aurel Nazmiu. From my eye and my understanding that TFG is great at what they do, these are strong models. However, as indicated by my essentially needing to say, “follow these Twitter accounts and you’ll eventually see some projections,” it lacks the public value of providing that always-accessible interface that FiveThirtyEight offered. (With TFG being a B2B enterprise not really in the consumer-facing content game, it’s a fair enough strategic/product decision from them to keep the models largely internal and refrain from committing resources to a public product.)

• Club Elo: If you are familiar with the concept of Elo in chess (or otherwise), this is exactly that but applied to soccer. If you are not familiar, well, read the site’s explanation. Elo is a nice concept that is especially effective at tracking a club’s quality over higher-level time horizons, but the way the data is packaged does not fulfill a true projection use case. Utility exists here, but not the same type.

• Bookmakers: It’s perhaps surprising we got this far without mentioning gambling, as I am sure a large portion of ardent FiveThirtyEight visitors used it to try to beat the betting markets. This hardcore analytics projection game is the backbone of bookmakers’ ability to make absurd sums of money. Of course, they have negative interest in letting the public under the hood by explaining their models. Additionally, to the extent they have public interfaces, they are usually terrible. There is a solid cohort of analytics writers who use betting markets as their projection models of choice, but you would struggle to convince me that my post-FiveThirtyEight itch would be scratched by going to a janky bookie website with shiny banner ads desperate to take every last dollar I have to find point totals that are not even necessarily raw projections but instead optimized for maximizing profit. That said, a bookmakers’ site can fulfill the needs of being always available and presumably well-calibrated. TFG’s Chief Intelligence Officer Omar Chaudhuri regularly shares Premier League projection updates on his Twitter using data from Sporting Index. Chaudhuri collects data from SI each gameweek and puts it in his interface. Tracking these through Chaudhuri is a nice way of seeing snapshotted projection benchmarks but again quite obviously falls short of the value FiveThirtyEight provided.

• Some random folks on Twitter: Various individuals have their own projection models and share them on Twitter. Of course, we have no clue how good they are, and the interface available to us is typically a static, ugly Excel screenshot. Jan Van Haaren, Head of Data & Technology at Club Brugge, collected all data-driven Women’s World Cup projections he could find. Even with the WWC being a relatively discrete and major event, which tends to attract more projections, it is telling that Van Haaren was ultimately only able to find nine projections. The majority of them are some random folks on Twitter. The others are either already listed here or do not seem to have more widespread models beyond the WWC. It is great that civilian privateers are building projection models, but again, this is simply not on par with FiveThirtyEight’s offering.

That’s… a very short list. (I am terrified that I neglected to include something notable. If you think I did, please let me know. If sufficiently relevant, I will update accordingly.) The limited existence of this civilian privateer group shows some promise for potential alternatives down the line, but it prompts the question: Why is this list so short?

To build a quality projection model, you need two key inputs: (1) relevant data and (2) sufficient analytical expertise to construct the model. Perhaps the most important point of this post is that relevant data is the far more limiting factor than sufficient analytical expertise.

Let’s start by addressing the second key input: sufficient analytical expertise. There are a ton of talented data scientists out there. There are a ton of soccer fans out there. A projection model is exactly the sort of thing that a data science student would seek to build to develop or showcase their skills, while more seasoned professionals would happily build on the side for fun. These privateers cannot necessarily build out a site as slick as FiveThirtyEight’s, but it only takes one or two to get a reasonable facsimile stood up. Perhaps more relevantly, many of these talented individuals work for companies where such models could be of some value to the business.

That runs us up against the real limiting factor: data. Soccer data is ridiculously walled off and expensive. I would quote some numbers, but (a) it heavily depends on the league/season/metric scope and (b) relevant providers (Opta and StatsBomb, mostly) probably would not even bother to discuss with me. My understanding is we are talking minimum five figures for relatively basic data, with minimum six figures for sufficient data needed to build a robust projection model. I will try to dig into why this is the case and discuss more in a future post, but the general reason is that soccer data these days is highly centralized, closely guarded due to being the product of surprisingly high-cost businesses, and simply very valuable.

Such costs of course price out the overwhelming majority of individual modelers. One-off international events like the WWC require less robust underlying data to have a passable model, with many of these models being built out with likely some basic result data and Poisson distributions. Yet, even for an enterprise, building a projection model gets very expensive very quickly: you need years of historical data to train and back-test your models, then, critically, perpetual live data feeding into and real-time updating the model. On top of that, you need talent who can build a strong model (a relatively easy component as discussed above, but far from a given) and feel that their putting time toward this is worth their wages and opportunity cost for that period of time. You need a product/design team and engineers to build out the interface. You need to ensure the model actually gets updated as soon as results come in. And you need this all to come at a high enough quality so as to not damage credibility to your business (which, after the 2016 US election, Silver himself can tell you is a risk even if you have a best-in-class model).

Projecting the future of projection models

Finally, allow me to make some projections about where the projection model space goes in the foreseeable future. Let’s start by assessing the futures of the currently participating players discussed above:

• Opta Analyst: This has a high probability of scaling up to a FiveThirtyEight-like product. The input data is going nowhere, they have the talent, they seem to want to make “The Analyst” product relevant, they are happy to share the projections publicly, and they already have some always-on stats hubs. It’s not hard to see this becoming a solid FiveThirtyEight replacement, to the point where I scoured the site to make sure it was not simply buried somewhere I could not find (and am still not fully sure it’s not there somewhere). However, there is an adjacent alternative that may be more likely discussed in the “new players” section.

• Twenty First Group: It seems they do not see building out a public-facing projection model as core to their B2B consulting business and could in fact see it as a competitive advantage to keep them entirely walled off from the public (a route StatsBomb takes with essentially everything they do). However, TFG seem to be more vocal in sharing their projections on social media than they used to be, and they just launched an interface-y World Sports Rankings. TFG has all the capabilities to productize their models and use them as essentially marketing for their business, though it is unclear whether they see it as worth the resource commitment.

• Club Elo: This just seems like a site that will look and function like this until the next apocalyptic meteor shower.

• Bookmakers: Their UI/UX will continue to improve as the definitely-not-gambling companies like FanDuel and DraftKings gain market share, but it has the fundamental positioning of not being quite what we are talking about here. At best, you are navigating through a virtual casino to see numbers fashioned for profit maximization.

• Some random folks on Twitter: This segment will probably see some proliferation as interest and talent in data analytics continues to grow, but until the limiting factor of data cost subsides, this group is stuck with at-best basic models. And interfaces of ugly Excel screenshots.

Now, perhaps more interestingly, let us consider some potential new entrants and why they may be interested in assuming FiveThirtyEight’s vacated throne:

• Wherever Silver goes next: Let’s start with the null hypothesis. Silver is certainly not done as a public media figure and modeler and will have a strong pick of where to take his career next. One could envision partnerships with The New York Times, The Atlantic, The Washington Post, and the like; a supersized Substack page; an independent website; or perhaps something more outlandish. The issue with any of these is Silver would face the same data cost issue. Even though he comes with a ready-built model, strong credibility, and a robust readership, I struggle to see The Washington Post perceiving the cost of the relevant data for a soccer projection model — never mind for all of Silver’s other sports models — as sufficiently worth it. There may be a scenario where the cost bar is achievably low, but I see this as a less probable outcome than we would perhaps expect.

• FBRef: FBRef is Sports Reference’s soccer product, launched in 2018. It possesses the foundational data sets used for most posts in this newsletter as well as just about any soccer article you read that uses data. Opta provides FBRef with tons of data. Sports Reference seems determined to make FBRef even more firmly the central source of public soccer data, including extending its premium Stathead product to soccer in the near future. FBRef’s entire function is a data resource for soccer analytics nerds, it has all the data, and it has the incentive to build such a product. I would be surprised and disappointed if FBRef does not venture into the projection business in the next few years.

• StatsBomb: StatsBomb is another data provider and B2B organizational partner for global soccer clubs, not too dissimilar from Opta. Despite originally launching as a soccer blog and continuing to host smatterings of public-facing content, the business seems to increasingly keep its data and insights behind closed doors, preferring to keep their assets for their clients except for one-off data releases. StatsBomb has the data and the capabilities, but launching a public projection model would run counter to the direction the business has chosen to go. Their models would be great, but we should not expect to ever see them.

• The Athletic: The Athletic has, for my money, the best comprehensive soccer coverage anywhere. They have strong analytics-oriented writers and traditionally have not been afraid to invest. However, under recent New York Times ownership, the previously venture-backed company has faced a series of belt-tightening measures. I don’t think The Athletic has the necessary data, probably not quite the necessary data science capabilities, nor the track record of building out content outside of writing and podcasts. I can’t quite see an investment being made here.

• ESPN, CBS, NBC, FOX, Apple, Amazon etc.: These are massive media/technology conglomerates with deep pockets and major soccer rights. This means they have the relevant data assets and talent (or at least the ability to acquire them), and incentive to create strong content, both in their television broadcasts and, in some cases, accompanying written content. One could imagine ESPN building off of Silver’s model — which, complicatedly, is grounded in ESPN’s SPI model, which Silver developed for ESPN prior to ESPN even owning FiveThirtyEight — and having an interface on ESPN.com, then those projections referenced on pre- and postgame content. They could even be looped in as live projections on game broadcasts. We have seen bits like this before, with ESPN integrating game projections in pregame content for basketball, college football, tennis, and other sports — though often with flagrantly bad projections, minimal context provided by broadcasters, and no accompanying online interface. Significant potential exists here, but I am skeptical of austerity-era media companies seeing the building out and meaningful integration of projection models a worthwhile pursuit for now. And if they do, I am further skeptical they would build them out to a satisfactory level that gets anywhere close to maximizing the models’ potential value.

While the tragic death of FiveThirtyEight’s soccer model has brought sad times upon us projection zealots, I am optimistic we will see a reasonable replacement soon. Any power (and data) brokers out there, I encourage you to consider the potential opportunity of pursuing a public model amid the current void in the market. Many paths to monetization and enterprise value exist beyond or aside from how FiveThirtyEight positioned its models, and the value to the public — both as soccer fans and data-minded individuals — has significant potential. I look forward to a gleeful future From the Byline post hailing the emergence of a new king.