Why all the football statistics now available still leave us uncertain as to what is going on.

by Tony Attwood

Some time back, I criticised a book called The Numbers Game – which purported to reduce all football to a valid statistical analysis – and I was reminded of it again when another book appeared doing the same sort of thing: claiming it had analysed every aspect of the game and was able to set out in statistics why some teams won lots of games and others teams didn’t.

Both books had the same problem – they assumed that refereeing was both accurate and unbiased.  Assumed it so much that at least in The Numbers Game, and I am told also in subsequent volumes that have followed this approach, there is not even an acknowledgement that there is this assumption made.  It is made as an absolute and inviolable truth.

Now obviously for anyone who wants to put forward an analysis of data, there are assumptions to be made.  For example, we assume that the laws of gravity are constant and identical across all football pitches.  We don’t go out and test this, because basically there is no need.  There is no evidence to suggest anything other than the fact that such gravitational anomalies that do exist are so small that their influence on the flight of the ball are negligible.  A ball whacked up high in the air accelerates back to the ground at a set rate of 32 feet per second, per second in a stadium built and paid for by tax payers as it will be in a stadium built and paid for by a football club.

But when it comes to people, matters are not so easily resolved.

Yes they can collect data and generate things called a “pressing index”, “room control” and “pass effectiveness” and from this they can make all sorts of predictions about how good a club is and what it ought to get its players to do, for the club to be better.

And this data can be used to create ball possession statistics, pass rates, running distance or heat maps – but they don’t take into account external variables, such as the size of the pitch.  However that can readily be factored in of course, because it doesn’t change.  (It used to change – I remember the time when Crystal Palace reduced the width of their pitch in an attempt to avoid relegation – but that is not allowed now, so again the stats can be valid).

But referee variance is hard to factor in.  I thought of this particularly today when listening to a BBC Radio 5 discussion about the recent controversial referee decision in a world cup qualifier.  It was, as most radio and TV discussions, hampered by a complete lack of knowledge on the part of the pundit and the programme host about video evidence, bemoaning the fact that it had not been introduced, while utterly unaware that it did exist in some countries.   How they known that parts of Europe have it, they could then have asked the more interesting question: why does it not exist in this international match?

But they didn’t and such ignorance is a major hindrance to good debate.  And that was a shame because that debate, properly handled by a knowledgeable presenter and a knowledgeable pundit, could have asked, “how often do referee errors of this nature happen?”   And “do they happen to one country more than another?”   And “why, when the technology is available, is it not being used in this match?”

Statisticians do realise that statistics are always open to question, of course.  One example that is often quoted is the Champions League game Celtic Glasgow against FC Barcelona on 7 November 2012 . Here’s one commentary: “Barcelona dominated the game seemingly as they liked: 84 % possession, 25: 5 shots on goal, 955 passes and an outstanding pass rate of 91 %. Celtic made only 166 passes and delivered 38 percent bad passes . But the Scots won 2-1 and the first goal from Barcelona fell only in stoppage time. It was similar at the World Cup semi-final in Brazil 2014: The hosts had the better pass rate, more shots and were more often in the attack third. But Germany won 7-1.”

Such realisations lead to the introduction of more factors in the statistics.  As Daniel Memmert of the Institute of Training Science and Sports Informatics at the German Sport University Cologne said recently, possession  “says quite a lot about the dominance of a team, but rarely anything about their actual chances of victory.”

So David Memmert and the sports informatics specialist Jürgen Perl of the University of Mainz have gone further and set aside what is now called “naïve data analysis” to consider what are now thought to be “tactically relevant aspects, known as key performance indicators to determine more about the qualities of a team.”

Thus we now have, for example, the Pressingindex which shows how fast a team attacks the opponent after losing the ball. And “room control” which describes the dominance of a team over the spaces on the football field and is defined “by the area of ​​the grass that players of a team can reach in front of the players of the opposing team.”

Now what is interesting is that the analysis, funded by the German league, analysed 50 games in the Bundesliga season 2014/15 and saw how space control, pressing index and pass effectiveness parameters related to the results.

But still the point remains this is still “naïve data analysis”, because if (and it is only “if”) the referee is either not as competent as PGMO (for example) claims, or makes multiple mistakes which don’t “all even out in the end” then there is a broader overarching problem, which means that the data analysis still doesn’t really tell us what is going on.

It is interesting that our analysis (160 games analysed) which came complete with video evidence and which is available in detail for everyone to see on this site, didn’t just do 50 games, but did over three times as many games.  The conclusion was clear – referees make far more mistakes than is commonly realised, and they don’t “all even out in the end”.

What is just as interesting is that ever since the report was released two things have happened.  First the broader football world has ignored it.  Second, on this site people have tried to discredit it through pushing forwards three viewpoints over and over and over again.  (I don’t publish them all, but enough for anyone interested to be able to have a read through.

Those three viewpoints are:

a) that the 160 games survey was carried out by Arsenal supporters.   Although that is true, the existence of all the video evidence allows anyone to look for bias.  And it should be remembered that the conclusions reached were the same as those on the Referee Decisions website in which the analyses were undertaken by referees whose allegiance was with other clubs.

b) that we are pushing the viewpoint that referees are bent.   In fact what we are saying (so often that it is getting quite boring) is that we can’t tell if referees are incompetent or bent, but that the way the PGMO is set up as a secret society makes it impossible to anyone to know what is going on.  Quite why PGMO has set itself up like this, and has also restricted the number of referees in a way that other leagues have not, we don’t know. It would be nice if they would tell us, and would open up a bit, but their secrecy and decision making is suggestive of the notion that something is wrong.

c) that we have not provided any evidence.  This attempt to discredit us is getting very tedious, especially when we have produced so much evidence.   A recent example was the demand from a reader for evidence to show that the other main leagues in Europe have more registered referees than the PL does.  It’s an important point because if there were to be anything wrong with a referee, the fact that in the PL he could referee one club multiple times in a season would give him more chance to fix matches.

In fact we have published the data analysing the major leagues of Europe in this way on this site.  But by endlessly writing in comments demanding to see this data, those who deny the existence of a problem are attempting in a simplistic way to force us to spend hours going back and repeating over and over the data, rather than get on with new work.

There used to be a number of people who suggested that what was wrong with Mr Wenger as a manager was that he never changed his style or playing.  This was often accompanied by a fake quote assigned to Albert Einstein saying that doing something that failed over and over again in the hope that it would give a different outcome next time was a true sign of madness.

In fact doing the same thing over and over again – suggesting that we haven’t produced statistics and analysis when we have, and ignoring each new set of data that we produce – is what those opposed to our approach do, but it is not a sign of madness.  Rather it is a sign of desperation.   Millions of pounds are invested each year into analyses of football matches, and always on the assumption that referees are both capable of making the correct decisions almost all of the time, and not biased for or against one team at any time.

The fact is that if, as our figures show, the error rate within refereeing decisions is very high, then it becomes harder to see if there is bias.  If the average referee makes two errors in a game, and both are against one team, that is easy to spot.  If the average referee makes 22 errors in a game, and a couple of important ones are made against one team while the rest even out, that is much harder to spot.  One is looking for two “deliberate” errors among 22 random errors.   That is tough, and one can understand why the books that analyse football matches shy away from the task.

If there are 50 errors a game that is harder still to analyse out.

But without an analysis of referee errors alongside all the analysis of player activity, ultimately all the statistics leave us uncertain as to what is really going on.

• Chelsea’s technical director goes but Arsenal remain the masters of football chaos.
• As long as we fail to ask “why?” in football, nothing will ever change
• Stop, don’t move. There’s a new player we are going to sign. Honest it’s true. Or so I am told