EPL REFEREE REVIEW 2012: THE FIRST NUMBERS : How many games, the refs and their competency

By DogFace and Walter Broeckx

Untold Arsenal has a team of qualified referees who have reviewed more than 40% of the EPL games from last season. The reviews themselves were based on full match video footage with the advantage of video technology features such as slow motion and pause.

By reviewing those 155 games we have made a database of more than 7000 decisions that have been judged by our panel of dedicated and qualified referees.

The numbers you will see are based on those decisions and those reviewed games.

Welcome to the start of our investigation; in these first two articles [of increasing granularity] we will introduce you to the data analysis tools we will be using for this series. Initially we will take a holistic approach i.e. we will crunch the Ref Review numbers of the league as a whole with all teams and all referee’s included.

With each subsequent series of articles we will attempt to drill down into more specific questions such as referee(s) performance and inconsistancy with regards to a regional and team perspective.

For the purposes of analysis the 155 games we have reviewed have been boiled down and split into the perspective of the two respective teams – this is why there are 2 rows of ‘concentrated’ match data per game i.e. the referee’s performance against the home team and the referee’s performance in regards to the away team – these are essentially a reflection of each other and creates a zero-sum calculation when all teams/referees are included – but as you will see later, when we get into the specifics of home/away, teams or regions, this method will yield some interesting results.

This is worth bearing in mind, to avoid confusion, when studying the graphs with regards to the stated numbers of matches analysed.

So here we start with the numbers part.

The first series of numbers is how many games we reviewed of each team and how much this is from the total games they played in the PL last season.

Overview of games reviewed

PGMOL Vs EPL 2011/2012

Let me start by saying sorry for the supporters of teams further down the league table. To do our reviews we had to work with the games that were broadcast by the different sport channels on TV (in different continents even!).  Our ref reviewers don’t decide which games are on TV so we just have to live with what they show. And they usually show the top teams like Manchester United, Manchester City, Arsenal, Chelsea and other teams that are close to the top. The good thing for the smaller teams is that they are in the reviewing system when they play the big teams – this should therefore be enough to shake out the theory that the big teams get all the calls – a common complaint from the supporters of the teams lower down the league.

I know that in a way this isn’t totally fair for the small teams but we just have to work with the things we have at our disposal.

As this is primarily written for an Arsenal supporters blog it is not a surprise that we have done all 38 games of Arsenal. The next team we covered the most is Chelsea followed by Manchester City and Manchester United. We have done 32 of their games and that is 84,21% of their total games in the PL; next is Tottenham and Liverpool with around 20 games reviewed and that is just over 50% of the games. We then have the teams like Newcastle, Sunderland, Everton, Norwich, Blackburn, Bolton and Fulham – we have full match reviews for something between 40% and 26% of the games of these teams.

And finally we have the teams that didn’t get much coverage live on TV. But even from those teams the lowest number of reviews is 6 and that is more than 15% of their games. I would have loved it, if we could have offered more but that is the way things went.

So here is cry for help. Anyone who has access to video archives of games if he could open up his archives to us. Just imagine if we could pick out any game and start reviewing it. But that itself wouldn’t be enough.

So here is another shout for help. If you are a qualified ref and you want to help us in the future. Please let us know! This project shouldn’t stop right here. In fact it should go further from here. If we could get in more referees to review the games our numbers will become even better. So if you are a ref yourself and want to be involved in this: give us a shout.

If those two things would happen in the future we could set up something that would be a kind of shadow-PGMOL organisation that would be able to judge the refs in the open and not behind closed doors like the PGMOL does.

So why don’t you just join us?

Like I have said in the introduction article we can only use what we got. So we will treat those games as the data for each team. Of course doing a team over 30 games brings up a better overall view then doing a team just 6 games.  If you are very critical about that, fine. Then why don’t you look for a ref in your friends circle and ask him to join our team. So we can do more games and most of all, do more games of the smaller teams (with all respect).

The next set of numbers we will show is how many games each ref has been reviewed and how much this represents their season in the EPL.

Overview of referee’s reviewed

Referee Reviewed Percentage
Andre Marriner 10 43.48
Anthony Taylor 5 20.00
Chris Foy 9 32.14
Howard Webb 17 54.84
Jonathan Moss 1 5.26
Kevin Friend 7 29.17
Lee Mason 6 22.22
Lee Probert 9 33.33
Mark Clattenburg 9 36.00
Mark Halsey 9 33.33
Martin Atkinson 10 37.04
Michael Jones 12 42.86
Michael Oliver 8 30.77
Mike Dean 18 52.94
Neil Swarbrick 2 10.53
Peter Walton 7 38.89
Phil Dowd 12 38.71
Stuart Attwell 3 18.75

So in numbers the most reviewed ref last season was Mike Dean with 18 games. Closely followed by Howard Webb with 17 games.  But if you compare this with the total games they did in the EPL last season you can see that the the 17 from Webb represent a higher % of games from his total EPL games compared to Dean.

The same remark about the team can also be made about the refs. The more one reviewed the refs the more accurate our data will be.  So the numbers from Johathan Moss, Neil Swarbrick and Stuart Attwell have to be looked at with lots of question marks.  As has been said before: also refs have their ups and downs. And so they could be more vulnerable to mistakes when they have just a few games done in the reviewing system.  We will give their numbers but will remind you to keep the bag of salt ready just in case you see some strange things in their numbers.

But I think it is fair to say that from every ref we have reviewed at least 10 times we will have a great insight in how they have performed and the more reliable their good points will become and the more visible any flaws will become.

So it will be most interesting to see the final numbers from Marinner, Webb, Atkinson, Jones, Dean and Dowd as we have done more than of their games and in % of their total games. In fact we have covered more than 1/3 of their games. There are only 3 refs whom we have covered less than 20% and those refs are Jonathan Moss, Neil Swarbrick and Attwell.  But as with the teams we will try to cover the few games we have but you can understand that their numbers are not as reliable as other are.

How competent have the refs been in the games we reviewed?
PGMOL Vs EPL 2011/2012

In this graphic we have made the difference between the different types of decisions. With the “low” decisions we talk about the fouls on the field. The “Medium” decisions are in fact the decisions about yellow cards. And the “High” decisions are the decisions about red cards, penalties and goals. And for all those different types of decisions we have the correct and incorrect decisions.

As you can see on a total of all the decisions the total of correct decisions was 72,49%. The incorrect decisions was 27,51%. This means that when a ref has to make 4 decisions 1 of them is wrong. And if we put weight on the decisions the number of correct calls goes even further down.

The top referees in the country. Professional referees,  the best possible preparations, rather well paid and still they only manage to just hit the 70% mark of correct decisions. This is low. Too low in my opinion.

And certainly if you keep in mind that in case of doubt we gave the decision as a correct one to the ref. So the chance that if we would have been able to determine our own camera angle on some of the decisions, the total score of correct decisions could have even been lower.

I said in my introduction articles that a minimum score of 70% would be the minimum for a ref. But for all the refs together this clearly indicates that there are some refs who get a higher % of correct decisions but that in some cases you have refs who get a score below 70%. And this is something that I find unacceptable for what should be the best league in the world and should be refereed by the best referees in the world – but at least now we have an average set of figures for basic match competency in the EPL that we can reference in later articles and this is an important first step.

Competency per type of decisions

PGMOL Vs EPL 2011/2012

There are two types of decisions that at first sight are fine.  If you allow me I will start with the offside decisions. Mike Riley gave some numbers this season and said that 99% of the offside decisions were correct. We only come up to something of 90,199%. And I know that a lot of the correct decisions were given as correct just because we couldn’t check them.   So there is a big difference between our verified number and the unverifiable number of Mike Riley.

Let us move to the goal decisions. 91,753% of the goals are correct. But that is just over 9 out of 10 goals! This means that you almost have 2 or 3 goals each match day that are wrong given or wrongly disallowed. And we must keep in mind that goals is what wins games. So it is the most important decision we must get right. This number should be around 99% correct but we still are a bit away from such a score.

The other decisions are the fouls in general. The score there is 71,960% correct and this is in line with the overall competency of the refs one could say. Not great, just enough.

Another important issue is the penalties. I know it is difficult with some players diving around a bit. So the refs only getting 62,241% of the penalty decisions correct is also because of the bad behaviour of some players. But still the importance of these decisions justify that something should be done about it.

If we look at the red cards we see…a complete disaster imo. I have seen players kick each other off the ball, I’ve seen players punch each other off the ball, I’ve seen elbows against heads, I’ve seen horror tackles… some even unpunished – not even a foul given – let alone a red card.  It really is time to clean up football. The things I have mentioned do not belong on a football field and should be banned. And if the ref is not brave enough or did not see it, the FA should take action. But last season they have let me and the referee world down on that by not taking serious action against some actions.

The final table shows that the yellow cards are only given correct for not even 60%. One of the great issues here is the consistency. It is great to see a yellow card given when a player comes in to the ref waving his arms and protesting a decision. But when another player from another team does the same and the ref gives no card you really do wonder what is happening. Consistency is what we ask and this is very hard to get it seems.

But we surely will come back to this later in this series.


Who is not fit to referee?

Let us go one step further and move on to a next graphic. In this graphic we can show you who has been top in making wrong calls.

EPL Vs PGMOL 2011/2012

As this is not filtered on home and away games the score of the refs will be as high above as below the zero line for each ref. But of course the higher the lines in total are the higher number of wrong calls the ref has produced. So Peter Walton is very much in front of this table.  The more to the right the better the ref one could say… but please bear in mind that we have more referee/match reviews for some refs than others and this is shown on the graph by the number just below the referee’s name – the best referee looks to be Jonathan Moss, but we only have one match review (refereed 2 teams in our data) – so we must take this with a pinch of salt.

Let us take it yet again another step further and put the weight from the calls in it.

EPL Vs PGMOL 2011/2012

It is interesting to note that with the weighting applied (big calls worth more than small calls) the order of competency changes.

Now we don’t see Peter Walton on top (but still a good second) but Mark Halsey. If you ever want to have a look at the refs who have been having bad games last season just look at those tables from left to right and you will find the answer.

But we will dig in to those numbers later on in a more detailed way.

Next article in this series:  something about bias. Was Mike Riley correct when he said there is no bias in the refs in the PL?

37 Replies to “EPL REFEREE REVIEW 2012: THE FIRST NUMBERS : How many games, the refs and their competency”

  1. Great work, look forward to the rest of the series. Hope it leads to action from the PGMOL and the authorities.

    Scary results, there must be a way to improve these.

  2. Fantastic bit of work. Do we have any newspapers or serious tv documentary programmes interested yet?

  3. Well done both of you and thanks.

    One thought: could the results for red and yellow cards be split into –
    red cards given, correctly and incorrectly

    red cards NOT given that should have been in the opinion of our panel.

    Ditto for yellow cards.

    I suspect that this will throw up some very interesting insights.

  4. @Gf60 – I had the same thought… unfortunately this was a graph that I had to create this morning, it was all a bit of a rush and didn’t have time. I will look into it.

  5. Not surprised about the yellow/red cards. If you look at the amount of cards issued across Europe the premier league is at the bottom of the table by a fair amount.

    This indicates either:
    1.The EPL is the “cleanest” league in Europe…

    2.Referees are more lenient in England

    3.EPL referees are the least competent in regards to what deserves a card & what does not.

    4. A combination of 2 & 3.

    I think they’re just rubbish.

  6. Dogface
    Of the 40 incorrect goals given/not given, how many of those are a direct result of pens given/not given?

  7. @Stevie E – I dunno, but we wouldn’t count a not given penalty as a goal that should have been… I think most of these were when the keeper (or others) were being fouled to gain an advantage, handball, ball did/didn’t cross the line or offside etc.

  8. Loving this data. Any chance you can attach some larger graphics to see the names better?

  9. @Dogface
    The reason I ask is, if half of the incorrectly given pens are also chalked up as incorrectly given goals, by eradicating incorrect pens (by bringing in TV replay FFS), the incorrectly given goal figure would drop to almost zero. So much for goal line technology…

  10. Mike,
    If this very important work does start to get serious mainstream media interest (such as TV/news documentary), I wouldn’t publicize that fact too far in advance; not until it’s a done deal. In the, alas, real world where PGMOL and backers rule the roost, such caution would give this work a far better chance than otherwise to reach the widest footballing public. I wouldn’t count on those deeply invested in football’s status quo to restrain themselves in trying to condomize or otherwise discredit these emerging results.

  11. Xavier,
    You might want to try using the keyboard combination of CTRL and Plus sign to enlarge your screen image.

  12. Wengerball,
    Interesting points. Where would you find the numbers that compare EPL vs non-EPL carding? Is there a link, or would you just write up the comparison?

  13. A couple of things stick out for me with that weighted deconstructed graph:

    1) While Howard Webb is an imposing figure and asserts his control well on the pitch, he is a rather average ref.

    2) Relative to their positions on the chart, Dean, Probert, and Webb made relatively fewer mistakes on the bigger calls, but made more mistakes on the routine calls.

  14. @bob,
    I tried that already. The picture gets bigger but the graphic is still low quality so reading the ref names and the percentages of the “All Matches” and “Reviewed Matches” is a chore.

    Not that I can’t make them out but hq graphics, especially with data like this, are always appreciated =P

  15. Sorry about that – full interactive HD graphics will be avaliable once we have a wealthy sponsor to pay our mortgages & bills so we can do this full time.


    Seriously though – I can generate at larger resolutions but it ups the traffic and can slow down the site, we are trying to cram as much info as possible on these so it’s a bit of a trade-off.

  16. Stevie E,

    When a penalty should have been given we don’t count that as a not given goal. We only mark it as a not correct penalty decision.

    When a team scores direct from a not correct awarded penalty it is counted both as a wrong penalty and a wrong goal. There are a few of these in the decisions that I can remember but only a handful

  17. Walter
    Thanks for that, I was interested in wether goal line techology would have much of an impact if the cause of the problem would need to be cured in other ways. I wonder if the clamp down on diving (except for man u) has made the refs even more reluctant to give pens?

  18. A grand job and well done.

    On these results, Walter, in your view are any of the reviewed referees worth their salt?

    How much of the poor refeering is to do with the players?
    They do cheat, some much more than others; some seem to cheat habitually. Are some referees less able to spot it? Has any of that come through from the ref’s reviews?

  19. great job Dogface and Walter..shocking numbers on the red card part no wonder many players get leg breaks nowadays

  20. Matt Clarke you will find some kind of answer to some of those questions in the next days.
    But not going to give it all away today 😉

  21. 🙂 thanks Walter.

    [what follows is an attempt to post a Facebook like link for this. FB have reportedly removed support for such, so it may well not work…]

  22. @bob

    I am in the process of collecting red card data for the last 5 years in the top 5 leagues with the view to doing a write up.

    Here is a quick breakdown for 2011/12 league->yellows->red


    Bundesliga is 18 teams so there are 148 less games than EPL or about 20% less games, but the card count is about 8% less than te premier league.

  23. Wengerball,
    interesting numbers…
    Lets see… a first thought:

    the leagues where they give more red and yellow cards played the 1/2 finals of the euro’s….

    Players better protected by the refs > players less injured > players fitter for the euros > better results….

  24. Great and informative article. Can’t wait to read rest.

    @Dogface now I’m really sad that I didn’t won lotto on Sunday (from that € 11 million I could have sponsored this site little bit).

    @Johnny Deigh: Walter has mentioned earlier that the best way for ref to tilt the match is by making “mistake” in smaller routine calls, since more people ignore those mistakes but 99% will see mistakes on big calls. Just made me wonder more about fairness (bias) of Dean, Webb and Probert.

  25. Thanks for that peace Dog and Walter, so this time around do you think we are going to be seeing the worst or the best of refereeing? Dog face your name shake snoopy dog was snipped up by a dog in Amsterdam with a lot of joints.
    Do you think the EPL is going to get us some extra refs or we will have the usual suspects. I would love to see that lady ref in the middle this season.

  26. This is fantastic work and I can’t wait for the next instalment, well done. Without wishing to impose even more work on you, would it be possible to do a ‘league table’ of penalties won as a result of diving. (I suspect it may be Man U at the top of the table but I could be wrong).

  27. Walter & Tony….get ready for another cyber-attack against UA….data like this is a real threat to the PGMOL and the EPL, as well as the FA and the media AFC hating hacks calling themselves journalists!

  28. Great stuff. “99% correct” on offside calls indeed… classic overconfidence.

    So we know that there is a huge amount of room for improvement on accuracy – yet another huge argument for televised ref support. You really have to wonder how much white noise the matches suffer when a quarter of refereeing decisions are wrong.

    But OK; inaccuracy is human and doesn’t necessarily mean the game is ‘broken’ (just more random)… what will be much more telling will be the bias study… looking forward to it very much!

  29. @ Phoenix

    Not “classic overconfidence” in my opinion but arrogant, unsubstantiated bullshit……

  30. Well done to you guys. You are an asset to the Beautiful game.

    @Walter -I love your simplicity in working out the Euro impact because of cards. It is also true of skills in the country. The kids with skills get their legs broken or kicked off the park. There is so much to be said for academies within clubs. At least some of the talent is protected.

  31. A failing in much of the analysis done by several sources is the over-simplistic way that bias is treated. Yellow card counts for each team is often used to show referees being even handed.

    They don’t tell the whole story. Take for example – one match where time wasting was evident from the early goal. The referee only booked the goal keeper in time-added on for time wasting. Hpw absurd – why not book the goalkeeper a lot earlier? yet the stats will show that the goalkeeper was appropriately booked for time wasting – Farcical refereeing.

    Then there is the “I’ve been massively biased I can now afford to be fair” syndrome. Theat’s where the bias of the referee has been crucial in allowing one-side to go ahead and then decided to try and appear to be impartial by then dishing out fouls and cards against the winning team. In final analysis that too is used to justify impartiality of referees. The view being that I can’t have helped one team win because I booked them the most syndrome.

    The most telling statistic done in past years was the number of cards per fouls committed. The team that committed the least aggregate fouls in the season were ironically top of the league in getting cards per minimum fouls committed, i.e. it took teams like hull over 13 fouls to get a card and arsenal players only 6 fouls to get booked. This sort of statistic is what statisticians call significant. Had no bias been present it would have been expected that the cleanest team would at worst not head the league of cards per minimum fouls.

  32. Bias – weight or opinion that leans away from the norm in a abnormal fashion. That is my description.

    I have watched most of the games televised and many of the Arsenal home games live and I believe there is bias from most officials. The question is always – Is the bias unfairly influenced? In my opinion the answer is YES!!!

    Only the stats show some of it.

    @len you are so right in your view – the ref who knows time wasting is happening (usually with 20,000 fans screaming to draw his attention to it)ignores the card till ‘injury time’.

  33. Hey DogFace, great article. Any chance I can get a look at the data that you used for this? Let me know, I can give you more info if you need it. Shoot me an email.

Comments are closed.