By the Untold Team
Imagine a game of football between two fairly balanced, evenly matched teams. Each team commits some fouls, but neither team is much worse than the other. We’ll call the teams the Reds and the Blues
And let’s imagine (since this is all quite imaginary) that this game between these two teams is actually played out twice, each time with a different referee.
The first referee Mr Random, decides (for reasons only known to him) to give his decisions in a pre-arranged order. So the first time there is an event (a possible foul, a throw in, or anything else he has to rule on) he gives it to the Reds. The next time, he gives it to the Blues. And so on.
So the first issue might be a throw in, which should go to the Reds. But Mr Random has decided that he will give the first event to the Blues. Next up comes a goal kick to the Reds, and as per his agreement with himself Mr Random gives that to the Reds. So it goes on.
The fact is that Mr Random will end up with an accuracy level of about 50%, not because he is a good referee (obviously he isn’t) but because by pure chance half his decisions will be correct and half not.
Now consider the second game between Reds and Blues and again the two teams are balanced and equally fair minded. They commit some fouls, but in equal numbers, the get roughly the same number of throw ins, goal kicks etc.
But this time Mr Bent in the referee, and he wants Reds to win. So every throw in, goal kick etc that should go to the Reds, does go to the Reds. But every time the Blues get a decision, Mr Bent reverses it and gives it to Reds.
Since the teams are balanced on the pitch 50% of the decisions are again right (the decisions Reds should have had) but half are wrong (the decisions Blues should have had but were given to Reds).
Now my point is that both Mr Random and Mr Bent score 50% for accuracy, even though both of them are useless as refs and should never ever be allowed near a whistle let alone on a football pitch.
50% is the absolute bottom line, the level which a chimpanzee giving out decisions randomly would get.
We also know, that in an infinite universe, everything that can happen will happen, so one day Mr Random will indeed score 100% because by chance he will get every decision right even though he is not trying to achieve that. Also one day he will get every decision wrong, again by chance.
Moving into the real world, we know a perfect ref will get 100% of decisions right, but here again no one is perfect. But certainly we ought to be looking for 90% plus, and if that means we need to use video technology so be it. In essence, let’s use anything that makes the game more accurate, without actually changing the nature of the game itself.
Now take a look at the percentage marks that the referees are being given by our assessors. Our marking system is slightly more sophisticated because the most important decisions are weighted, but the basic point is the same: someone getting near 50% is no better than Mr Random or Mr Bent, or come to that a chimpanzee handing out decisions on a whim while trying to write the works of Shakespeare on a typewriter.
In fact anything below 75% starts looking exceedingly suspicious, once again because even handing out decisions on a random basis gets you 50% accuracy. You can only get below 50% if (for example) Reds and so much better than Blues and so most decisions (corners, penalties, goals etc) should be given to the Reds, but the ref is so biased that he is giving more and more decisions to the Blues.
Now as it happens, in Belgium there is a system of marking referees
- A score of +90% is an excellent performance.
- Between 80-90% is what should be expected at least of a good ref
- Between 70-80 means there are quite a few points for the referee to work on and he/she should be undertaking training with these points in mind
- Below 70% for a referee in the top division means possible demotion at the end of the season.
The referee committee in Belgium accepts that refs can have a bad day. And one score below 70% will not see a referee demoted.
But if you have two such games alarm bells are ringing of course. That could be an indication that something is wrong. Not good enough, biased, physically not fit enough…
We think this system is fair, and so since PGMO won’t act we will, and we are now going to start noting referees according to this system.
As an introduction to where we are going with this you might like to look at the list below. You will notice that one referee in one match was so awful he was actually below the level that would be achieved by a random referee.
| Referee | Match | 1st Half | 2nd Half | Full Match |
| Jon Moss | Arsenal vs. Stoke City | 93.7% | 70.5% | 81.1% |
| Norwich City vs. Arsenal | 69.5% | 60.0% | 65.1% | |
| Southampton vs. Arsenal | 63.6% | 55.8 | 58.9% | |
| Martin Atkinson | Arsenal vs. West Ham United | 65.2% | 52.6% | 59.5% |
| Arsenal vs. Tottenham | 40.9% | 59.5% | 53.1% | |
| Lee Mason | Crystal Palace vs. Arsenal | 77.2% | 64.0% | 70.2% |
| Arsenal vs. Everton | 63.1% | 63.3% | 63.2% | |
| Kevin Friend | Swansea City vs. Arsenal | 66.6% | 90.0% | 77.2% |
| Aston Villa vs. Arsenal | 68.4% | 100% | 79.3% | |
| Andre Marriner | Newcastle vs. Arsenal | 82.6% | 82.6% | 82.6% |
| Arsenal vs. Manchester City | 68.9% | 66.6% | 68.1% | |
| Anthony Taylor | Arsenal vs. Manchester United | 67.7% | 71.4% | 69.4% |
| Arsenal vs. Newcastle United | 78.5% | 44.4% | 54.0% | |
| Mike Jones | Watford vs. Arsenal | 50.0% | 88.8% | 71.4% |
| Liverpool vs. Arsenal | 56.2% | 60.0% | 57.8% | |
| Roger East | Arsenal vs. Bournemouth | 64.7% | 81.8% | 71.4% |
| Mike Dean | Chelsea vs. Arsenal | 38.6% | 66.6% | 50.0% |
| Michael Oliver | Arsenal vs. Liverpool | 72.2% | 75.0% | 71.1% |
| Craig Pawson | Leicester City vs. Arsenal | 75.0% | 90.0% | 81.8% |
| Mark Clattenburg | West Bromwich vs. Arsenal | 50.0% | 41.1 | 33.9% |
| Robert Madley | Arsenal vs. Sunderland | 88.2% | 100% | 93.1% |
Now just in case you are worried about those final ratings, here’s a word of explanation.
If in one half of a match a ref makes 10 decisions and 5 are wrong and 5 correct he will get 50% for that half. But if in the second half he makes 20 decisions and 5 are wrong and 15 correct he will get 75% correct for that half.
When we get to the total over the 90 minutes he then will get 20 correct and 10 incorrect decisions and a final score of 66% in total. And not the 62.5% one might expect if you just add the half time percentage scores.
So, to move on, who are we worried about?
Obviously Mark Clattenburg is in our red zone. He can thank his stars he is English and not Belgian, as he would almost certainly be removed from active service in Belgium with a score like 33.9%.
Mike Dean could readily be replaced by a robot handing out decisions on a random basis. The result would be pretty much the same, and we suspect in Belgium he would also be removed pretty quickly.
As for the rest, maybe very, very strong warnings. Indeed all those who have a red mark just in the percentage column really should be looked at very closely indeed.
Unfortunately, if all seven refs with a red mark were removed (and certainly some of us think there is a good case for that) PGMO would collapse because of its policy of strictly limiting the number of referees that there are available. We have always been very critical of that policy, because it means that a referee is liable to take control of matches involving a specific club far too often. Now we can see a second reason for not having so few referees. When you hit a crisis like this, there simply is no one else available.
- Six days to go and the Arsenal transfer machine is about to explode into life.
- When TV, radio and the newspapers invent a football story with no basis in fact, what’s happening?
More anniversaries
- 26 January 1891: Royal Arsenal 0 Everton 5. After three games in which Arsenal scored a total of 16 goals the club (which was playing its final season as amateurs) was brought down to earth by first division Everton.
- 26 January 1895: Crowd incidents in the Woolwich Arsenal v Burton game led to the ground being closed by the FA for two games. The original sentence proposedwas that the ground would be closed for the rest of the 1894/95 season. However, on appeal a “compromise” of 6 weeks suspension was agreed upon by the FA. See also here.
Dean and Clattenburg were the refs when Arsenal lost.
Is it any coincidence the gunners lose when the mo are below par?
I still believe in the conspiracy theory to stop the gunners winning the epl.
In 2013 the Premier League contributed 3.5 million pounds towards funding the PGMO. That figure is in the PL’s published accounts. I think the FA and the Football League provide the rest of the PGMO’s funding.
Domestic TV rights for the next three years were sold to Sky and BT for 5.14 billion pounds, and overseas rights will probably go for 2.9 billion.
Doubling – trebling – PGMO funding would be a drop in the bucket, and it would allow for:
1. Twice as many referees.
2. An independent review body of referee performance – along the Belgian and Untold models.
Why not?
Several points.
1. You have not included the bias numbers – which I know will support Wolfgang’s point.
2. What you don’t take account of is when a referee considers whether to blow for a foul but then waves play on – that is NOT included in the tables IF it was the correct decision. This would have the effect of inflating the negative percentages:
Gives a foul/was a foul: Correct mark
Gives a foul/was not a foul: Incorrect mark
Doesn’t give a foul/was a foul: Incorrect mark
Doesn’t give a foul/was not a foul: No impact on score.
I have a theory that this is how the PIGMOB come up with their ludicrous high-90s stats: Every time one player runs past another one, with no foul, the referee scores a point. Clearly, that will happen hundreds of times a game.
But as the PIGMOB are not open to scrutiny we just don’t know.
However, I would appreciate Walter’s view on how this is handled in Belgium?
Pete,
1. For the time being bias is not included. A data sheet with all kinds of facts and figures is in works.
2. Waving play on (Advantage) is part of each ref’s percentage, whether correct or incorrect.
Heres a bit of fun.
Let us sort by % correct based on whole game and then add a bit of geography:
NAME OF REF TOWN/COUNTY TEAMS COMPETING GAME WHOLE GAME
M Clattenburg Newcastle West Brom vs. Arsenal 33.90%
Mike Dean Liverpool Chelsea vs. Arsenal 50.00%
Martin Atkinson Yorkshire Arsenal vs. Tottenham 53.10%
Anthony Taylor Manchester Arsenal vs. Newcastle 54.00%
Mike Jones Chester Liverpool vs. Arsenal 57.80%
Jon Moss Yorkshire Southampton vs. Arsenal 58.90%
Martin Atkinson Yorkshire Arsenal vs. West Ham 59.50%
Lee Mason Manchester Arsenal vs. Everton 63.20%
Jon Moss Yorkshire Norwich vs. Arsenal 65.10%
Andre Marriner Birmingham Arsenal vs. Man City 68.10%
Anthony Taylor Manchester Arsenal vs. Man Utd 69.40%
Lee Mason Manchester Crystal P vs. Arsenal 70.20%
Michael Oliver Northumberland Arsenal vs. Liverpool 71.10%
Mike Mick Jones Manchester Watford vs. Arsenal 71.40%
Roger East Wiltshire Arsenal vs. Bournemouth 71.40%
Kevin Friends Leicester Swansea vs. Arsenal 77.20%
Kevin Friends Leicester Aston Villa vs. Arsenal 79.30%
Jon Moss Yorkshire Arsenal vs. Stoke City 81.10%
Craig Pawson Yorkshire Leicester vs. Arsenal 81.80%
Andre Marriner Birmingham Newcastle vs. Arsenal 82.60%
Robert Madley Yorkshire Arsenal vs. Sunderland 93.10%
Let us see if any of our matches in the above list had a home town referee:
Mike Jones Chester Liverpool vs. Arsenal 57.80% He lives in Chester (which is about 19 miles from Liverpool).
Anthony Taylor Manchester Arsenal vs. Man Utd 69.40% He Lives in Manchester
Mind you cuts the travel expenses down.
Not suggesting any cheating but just the local influences that may affect decision making. A concern shared by the former head of the select group of referees (PGMOL) Keith Hackett.
Interesting isn’t it.
Sorry about Untold messing up the formatting, looked great in Excel!
Anyone know how to resolve this?
Usama – but an advantage is when there WAS a foul but the referee allowed the play to continue rather than awarding a free kick as the outcome with playing on is superior to the offended-against team. The referee made the correct decision – there was a foul, just not awarded…..
But if there was no foul then the default is for play to continue. The point I’m making is that the referee momentarily considered whether there was a foul, decided there wasn’t, so did nothing (a bad ref might have decided to award a foul).
The system we are using does not record these “correct non-decisions”…
Kev
That is my territory (the data I am still playing with).
The formatting problem you are seeing is a combination of two things. One is that wordpress eats spaces. It looks like it only allows a single inter-word space for all occasions, not even a double space after the end of a sentence. The second is variable width fonts versus fixed width fonts.
Pete,
I understand the point you make about non-decisions. The system we use does not includes the “correct non-decisions” because these are part of regular football play. So if nothing worth making a decision has happened in the field of play, then the referee should not do nothing and just carry on with the play.
For example a player sticks out a foot and intercepts the ball first, then there is no need for the referee to make, all he has to do carry on.
But if for a clean and normal football play the stops the match in favor of any team then he has made a mistake and thus an incorrect decision.
Pete,
Walter or Don can explain the non-decisions much better, because I believe they are not part of a referee’s percentage report, they are part of a referee’s ability to control the match and keeping the spirit of clean & fair play in flow.
As written, this thread is relying on binomial probabilities. Yes Pete, there are always the false positive and false negatives in any detection problem.
The variance of a binomial probability is Np(1-p). For a probability of 50% (flipping fair coins), the distribution is symmetric and p=1-p. Hence, the variance is N/4, and the standard deviation is 0.5*sqrt(N). If N is large enough, we expect about 67% of events to occur within 1 standard deviation of the mean. What’s large enough? Good question, it depends how fussy you are.
If I suggest 16 (a perfect square), our standard deviation would be 2. We are expecting 8 events detected (out of 16), but we are reasonably certain that numbers between 6 and 10 could also happen.
If I suggest 36 (to avoid talking about observing half events), our standard deviation is 3, and our reasonable range is from 15 to 21 (out of 36). 15 out of 36 is about 42%, which is still significantly higher than the low of 33.9% for Clattenburg above. Well, if we go to 2 sigma confidence instead of 1 sigma, that is detecting 12 our of 36, which is 33%. Close to what Clattenburg had for a score.
This is really sad. It is sort of saying, that at best on that day, Clattenburg was no better than flipping a coin, and having a bad day on top of that.
For Clattenburg to score that low (by random chance), when his accuracy is actually something like 90% is highly unlikely. And 90% is what his score should be, if he is one of the best referees in England. It is more likely that there is some other explanation for such a poor score. Maybe his cat died? Maybe he is diabetic and he was in need of insulin?
Where there’s betting, there’s corruption. Where there’s power, secrecy and no accountability, corruption lurks.
It’s long past time that the PGMO was subject to continuous and independent review for the good of the game.
Unfortunately, that won’t happen because it’s in the commercial interests of both the Premier League and Sky/BT to pretend that their “product” – such an ugly word – is squeaky clean, untarnished.
Change will only happen when a whistle-blower steps up.
Pigmob=bent
Usama – thanks.
I agree that it is very hard to account for correctly allowing the play to continue if there wasn’t a foul. But, from the point of view of Mr Random in the article, he may give fouls when there wasn’t a foul. Every time he makes a mental decision. From the reviews, the number of incorrectly awarded fouls is very low – the bias is far more towards not awarding legitimate fouls.
Perhaps if we were in Italy (or Belgium?!) there would be many more incorrectly awarded fouls?
Anyway, my point is that these stats exclude incidences where a referee has considered whether or not to award a foul (and there is no easy way to track this) and has correctly decided not to. Therefore 50% is not quite as bad as if the referee was truly random.
Pete
There are also likely to be instances where the referee never knew that a foul had taken place.
Your position is supposed to be (partially) covered by the “advantage”. If the referee sees a foul and decides not to call it, he/she should indicate “advantage”. By doing so, the referee informs the players (and the viewing public) that a foul was seen, but not (immediately) acted upon. In some circumstances, at the next stoppage in play, the referee can come back to act on the foul that was “ignored”.
But this business of calling/not calling, detecting/not detecting, inventing/not inventing fouls is complex.
One idea I wondered about, is to have a number of referees at the game in the stands (preferably near ground level at the half way line) with something that indicates which way the head is pointing (or better yet where the eyes are pointing) and a “lie detector” attached to them. Note when the referee is “reacting” to something, and in which direction they are looking.
I’m surprised that Taylor actually got an acceptable score as he’s a completely incompetent Official, but I guess that just proves that even the worst can get luck sometimes… I think I’ll buy a Lotto this week!
Gord thanks.
It is bloody annoying because you lose the impact of the point you are trying to demonstrate.
Great article today, it highlights just how much we need to ‘help’ the referees. The advantage with having a video is that it reduces the amout of influence a referee can have on results. Tales seconds. There will still be contensious issues, which is fine but it may stop the possibility of home team bias, whether subliminal or deliberate, on the big decisons like penalties etc.
Personally, I find it hard to believe refereees would deliberately cheat but I do believe there is a North v South bias. I also believe the rules interpreted by the FA and in turn PGMOl are a bit prehistoric – by this I mean we seem to allow a more robust and physical game then they do on the continent.
Referees are no different from most fans; they listen and read the crap generated in the media and the opinions of the plethora of ex players (who arent managers for a reason) so they are influenced. The anti-Arsenal media (no I am not paranoid it has been going on for years) and shows like the Talksport ones, don’t do us any favours because it paints a picture that we are a bunch of softies and kicking us is a legitimate tactic. Said this for years.
Anyway, my tuppeneth – great article today, I hope someone in the mainstream picks up on it(not holding my breath!).
*contentious
Correction:
The only time I thought our game could be bent was the Riley match at Old Trafford that ended our unbeaten run. I have never seen such biased officiating, some of those decisons beggar belief! What was the Untold analysis of that game, must be lower than 30% surely? And Riley’s head of the referees too now..!!
Walter,
Just to know professional competency of English Refree’s , it would be great to know how they fared in Champion’s League and International matches they reffered.
Lalit,
On more than one occasion I have watched English referees do international matches… and… sometimes they seem to be different persons. Much stricter, much more in line with the FIFA guidelines than when they do PL matches. Mind you I have seen them make mistakes but they will never allow in those matches what they allow in PL matches.
That is because they know that the assessors in those matches will judge them differently than PGMO assessors do and they adapt their refereeing to that.
Lalit,
but to be honest I have never judged them in those matches like we do the Arsenal matches. Would be an interesting experiment come to think of it.
Walter – So what you are implying is that English domestic referees are deliberately instructed to referee in a way that conflicts with the Laws of the Game and their standardised international interpretations – rather than some inbuilt subconscious fondness for the “anything goes” English tradition?
Lalit/Walter/Pete
I have always thought, the odd calamity accepted, that our referees perform pretty well in Europe and on International duty.
That is why the way they referee on the domestic stage is so infuriating.
That is why I am convinced that the diabolical way in which they referee Arsenal in particular, must be the result of something ‘other’ than mere incompetence.
So what is the ‘other’ ?
Well, personally I am in the conspiracy camp.
But why a conspiracy? Why against Arsenal in particular?
I’m sure I don’t know, but there’s been many a theory muted on here, ranging from the first words spoken by Wenger on the steps of Highbury when questioned about those ‘rumours’, to the way we run our Club, ie not bankrolled, to the simple fact Wenger is French.
I don’t know, but I’m convinced the Refereeing bias is there, and the excellent, and yet to be convincingly challenged or disproved Reviews (I know what I see with my own eyes doesn’t cut it by the way) supports that view.
The bottom line is I concur entirely with the notion that:
“English domestic referees are deliberately instructed to referee in a way that conflicts with the Laws of the Game and their standardised international interpretations”
But why particularly against Arsenal?
I cant pretend to know for sure. But anyone who has had the misfortune to read my consistent rants against the media will know exactly where I think the blame lies.
And what a great idea to review our referees when performing out of the premier league.
Personally I think they will score pretty well, after all, I know what I see with my own eyes 😉
Jambug
I think your assessment, as explained above and on many other occasions by your good self, is absolutely plausible, and in my opinion correct.
Without doing any analysis, I would guess the PGMO ref in CL would be half decent . It is the EPL that suffers from specialist vision & incorrect decisions.
The interesting area would be the financial accounting of PGMO personal/related accounts. More chance of flying pigs than a fraud investigation of football in England.
Pete,
I have seen Dean do amazingly good matches in Europe. Believe it or not. 🙂 And then he comes to Arsenal the next week and screws us over with a performance that looks completely different.
I don’t know why the change of course.
But the fact that the FA and the PGMO only take refs from the North to do matches in the Pl as they do their PL matches in a Northern style speaks volumes for me. And as a non Englishman I didn’t even know that there was such a difference between the playing style between North and South. Let alone a difference in refereeing style. Keith Hackett confirmed this difference, Untold only reported that he mentioned that difference. So I don’t take any credit for that.
I’m surprised it’s only 7! Perhaps the problem is, you get what you paid for? The Premier League refs get a nice Retainer fee around £38,000 – £42,000, but only get £1,170 per match.
There should have been more complaints for an even distribution of matches from the refs themselves! Atkinson has done 22 matches, where 6 Select Group referees (East, Marriner, Friend, Mason, Swarbrick, & Scott) have refereed fewer than the 15-16 which would be expected if exactly distributed equally.
A couple matches also done by the National Group (Attwell, Tierney, Stroud, & Hooper)
For comparison, a La Liga ref is paid £4,700 per match (no retainer fee). Last year, with a smaller variance between matches officiated by refs, the lowest number officiated by their Select Group was 16 matches = £75,200.
Premier League referees would still have to officiate 28 games to get the equivalent amount even with the highest reported retainer fee (42,000)!
Last year, a Spanish ref that did the fewest matches earned more salary than all but 4 PL refs (Atkinson, Moss, Dean, Taylor). The assistant refs get paid about £1000 per match also.
The solutions:
1) Get rid of the retainer fee
2) Pay a much higher Referee match fee similar to other top leagues
3) Pay much higher Assistant Referee match fee similar to other top league
4) Distribute matches evenly between all Select Group officials
The Salary comparison from Total Sportek:
League Referee Match Fee, Assistant Referee Match Fee
PL: £1,170 / £475
Spain: £4,700/ £1,550
Germany: £2,800/ £1,400
Italy: £2,819/ £1,300
Well, if we can’t have consecutive spaces, let’s see if substituting them with another character works. Plus, the fixed font issue. The only problem here would be that the text is slightly more difficult to read, but not overly. Formatting exercise:
M Clattenburg --- Newcastle ------ West Brom vs. Arsenal --- 33.90%
Mike Dean ------- Liverpool ------ Chelsea vs. Arsenal ----- 50.00%
Martin Atkinson - Yorkshire ------ Arsenal vs. Tottenham --- 53.10%
Anthony Taylor -- Manchester ----- Arsenal vs. Newcastle --- 54.00%
Mike Jones ------ Chester -------- Liverpool vs. Arsenal --- 57.80%
Jon Moss -------- Yorkshire ------ Southampton vs. Arsenal - 58.90%
Martin Atkinson - Yorkshire ------ Arsenal vs. West Ham ---- 59.50%
Lee Mason ------- Manchester ----- Arsenal vs. Everton ----- 63.20%
Jon Moss -------- Yorkshire ------ Norwich vs. Arsenal ----- 65.10%
Andre Marriner -- Birmingham ----- Arsenal vs. Man City ---- 68.10%
Anthony Taylor -- Manchester ----- Arsenal vs. Man Utd ----- 69.40%
Lee Mason ------- Manchester ----- Crystal P vs. Arsenal --- 70.20%
Michael Oliver -- Northumberland - Arsenal vs. Liverpool --- 71.10%
Mike Jones ------ Manchester ----- Watford vs. Arsenal ----- 71.40%
Roger East ------ Wiltshire ------ Arsenal vs. Bournemouth - 71.40%
Kevin Friend ---- Leicester ------ Swansea vs. Arsenal ----- 77.20%
Kevin Friend ---- Leicester ------ Aston Villa vs. Arsenal - 79.30%
Jon Moss -------- Yorkshire ------ Arsenal vs. Stoke City -- 81.10%
Craig Pawson ---- Yorkshire ------ Leicester vs. Arsenal --- 81.80%
Andre Marriner -- Birmingham ----- Newcastle vs. Arsenal --- 82.60%
Robert Madley --- Yorkshire ------ Arsenal vs. Sunderland -- 93.10%