The previous article in this series can be read here
By DogFace and Walter Broeckx
Untold Arsenal has a team of qualified referees who have reviewed more than 40% of the EPL games from last season. The reviews themselves were based on full match video footage with the advantage of video technology features such as slow motion and pause.
By reviewing those 155 games we have made a database of more than 7000 decisions that have been judged by our panel of dedicated and qualified referees.
The numbers you will see are based on those decisions and those reviewed games.
Is there any bias?
People who have been following football a bit know that at a certain moment during last season Mike Riley, head of the PGMOL came out saying : there is no bias amongst refs in the EPL. We just had to take his word for it. But even before Riley had spoken we were determined to see if there is a kind of bias. Or worse…are there a few different kinds of bias in the EPL?
And so, to attempt to establish this, we need to look at the numbers of the calls where refs didn’t follow the rules. There will of course be genuine mistakes – but these should ‘even out’; what we look at is the bigger picture and that will be shown as a conscious or unconscious inconsistency in the number of errors made… and you can see this in the next graphic as we take on the fact that pundits tell us that no form of bias exists and it all ‘evens out’ over the course of the season.
Summary of bias – all teams/all referees:
Again in this graphic the “medium” decisions= yellow cards, the “high” decisions= red cards, penalties and goals and the “low” decisions are the other decisions the little fouls somewhere on the pitch.
Our numbers clearly show that when you are playing at home you are likely to get some 10.077 weighted points value in your favour. This is the total decisions included the weight system we use to determine the weight to each decisions. This number is based on the errors a referee has made during the season for all home teams in those 155 games we reviewed. But when you are away from home you will only get some 7.458 wrong weighted points value in your favour – this gives us an overall average weighted Bias advantage of 2.619 weighted points per game – which translates as 1.826 incorrect decisions in your favour. This is also borne out in the average Points Per Game for Home and away teams with the home teams taking an average of 0.677 EPL points per game more that the away teams in the games we reviewed.
This means that there is a difference, when it comes to the ratio of referee mistakes, of around 57.468% in the home teams favour – this is a swing of 14.936%, in judgement, depending on being the home or away team.
I can imagine some of you saying: but this is normal. But I don’t think it is normal, and in this I agree with Mike Riley because he said that there is no bias so there should be no difference between playing at home or away. But there clearly is a difference as our numbers show. In a further stage we will try to find out if we can see patterns when we look at the individual teams and try to find out if some teams go over those numbers or some teams go below those numbers.
But what you surely must remember is that the ref clearly feels the pressure of the home crowd and that this pressure reflects in their decision making.
The next thing we can deduct from this is where the old platitude “it all evens out” comes from. Because over the course of an entire season for ALL teams it can be said statistically to “ALL EVENS OUT”. Of course it does, viewed as a whole this calculation is zero-sum; just take a look at the above graph at the Bias points for the ‘Combined’ totals.
Experts can look you in the eye and tell you it all evens out at the end of the day and this is true when you look at the league… but this may NOT be true when it is broken down regionally or, indeed, on a team-by-team or referee by referee basis.
To explain – whenever a referee makes a mistake this decision goes in favour of one team and there is always another team who is suffering from the decision. So if there are 100 wrong calls in a season there will be 100 teams being lucky and 100 being unlucky. And so you could say that it [luck] all evens out.
Of course when you are supporter of a team you don’t care about the global fact that it evens out. What you want to know is: does it even out for my team? Because if your team is involved in 100 wrong calls during a season and you are on the bad end 99 times you will not find it a big consolation that from a league point of view it evens out at the end. No you will and rightfully so, say that your team has been hard done by the refs. And that it is an utter disgrace. And you will not believe the pundits when they come up with the old apology: it evens out at the end of the season.
We at Untold look a little deeper than this and seek not only to scratch the surface of that platitude but also to smash through the shell and lay its contents bare… We will try to show with the numbers of each team; if this was the case for each team during the season. And you will be able to see how much the difference there is between the average home and away bias and the home and away bias of your own team. We will attempt to discover if the EPL egg is healthy or rotten.
In the next graphic we will give you an overview of how the incorrect calls have had their influence on the different teams. this is not about home or away, no this is about the total influence of incorrect calls on each team.
In this graphic you see the decisions in favour of a team are situated above the zero line. The decisions against a team are below the zero mark.
At the centre of the graphics is the zero-line. And if all the mistakes would even out as is being said by the pundits then all the teams would have the same height in the different incorrect calls. Above the zero line is the decisions in favour of the said team and below the zero line are the decisions going against said team.
It should have the same height in total and in the different types of decisions. Well it is clear to see that it doesn’t even out when you look at a team to team basic.
And we even see a few strange results in this but we will dig deeper in that when we talk of the individual teams. But just want to point at Stoke who seems to be a team that gets both hot and cold from the refs last season.
It is interesting to note that with the weighting applied (big calls worth more than small calls) the order of bias changes. Also it should be noted that teams such as QPR have only had 8 matches reviewed and, in their case, 2 of these matches involved a couple of real stinkers (or outliers) from Mike Dean Vs Arsenal – so their position in this table is perhaps not representative of their entire season.
We will focus on each individual team more in the following articles but if ever someone tells you that it does even out just link him to this article and he will have to admit that it does not even out. NOT on a team basis. And that team basis is the only important thing for any team supporter.
The final conclusion for this article: There is a home and away bias and we have this proven by the numbers. I hope Mike Riley reads it and admits being wrong when he said there is no bias in football.
The old platitude that “it evens out’ is not true. The only way it evens out is when you look at the league in a whole. So whenever someone tells you it evens out you ask him to prove it on a team to team basis. We proved it doesn’t even out with our numbers. They only have … just what do they have to prove what they say?
Next article in this series: Digging deeper : is there a regional bias to be found?
Walter, Dogface,
Now just to step back for a quick moment: Do I detect a coincidence in all this? Just to clarify (wink wink), would all this not be about those solemnly selected to referee for the Barclay’s Premier League? Couldn’t be the same Barclay’s whose ceo has just yesterday had to resign for his minions’ manipulating the Libor exchange rates, could it? No, couldn’t be. I mean, what’s the statistical likelihood of the ceo of said enterprise stepping down on the very day that UA launches its probe of the football monitors? Is the PGMOL boss next? What’s this world wide web coming to? Respect.
@Bob – lol, the front and back pages are but two sides of the same coin..
I was waiting until the end, to congratulate you guys on a job well done, but I am going to go ahead and say well done now as this is shaping up to be a great investigation which deserves wide recognition.
I am enjoying the journey of discovery too.
Keep it coming guys. Brilliant stuff.
http://www.debatabledecisions.com/tables
Luck or not.. Arsenal has never been favourable to ref… that’s why we dislike a lot of refs.
Amazing!
Can not wait for the rest of this. quite surprised by the chelsea scores above as well….not so surprised by the against call for the gunners….i’ll go ahead and say it now…disgrace.
Strange how even though Swansea are said to play a similar style to Arsenal, they have the fairest record in the above charts whilst Arsenal have the worst. This clearly points to it being not to do with our playing style being suceptible to bad decisions from refs but more a specific agenda to damage Arsenal.
Goona Gal,
come on you don’t tell me you were going to wait for another month or so to comment 😉
I’m working my way (behind the scenes) through the next articles about the regional bias… even for me this is a journey in to an undiscovered land. With thanks to Dogface for putting out the signs 😉
Oh and almost forgot… England is an amazing country to discover…
Stuart,
Kudos for pointing out that, given Swansea’s similar style, the disproportionate miscalls against us are mainly NOT about our “beautiful style”. Over the last two seasons hereabouts, there’s been several bouts in which sage comments would argue, what else do you expect given your possession-oriented game versus your opponents. Now, with these statistics as a reference point, counter-arguments like yours become possible and demonstrable and most welcome. Cheers.
It looks to me as if the teams are handicapped more or less – just like horse racing the lead pads filled according to the fitness of the beast to achieve the desired outcome.
Dogface and Walter,
Thanks for all this hard work, the graphics bring home the degree to which the Arsenal results are out of line compared with every other team and is fully in line with my perception of the games I saw at the Emirates over the last two seasons. The other stand out questions relate to Swansea where the results seem to be far more correct than for any other team and Fulham, QPR and Stoke where the numbers of favourable decisions is far higher than for everyone else. My gut feeling is that Swansea are a ‘clean team’ not committing many fouls, might the numbers indicate this as a factor for them, fewer fouls so less chance for any opportunity to alter results without being noticed? I await the further breakdown with eager anticipation.
To Amy Lawrence, Guardian Football Department,
Dear Amy: Now that Euro2012 is done, here’s a window of opportunity to introduce something (at minimum) both reasonable and “hot” into the realm of the Guardian football department’s purview. I mean it can’t be much compared to your colleagues’ stoking the rumor mill over the latest itch scratched (or not) by RvP and Theo; but it’s only about the very legitimacy of the pitch-minders of our favorite pastime. So, to complement Guardians’ wall-to-wall-coverage of the Barclay’s resignation, how about a wee taste of these, Untold Arsenal’s reports, about Barclay’s assignation of such refs across the league? You have a reputation as a fair broker of Arsenal’s journey, even if you think that 50/50 for or against Arsene/AFC is fair. But, in all seriousness, these reports will generate tons of readership of your article or blog column and hopefully tons of commentary. Even take a position that’s skeptical or dismissive. But as a reporter, a journalist, whose beat is Arsenal, these reports now exist, are popping up faster than whack-a-mole tactics will be able to handle them, and it’s your opportunity to be that fair broker by alerting your readership to their existence and whatever your fair reading of them happens to be. C’mon Amy. Strike a blow for fair and balanced, and take that motto back from Fox News and help, in your way(s), to shine a light for a fairer pitch for Arsenal; and, by extension and the intention of these reports, a fairer – if not totally fair – pitch for the EPL. I (and some here) look forward to a sign greater than silence or a single digit. Cheers, Amy, for your kind attention.
All – thanks for reading.
By the end of this series you will have a full grasp of the real state of play within the EPL… these first articles are merely an introduction – when we get to specifics you will be able to bookmark and use our articles as a reference for any given match-day.
Over the course of the next few seasons we aim to have full visibility and total awareness of actual team ability in the EPL and bias and competency within the PGMOL.
@ Walter – haha, exactly! Besides I am very grateful that you guys reaffirmed that my own two eyes are working perfectly fine regarding the bias against Arsenal.
Also you demonstrated through a great infograph that illustrates how Stoke have managed to remain in the league.
This is fascinating and what instantly came to my mind was how apt after just having our 125th season anniversary in which the Arsenal adopted the clubs first motto ‘Forward’. This must be the most forward and challenging undertaking by true fans of football anywhere on this planet. That it is being undertaken by Arsenal fans makes me even more prouder to be an Arsenal fan than before(believe me I did not think that that was possible). My only concern here is that the doubters of this(PGMOL,FA,Media blah blah) will still dismiss this because you have not covered every single game. I understand that your resources are limited and voluntary but unfortunately Dogface we won’t have a full grasp of the real state of play as the doubters will quote the above (yes I know you are at pains to make it clear that you have only covered 40% of the games and that is why sadly it is not a full grasp). Never the less what you are doing here is groundbreaking and hopefully more independent referees (not Arsenal fans) will become attracted to this and get involved. As for the media/football organisations reaction to this (towards what you are trying to achieve) is interesting in itself. I wouldn’t be surprised if it was a knee jerk reaction with all of a sudden decisions going more in our favour in the hope that what you are doing will go away. Now that you have started this I doubt very much that will happen. Brilliant stuff keep it coming
Great job lads…I cannot wait to read the rest of the series.
@Rob – As I clearly stated; a full visibility is our “aim” … “over the course of the next few seasons”.
40% is a representative league sample for now – we have covered a lot more than 40% of the ‘big 4’ (as stated in the previous article).
We are well equipped to defend our conclusions and we believe that any dismissal of our work would be suspicious – at the very least it demands a full and retrospective investigation by a fully impartial body if not by the FA but also from the Department for Culture Media and Sport.
Rog B
Although the PGMO could argue not every game has been covered, it can still be proven that things didn’t even out for Arsenal over the season. I think they’d be crazy to challenge the guys to cover every game and uncover further indescrepancies.
No no – Stuart, this is what we want to do – we just need more resources.
Exactly, I’m with you Dogface but I can’t see anyone at the top (publicly) taking it seriously I’m afraid. I can hear it now “It’s just some sore loser Arsenal fans who got a bit carried away” or something along those lines. Would be a shame though as they could gain some real credibility from ackowledging and acting upon the findings or even funding you to do it properly to show real transparency.
* When I say properly I mean all the games
I think its time to change the banner Tony, the “supporting Lord Wenger title” is, in my humble opinion, a little to brash considering where we will be going over the next few weeks.
Frigging great stuff @Dogface and @Walter.
Loving this site 🙂
Well, well, well! I was not expecting Fulham to be high up the favoured chart, even more than ManU. What do I know? The Arsenal position is not a surprise though. This is a real eye opener.
I’s amazing how much time you guys must have spent to do this. Viewing all the games, doing all the calculation and then write the articles and publish them here. Pure dedication! You have my full respect. I do hope that it will make a difference in the end.
Great job Walter and Dogface!
I was aware about bias against Arsenal but that it was so big (all wrong calls for us being less than big wrong calls against us). This just proves how great team we had last year to finish 3rd even with all those wrong calls.
The fact that Arsenal finished 3rd in the face of all this bias is an incredible accomplishment. A lesser team may have got relegated.
Walter, Dogface, Johnny Deigh,
Yep, these are the numbers (in yet another language!) that spell “The Great Turnaround,” yes, the hitherto Untold – and rather Now Told – story of 2011/2012. Now Told Arsenal. Has a bit of a lilt to it. Beautiful work, chaps.
Walter & Dogface
This is a fascinating start to the detailed analysis and does pose some questions which Riley would find difficult to answer.
With QPR doing so well from ref bias, questions need to be asked about the Tune Group sponsorship of the PGMOL – a conflict of interest? NB does the PGMOL publish detailed accounts? Why do Stoke do so well from the refs – is it to take attention away from MU?
With so many wrong decisions going against us the team did very well to finish third.
Walter et al – great work.
Lets not run away with ourselves here, facts are facts and unfortunately Untold is only representing 40% of them..@Stuart unless all of the EPL games have been covered how can it be clearly proven that things didn’t even out for Arsenal over the season? What this shows is that with 40% of all EPL games covered by Ref Watch it only ‘indicates’ that things don’t even out for Arsenal. Once all the games have been covered only then can an accurate analysis be made. After all the suspicion that things do not is I believe why Untold is undertaking this new and bold course of investigation. All that I am anticipating here is the media/football authorities dismissive response.
@Dogface ‘@Rob’ I assume you meant me? I agree with you entirely but I can see their response already. They have only covered 40% of the games,that is not even half blah blah. I might not sound like it but I am totally with Untold on this but if this were a test case in a court of law then I am afraid it would be thrown out due to a lack of evidence.. Sorry
Wow, Stokes weighted for calls is off the chart, Qpr have done nicely, I wonder why? Cannot wait for the breakdowns. Hope you can cover all EPL games in the coming season/s.
@Dogface, I reckon the betting firms just might buy you off, Or come after you. Either way good luck.
@Anatra I do truly believe this will make a difference in the end but it will probably take a couple of seasons with 100% factual analysis before the authorities grudgingly do something about it. Then again Rome wasn’t built in a day…
Rog b,
Hope you enjoy the after-life; but studies in real life as opposed to an abstract standard of 100 or it’s not worthwhile in the eyes of the authorities (the PGMOL? Barclay’s? the FA?) can have real impact before a jury that counts: among Arsenal fans who have been far from unanimous regarding what ails us and what is to be done; and in the court of public opinion among football fans, where making serious inroads toward fair play for all – which this process is about – can gain real traction. There is a real here and now pragmatic possibility that do not need to wait seasons until significant numbers of fans (of any club) consider this a serious work that bodes well for them and their side.
You protest otherwise, but, with all respect, please consider the downside to your standard of what counts – namely, nothing until that season when 100 percent of all games are covered. But what then to the argument: oh, well it’s only one season; can it be replicated. Or the argument: only a decade of 100 per cent coverage is worth anyone’s allegiance. Etc. Your timeline for validity is itself problematic; and the burden of proof is only you to argue exactly why WAIT X number of seasons to claim validity before the football authorities (is it?). Especially when it’s the competency (at minimum) of these very football authorities and their media toadies and strategic partners that are implicated by these gathering results.
sorry, Rog b: the next to last sentence above should read ^ “the burden of proof is on you”
p.s. And, to conclude: even with 100 percent times 10 seasons, today’s stewards of the former beautiful game won’t be impressed, proven-to, bow out because they are shown a countervailing truth to their litany of assurances. What matters is what fans like you and I think about what’s before us and what we then advocate.
@ Walter/ Dogface/ Ref review team –
If I may also say, just in case there are those who wish to bismirch your work carried out, in my understanding of the market research concept of ‘sampling’ 40% is a respectable figure to use to extrapolate and draw conclusions. In addition to this I would go further and arguably state that there is a high level of weight to the research due to the consistant scientific approach applied.
Lastly I would like to say that whilst we have to acknowledge that bias exsists, we can(well I can) place reasonable limitations on this as we have been given the opportunity to observe and challenge the primary research (ref reviews) which formed the basis of these findings throughout the year. The fact that the findings withstood the rigors of testing and scrutiny without losing it’s principal aim, must be respected.
‘it would not stand up in a court of law’ is a cheap throw away comment in my opinion.
…and it’s a good night from me…
@Rog B, as far as statistics goes 40% is a massive sample and for anyone to imply otherwise isn’t a valid argument and there is always an unlimited supply of nonsensical arguments you can use against anything, see Evolution denial/intelligent design for instance, so waiting to have 100% of all games reviewed would just make them choose a different illogical argument to ignore the findings.
Great work on the reviews everyone involved! One possible suggestion that I think would be interesting to see (if you weren’t planning on doing it already), would be a pie chart for each ref showing the teams that get the most incorrect decisions in their favor(%) from that ref and another pie chart for the teams most unfairly treated by the same ref.
I think it might be a good quick visual guide to show each refs favorite teams (useful for future pre-game reference).
These graphs tell a real true story. Its the tippy tappy teams ( teams that pass the ball) that seem to have less bias apart from the team that set the example – Arsenal.
The data that has been collected is from camera (broadcast) based evidence – games that have been recorded by camera. The truth is worse when judged from the live match. The cameras catch most aspects of the incidents but not all are shown (bringing another set of bias into play). Media bias has a very real part in these stats.
The reviewers being referees make the stats reasonable but there are those that will state the obvious – its an Arsenal bias that is being reflected.
As an Arsenal supporter I am biased, but that does not make me wrong when I call referees cheats. I do at the games where some of my fellow supporters feel I am being harsh. I believe ‘harsh’ is because there is some level of accuracy.
Many supporters want Wenger out and this or that player brought into the team. I always say what Arsenal need is a couple of unbiased referees. How many times has the Arsenal player been booked for telling the truth? I am sure if there were 10 unbiased referees in PGMOL the game would be magnificent. If there were more then even Syria might be at peace.
Finally – just for all those tippy tappy media people out there – did you watch how tippy tappy Spain won the World cup and the Euros? —- Just so you know, I think Arsenal play as good a game as Spain if not better.
Thanks for this article guys and your ongoing efforts.
Rog B,
if there is one team we can draw conclusions if it evens out then it is on Arsenal as we did all the Arsenal games.
@Bob @MK I believe you are absolutely right in what you are saying and what Untold is trying to do here, I am just trying to replicate how the doubters will reply to it thats all. These are not my opinions I am trying to put across purely how I imagine the ‘powers that be’ wiull respond to it. Sorry for not being a master of type, I think you have misunderstood me.
@ all that responded to my post You all seem to have taken what I posted as if I was saying that. In my reply I quote ‘all that I am anticipating here is the media/football authorities response’? Sorry but I can’t think of a clearer way of saying that.
Re The Stoke figures, I believe that is easy to explain, and it is nothing sinister. Stoke play with the intention of making a game as difficult to Referee as possible. They base their game not on technique, but on winning dead ball situations in certain positions, from where they can load the box with big men, and cause chaos. lets face it, it works, Its football, just not much loved outside the potteries.
The QPR bias could be a home bias, explained by the crowd being 2 foot from the pitch, and able to exert extra pressure on the Ref and Assistants.
Not scientific analysis like the great stuff above, but an observation.
Great work guys…really impressed not only with the heavy duty work in the background but also with the way it has been represented! Great analyical shown by the Untold team…their hatred for Arsenal stinks…these refs! Or, is it because unlike the Rednose we are not cry babies!!!
Sorry Rog – should have added a smiley face 🙂
Also – we have 100% of the Arsenal games – so yes, we can clearly prove that… we don’t have many QPR games so their position is less ceratain.
@MK – it’s in the pipeline mate.
@Rog B,
I did realise you meant it like that, so sorry if it sounded like I meant it that way or was having a go at you as it wasn’t intentional, i just meant that no matter what this site does, that type of person who doesn’t want to listen won’t and will make up some other excuse… sensible people will see it for what it is.
Cheers Dogface, looking forward to it!
Very interesting. Out of interest, are any of our ref reviewers non-Arsenal fans? I wonder what kind of bias we should acknowledge in the way an inherent Arsenal fan sub-consciously perceives a “fair” outcome to be? i.e. if this exercise were done by Chelsea-supporting refs, would there be much difference?
Also worth noting today, from Arseblog: http://deadspin.com/5923264/sorry-mario-balotelli-didnt-really-make-this-german-fan-cry-how-tv-lied-to-you-during-euro-2012
Demonstrates calculated manipulation of coverage… (granted, just to pull heart strings, but a worrying liberty to be taking nonetheless)
One for Anne there perhaps 😀
Rog b,
Alright, then. And now you’ve helped people start to frame some counter-arguments to the anticipated criticisms. You likely will recall that last May-June there were extended debates here on UA over these very points, many concerning Walter’s (excellent!) forthcoming end-of-season (2010-2011)review. Several commenters provided many of the likely counter-arguments, one thread of which launched the effort that is being presented today.
Walter, Dogface, UA
Based on those exchanges, I hope that UA could at some stage compile a list of top 10 objections/criticisms to come; and then knock each of them down, one by one, based on this new research. And/or perhaps issue an invitation to do the devil’s advocacy work that you’ve usefully begun here. Cheers.
Thank you Bob and all, No I missed Walters end of season review and am still to read it so I can’t comment there(my work/life commitments only allow for erratic binges on Untold unfortunately). My only objective was for Untold to anticipate the flak that I believe is forthcoming and by the barrage of flak I received myself (which contained both good counter argument and facts to back it up) I now feel fully satisfied that whatever is thrown at this ‘Testcase’ it will be swiftly countered with what are indeed the facts that you have presented. Great job,great comment discussion,great work.