How many wrong calls does each ref make per game – and in favour of whom?

Posted on by WalterBroeckx

By Walter Broeckx

In the last few days and weeks we have had visits from supporters of other teams following our coverage of the referee in matches involving them, and I would like to add a bit to the debate of the ‘it evens out at the end of the season”

I have taken the numbers from all the games we reviewed so far so we can see how things have evened out for each team.

So the first thing I have done is to see how many mistakes each ref makes in a game. And I put this in a table that you will see below.

I have taken the total number of games for each ref, counted the wrong calls they have made in those games and then taken the average total of wrong calls for each game.   In the final column you will find the number that is acceptable for each ref.

Ref Games Total wrong calls Average wrong calls/game Averave wrong calls/team
Walton 6 98 16 8
Atkinson 8 77 10 5
Dean 12 166 14 7
Dowd 11 83 8 4
Halsey 6 96 16 8
Jones 7 73 12 6
Mason 4 70 18 9
Probert 6 85 14 7
Webb 13 182 14 7
Atwell 3 18 9 5
Clattenburg 8 76 10 5
Marinner 6 82 14 7
Friend 5 50 10 5
Taylor 3 40 13 7
Foy 4 68 17 9
Oliver 7 50 7 4
Moss 1 4 4 2
Swarbrick 1 9 9 5
Total 111 1327 214 107
Acceptable wrong calls for each team 6

That last column is the average wrong calls divided by 2. Now I must point out that when I use the word “acceptable” this doesn’t mean that we just should accept it. No, I presume that they just are not capable of doing any better and that all those refs are as unbiased as a new born baby.

So based on these numbers any team can expect to be at the wrong end of some 6 decisions in each game. Around 12 mistakes in total for each game and 6 for each team. Because according to what the media wants us to believe: it all evens out.

Now I could go in to detail with each ref but let me say that Dowd and Oliver are the refs so far this season with not many wrong calls. I know Moss made the fewest mistakes so far but this was only based on 1 game so I will not count him. As I will also not really count Swarbrick when I talk about the individual numbers of the refs.

At the other end we find Foy and Mason with nine mistakes you can expect in each game going against you. Nine mistakes against each team will mean that they will make around 18 wrong calls in a game. And I really don’t think this is even close to being called acceptable.

But now I want to show you how this number of six mistakes against a team can pass the reality check with each team.

So a team that gets six decisions going against them can complain about a poor standard in refereeing. But it is no indication of bias against that team. The bias only becomes visible when you have more than those six calls going against you on average. On the other hand a team that has less than six wrong calls going against them can be very happy with the ref in general. I know that if they only get one wrong call against them but it was a penalty or a goal they will be unhappy. But then they have to console themselves with the thought that it will even out.

So here we give you the numbers of the decisions in real numbers against each team on average. And remember 6 is the average wrong decision going against you.

Team Games Numbers of wrong calls against each team Average “Normal” wrong calls against each team Bias or reality check
Ars 31 366 12 6 +6
MC 23 110 4,8 6 -1.2
Che 24 158 6,6 6 +0.6
MU 23 104 4,5 6 -1.5
Liv 17 78 4,6 6 -1.4
Tot 14 73 5,2 6 -0.8
New 7 44 6,3 6 +0.3
Swa 4 9 2,3 6 -3.7
Bla 7 32 4,6 6 -1.4
Bol 9 31 3,4 6 -2.6
Sun 7 30 4,3 6 -1.7
Sto 7 35 5 6 -1
Wig 5 20 4 6 -2
Ful 8 49 6,1 6 +0.1
Eve 9 64 7,1 6 +1.1
Nor 7 33 4,7 6 -1.3
WBA 5 42 8,4 6 +2.4
AVI 5 18 3,6 6 -2.4
QPR 5 24 4,8 6 -1.2
Wol 5 29 5,8 6 -0.2
TOTAL 222 1349 6

So in this table having a + in front of your number in the last column is a bad thing. To start with another team:  West Bromwich Albion gets on average not the normal 6 calls against them but they get 8,4 calls against them when we reviewed them.

And it is not really the biggest teams that have the best numbers when you look at this table. Swansea is having the best return when you look at mistakes made. They are not even getting half of the wrong calls against them that they should expect. And with all respect I think Swansea can be called a small team and so one would expect them to be on the receiving end of those things. Bolton is also a team that gets a lot of wrong calls in their favour. As is Aston Villa. About Swansea we might just add that they sometimes are called a team that is playing some kind of Arsenal style of football. Based on possession and passing the ball around. Just remember this for later on in the article.

Manchester United is a team that is on the good side of the wrong calls also. As is Liverpool and Manchester City when we look at the top teams in the league table so far. The only team that has come really close to numbers that even out is Fulham so far.

And at the wrong end of the mistakes by the refs we see that apart from WBA there is also a bit of work to do for Everton.

And when we compare all this to the numbers of Arsenal it is staggering to see how many mistakes are made by the refs against Arsenal. We come to double the numbers that are “acceptable”. One could say that for any mistake that is made in our favour we will pay by getting it back double.

And then I come back to the Swansea – Arsenal comparison. Both teams with a similar style of play. And yet such a contrast in the numbers of mistakes. And if you might expect the big team to be getting the favours it is the contrary in fact.

BUT let me finish this article with some good news. The PGMOL, SAF, SKY, the BBC, the other media they all have been telling us the last week that at the end of the season : “it will even out”.

So from now on to even things out we will have in the 7 remaining fixtures some 245  decisions going in our favour. That would mean some 35 mistakes in each game going our way. The other teams we have to face and who pretend it evens out will have the usual 6 in their favour.

In our remaining home fixtures we will have 8 penalties given to us, imaginary or not because that is what should happen when things even out. And I don’t take in account the ones that are given to some of our rivals so any penalty given to them should result in one extra bonus penalty for Arsenal.

So I hope Mike Riley is reading this and will instruct his refs to start the big even out campaign that he has promised us a while ago. We never claimed things would even out. But with all of the rest claiming it does, we will be treated very nice in the next 7 games. After all Mike Riley made us the promise himself. Didn’t he?

————————-

Arsène unedited:  on the subject of refs, plus trafficking of child footballers

Ref Reviews: Man City 2 Bolton 0   Wolverhampton 0 Man U 5

Match officials: Attacks upon linesmen and the professionalisation of refs

Corruption: A beacon of light in the world of match fixing and money laundering

Youth: Under 19 version of the Champions League?

—————————————-

On the Arsenal History site: the Bertie Mee decade

Peter Simpson: one club three managers

Recent Posts

78 Replies to “How many wrong calls does each ref make per game – and in favour of whom?”

  1. Quite brilliant Walter. If that does not shut people up then I don’t know what will. Thanks.

  2. well statistically speaking lower teams get the advantage but hehe why are we on top of the table for the team who doesn’t benefit from it ? 🙁 why do they hate us ?
    1- Arsenal 366 wrong calls
    lol we r TOP TOP of the list 😀 thank you Walter for the statistic pleaseeeeeeeeeeeee send this to the FA and let them check the Genius Howard Webb 😀

  3. Who totted up this information and is this based up every game in the league this season? If i pretend to be a cynical fan of a rival team wouldnt i be able to say “well a gooner would say that?”

  4. Fascinating article and I notice that for whatever reason, we seem to be a side, this year, who dispute decisions the least, but why are there 1327 mistakes by referees and 1349 against the teams where do this extra 22 come from?

  5. CrouchEndGooner,

    This was done by our ref reviewers who are also refs.

    And of course you can always check the debatable decisions website and if you look and compare both tables you will find a lot of similarities between what we found over here in over 110 games we have reviewed in the EPL so far and what they have found.

    And the folks from the debatable decisions website are getting pissed off when they are called arsenal supporters I think 😉 Because they are not. As far as I have checked not one of their panel is an Arsenal supporter.

    And yet their numbers indicate the same as ours do.

  6. Walking wounded,

    thanks for pointing this out. I noticed now that I had one game of one ref not counted in the tables of the referees.

  7. I think it’s time to have more scope put on the Manc’s, what you reckon people?

  8. We see the world not as it is but as we are.

    This maxim is true in all facets of life and as a Gooner I feel that there is bias against us and some other teams.
    But I wonder if how impartial anyone can be when they want to demonstrate a hypothesus they believe is true. Saying that I feel the dominant narrative of the Premier League since its inception is the key influence that filters down to all structures and rituals that we choose to ignore. Ref bias is just one of them.

  9. I don’t believe we can complain about not getting a fair split on the so called 50/50 calls,but so many blatantly obvious errors against Arsenal,its just not funny.
    The first QPR goal was easily and clearly offside…..few picked it up at the time,but it was…..and the critical errors are those the officials get us on.

  10. Perhaps what they meant by evening out is: a bad call in favour of Man United equals a bad call against another team. That balances the statistics.

  11. The match referees headed by Mike Riley and ably assisted by Howard Webb have really played a big part in Arsenal”s trophy drought!

  12. Did anyone realise to get arsenal’s numbers you have to add chelsea, MU and MC total and still have 4 points over. It’s a shame the way we get cheated out of points. I keep on saying we would not be fighting for 3 rd place had the referees done right by us. But the great thing about me I never give up on my team even when time is running out.

  13. Hi, grettings form Israel! been reading your stuff for a long time and am a big fan. You guys are doing wonderful work. keep it up. I often link to you in the foroum I write in and you are in the center of some heated debates let me tell you!

    That said, I have to say that for all your great data colection work, i see very little in the way of real statistics, meaning using statistics in a convincing way. I am no statistician(?!), but even i see that some tests can easly confirm or debase your argumaents. here for example, why don’t you do SD and t test it for siginfince? see if getting -1.5 on average is really statisticaly significant to say that it is lower then the average? some t test, ANOVA work, pearson… anything.

    Again, i am not an expret, but i know you can get a lot more out of this (and other) data both in the validity of it, and even new insight. for a site so heavy on data and the use of it, your statistical work is really poor (sorry). if you intend ppl to take you seriously, this is a must! mabye get some pro to do it?

    love your work. only giving my 2pens really (sorry for the spelling.

  14. @little me – we will be doing this at the end of this season on the entire dataset for all games reviewed. This is the first season that we will have reviews for non Arsenal games also – so we have some context to our stats.

  15. Well this comes off as biased even if it does tally with the debatable decisions web site. What I would love to see at the end of the season is a wrap up like this one, but where you also compare your findings with outside sources… A sort of “but don’t take our word for it, here is a similar analysis based on xyz.com’s data”. And it might help if you describe your methodology in the write up, as you did in the comment above, explaining that games are reviewed by other refs, revealing bias where it is or might be, etc)

    Anyway this is great piece. Fascinating.

  16. Much as though I really enjoy your website, even this committed Arsenal fan is holding his head at the sheer amateurism* and bias of this.

    Bias? Well yes. Flags should be waving frenetically at the conclusion of your analysis: an Arsenal website, run by Arsenal supporters (obviously), concluding that Arsenal get the roughest deal in the Premiership when it comes to refereeing? And by such a humongous margin.

    That’s not to dismiss the hard work and endeavour …

    I could tell you how you can get a more accurate and more objective analysis executed, but it would take more time than I’m willing to spend on the matter. More’s the pity, eh?

    On the other hand, you can always fall back to the position that philosophically you may well still be right (or at least ‘not wrong’). And what’s more, in an infinite universe you already are (woo hoo!). So well done you.

    But in this ‘here and now’ … being a not particularly convicted advocate of ‘us’ and ‘them’ you’re going to have to excuse me this time.

    *amateurism – not necessarily a bad thing. Often a good thing. Occasionally a great and wonderful thing.

  17. And, i wnated to add that every thing makes sense if you tink about it like this:
    What would you think happen if a ref from italy is in charge of a match between lyon and Inter? he can’t be fair no matter what.

    That is the case for every game Arsenal playes in England. we are considered foreigan by refs and every game is like beeing reffed by a ref from the other team’s country.

  18. Congratulations to the Untold Team for identifying the refereeing bias within the EPL.
    Is it possible to provide speculation/informed opinion on the underlying reasons for teams being on the positive/negative side of this bias?
    Is it money – and if so who are the donors/recipiants?
    Is it personal animosity? And if so what were the origins of the animosity and participants? e.g. Do the statistics for a particular team change with a change of manager? Does the bias follow a manager around? Do some teams fare better/worse when a particular player is playing? Please can you add MOTIVE to your analysis?

  20. I ve followed your site starting from when walter went to spain to read papers about the sell of Fabregas. The least I can say is that I am impressed. However, I ve ask myself this question; can a game where people are interested in the result ever be fare? Maybe as an Arsenal, we may complain but will Man u fan ever agree with us? Could we ve agreed with other fans if the bias was in our favour. Could it be that this same bias assisted us to go un-beating? Credit to the invincibles but there were many English men in that team. Finally, why did the Arsenal English men Cole, Sol, etc chose to leave Arsenal and it seem no ready made wanted to come eg joe cole, parker, barton, hart were all romoured but none came. I think the reason why English players started despising Arsenal is the same their ref hold for the bias. Wenger was left with only the choice of picking the under-ages that have not been infiltrated with the arsenal hate dat’s why we had Walcot, Gibbs, AOC, Wilshere, Ramsy, He…

  21. Are the Arsenal reviews conducted by a neutral reviewer? Or by Walter? While I agree that there is a bias against Arsenal I find this to be almost too much! If there is no chance of reviewer bias this is truly astounding….

  22. So funny, when people don’t like the evidence, they attack the investigator. I don’t like the results of your investigation, you must be an amateur. I can do much better than you, but I am not bothered to waste my time. (Yeah right) You have not proven motive, therefore your conclusions must be wrong. Never mind that I cannot dispute the evidence, I just don’t like it, therefore it must be wrong. I am a United supporter, so you must be talking nonsense. The world is full of intelligent people.

  23. The Arsenal reviews are done by refs who also support Arsenal.
    We have been asking other refs to be involved but no other refs from other teams joined our group. Unfortunately I would say. My wet dream for next year. 😉

    That is why I think it is important that there are other websites as I have mentioned before who do a work that is a bit similar to what we do but in a slightly different way. And who are not Arsenal supporters.

  24. In a way it is good that people ask us questions about possible bias. Because in fact we ask the same questions about the refs on the field. They should be impartial and above any doubt.

    Our team of ref reviewers doesn’t hide from their bias.
    Our team of ref reviewers analyses each game and gives the people who doubt it any chance of being critical about the review as we publish it in full.

    So if a ref reviewer would step out of line it can be corrected by anyone by giving his comment.

    But in the next days we will be adding more to this. In words and in numbers. Numbers not from Untold arsenal refs reviews by the way 😉

  25. Little me and Dsch – how many times should Walter refer u or give u links to “non-Arsenal” sites with findings which are similar to those of Untold?

  26. If I may add a last comment about people who criticise what we do.

    Last year I did all the reviews on my own and I only took the Arsenal games. Not enough, not good enough to draw any conclusions.

    Thanks to 2 other refs we have expanded our team to 3 reviewers. And we do more non Arsenal games than Arsenal games. So roughly around 70% of the numbers we present do not involve Arsenal.

    and still people come and say: not enough, need to do more.

    and in a way they are right. It will never be enough. It will be enough when we have 20 ref reviewers coming from at least one PL club each.

    So no one better than me who realises that this is not perfect. But we have to work with the meanings we have.

    I would ask the people who criticise us to work together with us tu make it better. And I don’t mean you have to join the review team if you are not a ref. No, I mean ask around if you know any refs (doesn’t matter which team they support) who are willing to spend their time on a project like this.
    It doesn’t earn you something and you will be abused at times by people who pass around for the first time and call you names but I can assure you that it is very interesting to review the games like we do and you can learn something from the reviews you do as a ref. I still learn from them in fact.

    And I would also like to add that it is amazing to do a review. Because I can assure you that after 5 minutes doing a game and concentrating on the ref his decisions and nothing else makes you forget which teams are actually playing.

  27. A perfect example of an Arsenal supporting ref reviewer is the Stoke – Tottenham game I would say. Look it up and see how an Arsenal supporting ref holds nothing back to show how Tottenham has been robbed in that game.

  28. It’s just not the case of the men in black, anyone looking at the bigger picture can see the FA working it’s biased magic. Last season Rooney deliberately elbowed the Wigan player, now look what happened to V.Kompany and even our own Gervinho. The FA stated that the ref handled the Rooney incident accordingly but Kompany/Gervinho felt the full letter of the law, even to the point of Barton was given a full pardon.

    This season Man C. have been stitched up since Dec/Jan, with the a real possibility of going 8 points infront are now 5 points adrift. How can it be every season Utd get a lift from the powers that be?

  29. Whilst I have no doubt there is a clear bias against Arsenal, I’m not sure how much can be read into these figures. The problem I have is you’re not comparing apples with apples i.e. some teams have been reviewed 5/7 times, whereas the big team have been reviewed 20+ times. Therefore the teams which had had fewer reviews cannot be fairly compared to the teams with 20+. If you pick 5 Arsenal games in which we haven’t been totally screwed, I’m pretty confident you could get our score down in the minuses. What you’re basically asking us to do is assume in the 15 games you haven’t reviewed the results will be the same. In addition, the likelihood is those 5 games have been against the big teams, not against their table position rivals or if these games are home or away. On top of that, excluding Arsenal, if you take an average of the bias, it pretty ends up at .1, which to be honest adds a lot of credibility to the argument that it does all even out in the end…
    I truly respect what you guys are doing here and read every review with interest, I just don’t think drawing conclusions based on incomplete data is a great idea.

  30. keep up the good work. fingers crossed you’ll get at least 3 non arsenal reviewers for next season. if ppl cannot see that this is the prep before the quest for the holy vid tech grail, then i feel sorry for them as they would rather act like a crab in a bucket than see equality in the game.

    try speaking to a manure fan who has watched a majority of games over a number of years but does not live in the city about refs. their go to reaction is denial understandably. but if your on point with your own knowledge you will see how they eventually concede.

    the same with the “arsenal bias” i dont need to be a ref to know that a kung fu flying kick to the chest is at the very least a yellow. or that booking song in the first 10 mins for virtually nothing is gonna tilt the game etc…

  31. Attacking Walter for nor “proving” bias is a red herring. Walter has demonstrated bias. If you want conclusive proof, you need a whole set of scientific criteria to define what would be admissable as conclusive proof . The conclusive proof argument is a straw man.

  32. plus on the rare occasion when arsenal get a decision in their favour unintentionally or otherwise, i would rather it not be the decider. and would hope that it would even out be it a loss or draw. it would be hard to take at the time but more acceptable than absurd decisions big and small that decide matches.

    its hard to imagine any fan not wanting vid tech used in real time.
    reluctance from those in power only confirm what happens so blatantly in the pitch.

    how many clubs are collectively campaigning for its use?

  33. what a dangerous word proof has proven to be. wmds and evolution spring to mind.

  34. @ak47

    Video technology in real-time, is the heads-up display typically found in fighter aircraft. The video technology people talk about for sports is not real-time.

    @many people, common to UntoldArsenal and new

    Bias is one of many terms that comes up in statistics. Accuracy and precision are other terms. Precision is a measure of the dispersion of the data, how wide is the distribution of data. Accuracy is a measure of how close some measure of central tendency of a sample, is to the true value from the population. Bias is the difference between the sample measure of central tendency and the true value in the population.

    There are two origins of bias for most people using the term at Untold Arsenal:
    1) Any bias in the officiating.
    2) Any bias in the reviewing of the officials.

    My reason for writing, is to comment mostly on the second point. While many people “expect” Arsenal supporters to have a bias in favour of Arsenal, nobody has looked at the data to see if such bias is present. I am in the process of producing a data set which starts from the UntoldArsenal reviews, and adds some other data and refines others (more accurate times for example). To me, I have no doubt that all the reviewers are behaving in a professional manner. If anything, I think it is possible they have slight negative bias (they tend to be harder on Arsenal than the other teams). But, at this time I have no analysed anything, I am just trying to get the data together.

    There is the odd mistake in the data (I’ve looked at about 1400 entries so far, out of something more than 5000 entries, and there are a handful of errors I’ve seen). Once I finish, the first thing to do is to see if I can produce reports that are equivalent to what UntoldArsenal has. If my data set is equivalent to UAs, then I can give UA a copy of my dataset as well as start analysing.

    While I have done a small amount of officiating in the past, what I bring to this data is almost 35 years of computer knowledge more or less centered on numerical methods and an awful lot of statistical knowledge (the minor in my M.Eng. was statistical physics, and my M.Eng. project was similar to a molecular dynamics model).

    I don’t think proof is going to fall out of the data. Proof is exceedingly difficult in statistics. A person can talk about things being very likely or very unlikely.

  35. The Spartan -I believe there is a bias against Arsenal. I also think that the argument of personal bias of the reviewer is not really interesting because the answer for it is clear and out of reach at this moment (having other refs and all that).

    What I am saying is that the one thing you can do now is prove that the data you have is saying something. Not because you “feel” or it “looks” like something, but with statistics. You can’t use average and distance form averages – statistical terms, and yet not do a test to prove your claim. It’s just meaningless and everyone starts arguing and your point is lost. Can you tell me what is the chance that the 12 average for arsenal is just luck? Well you can! Get the P-value with a t test and prove that the chance to get this result by blind luck is lower then x (5%, 1% whatever). Regular school stuff if you recall. Why not? This is really easy. If I had access to the full data I could do it in 10min in Excel…

  36. Gord – pls do! can’t wait for a statistic anlysis of this massive data that the ppl at UA have worked so hard on and with such commitment.

  37. We live in an “inerpreted world” and that is key to your investigation. As someone who has completed research and now works phenomonogically, I see bias as part of the human condition. As a Gooner I believe there is some bias against us.
    Intrestingly due such strikingly imbalance against us in your figures, over double the next team I can only coome to one educated conclusion accepting my own bias will affect this. Also there is no control group. Is that there is a bais influencing your results and figures relating to Arsenal.
    Strangely this bias as I see it provess your point though not directly the one that has influenced your research. That is, if the figures relating to Arsenal are so out of kilter and deviation from others that the qualified refs who have scored the games must be biased themselves. Therefore proving that bias exists within referees whether they like to admit it or not!
    The question arises after this is. How do we get the FA, the clubs, the players and most importantly the fans to accept this so something can be done about it?
    They say the camera never lies! This is true, but how it is interpreted is personal and individual. My favorite song is ‘Seadrum’ by the Boredoms. My mates hate it but they hear the same song!

  38. On a further note to the one above. No one likes to question the authenticy of their claims as this is at the heart of who we are. To question how we see the world and evaluate it asks us to reinterpret our nature of who and how we are.
    Why would anyone want to say to themselves perhaps I see it this way but in fact that isn’t how it is. For a ref or even the FA… God Bless them to admit that refereeing is biased, never mind in favour of whom, means accepting the concept and possiblity that what they see is not reality but, their reality.
    Oh yeah, reality only occurs when you close your eyes, you look away, you stop hearing, tasting, smelling, touching or your dead!

  39. Walter, this is a very interesting article. Would it be correct to expect when we play Man City and if we have an “average” PGMOL ref we could expect 12 wrong decisions and that City could expect 4.8? If so we could expect 2.5 times more wrong decisions than City with the consequence that we are 2.5 times more likely to suffer a match changing decision than City.

    I find it facinating that Riley is in such denial over the incompetence of his organisation – perhaps his his stastics come from a crystal ball!

  42. WRT Real-Time video

    I guess another name for this is “augmented reality”. In any event, Ars Technica is running a story this afternoon, about Google announcing that they have a research project to produce augmented reality glasses (like sunglasses).

    http://arstechnica.com/gadgets/news/2012/04/google-unveils-project-glass-augmented-reality-glasses.ars

    It will be a while once they are available, before there will be any kind of real-time video for football officials. If nothing else, the officials probably need cm resolution (possibly mm resolution) positioning sensors, but perhaps the various video high speed video feeds required will be able to differentiate one goaltender, the other goaltender, one team, the other team and the officials based on colour (ignoring shoes).

  43. @Aussie Scott – I looked again at QPR’s 1st goal, and Taarabt was played onside by Gibbs. There was a QPR player who was in an offside position at the time of the pass but he was not actively involved in or interfering with play, and so the goal was legitimate.

  44. All the (justifiable) flak is directed at certain refs and Riley. But there are paid – and anonymous (to me) assessors at these games. If they don’t perceive the ‘bias’ they must be thick. If they do see it– what happens next? The Prem and the FA are two of the bodies funding all this (with supporters monies?). Who are the people in these organisations who have the responsibility for checking that they are getting value for our money and not just presiding over a scam. There are so many people who must know at least as much as us- about what is going on– it is starting to get freaky. Why is it freaky? Because thousands of people, in total, are in a position of some ‘power to do something about it’ but refuse to do so. In my opinion this is big! When the truth comes out it will blow the Italian bribery job into the weeds and destroy the credibility of english football. Now there’s a thing!

  45. Ref Reviewer 02

    I get the bias argument, and realise it affects our credibility. However I have to say that our reviews are all verifiable, they are all available, and all of the conclusions we reach are in the public domain to be argued or to be contradicted.

    There is an open invitation to any qualified Ref to take one of our reviews and run it against the game it covers. I would love to see that happen, and for him to get in touch with us on his conclusions.

    We would happily take on reviewers from other clubs, hopefully that will happen and this thing will grow. That too is an open invitation.

    We understand the credibility issue all too well; it hangs over my head every time I do a review, and keeps my Arsenal bias in balance.

    Now if you want to talk about bias, what about MOTD who discuss Referee performances based on highlights put together in a cutting room?! What sort of nonsense is that? Or SKY who only show replays of the incidents that they choose to discuss, rather than the real key points.

    As far as I know, we here on UA are the only site, THE ONLY SITE who review a whole game for 90 minutes logging every incident that the Referee does, or should respond to. We are THE ONLY SITE dedicated to the absolute application of the Laws of the game. We do that not because we are Nerds or Geeks, but because we love football, and WE WANT OUR BLOODY GAME BACK! We do it not for ourselves (trust me here, it is tedious doing a review, 8 hours this week, 2 games) but because something is rotten at the core of our game and the power of the Web and digitisation allows us to expose the bias for what it is.
    If Mike Riley wants to log on and say our conclusions are wrong, and that he has the statistical evidence to prove it, he is more than welcome to. He has the power to shut us up. To dissect one of our reviews and say “this is nonsense, and here is why”. Instead he appears on BBC 5 live, answers a few selected questions, informs us all that Refs get 98% of offside decisions right, gives us no other data, and shunts back off under the stone from which he crawled. A feeble and pathetic attempt at credibility that fooled no one.
    We publish the data that the PGMOL are too frightened to reveal, not because they are corrupt (in my opinion), but because they are incompetent and self-serving. In reality they are employed by every one of us, to apply the laws of the game fairly, and without bias in the name of sport. Sadly we have exposed that is not happening.
    Rant over

  46. Just one question,

    The quoted figure of an acceptable 6 wrong decisions per game. Where does this figure come from?

  47. In fact it is simple: the day the PGMOL opens up to the public and publishes their reports we can stop at Untold. Until that day we are like you said reviewer 02, the only ones that are not just complaining but actually try to do something about it.
    How little this might be if it can grow and get bigger it will be us that will have started the process.

    How little this might look right now I sometimes compare it with how the self-immolation of one person Mohamed Bouazizi started the end of the dictatorship in Tunisia. Don’t worry I will not self-immolate myself for this but if our acts could lead to the current dictators in football losing their places I would be a happy man.

    And not for my own benefit but I want back the sport I once loved so dearly. And I really have the feeling I am not the only one with this feeling

  48. Stuart,

    The number of 6 wrong decisions is the result of the average of wrong decisions in all games we have reviewed so far this season.

    Like I said (I hope I made this clear enough in the article) that in fact it is not acceptable (we should always want the zero mistakes) but as things stand now this is the real number of wrong calls for each team. So in fact we have an all game average of 12 wrong calls in each game. One every 7,5 minutes in fact.

    So this is the number now and this is a number that should be brought down a lot.

  49. Excellent analysis Walter. We all knew we were unfair victims of slipshod reffing, but to see the extent has been a real eyebrow raiser.

    Thanks Walter.

  50. Wow, it really seems that this particular analysis has brought out the contrary forces in full force. Keep up the good work 🙂

  52. Well Anne in a way I rather like people questioning us. We cannot be too satisfied with what we do that easy.
    When it comes to referee things I want to come to perfection as close as humanly possible.

    Sleep well by the way 😉

  53. Hi Walter, am a researcher, i really like your research work above and more so the conclusions in the last three paragraphs. May be its time we embrace a bit of video technology to help even out…

  54. That should be also a next step Warui. I have experimented with it on a few occasions but I’m not a real computer genius and it takes me a lot of time to add such things.

  55. Hi Walter
    It’s of vital importance that people question what you do, your methods and results. If they didn’t they would be simply accepting, without question, your findings. This is no different from the media making statements and expecting the public to believe what they say “without question”. Or Mr Riley going onto 5 live and stating 95% of decisions are right…
    I personally mean no offence when I earlier questioned your findings, I simple feel it’s not fair to compare findings when the findings you are comparing aren’t like for like. Maybe, next season, you could consider picking maybe 6 fixtures of each team (home & away) so each team has, at the end of the season, equal results which can be compared.
    Either way, keep up the good work

  56. Stevie E, if all goes well we should be having at least 6 games of each team in our reviews and + 10 games of Liverpool/Newcastle and +20 of the other teams.
    If we would reach the +20 for Liverpool and Newcastle also we could have two groups to work with. And we could compare the results of both groups in a different way.

    And yes any critical suggestion is more than welcome for what we are doing.

  57. Walter
    That’s great, by the end of the season we should be able to see a clear picture emerge. I think you’re being followed by the way.. have a look at this in the Evening Standard (link to the full report on the Riley thread)-
    Moreover, bias is all well and good but the sort of refereeing favouritism that would help a team to be closing in on the Premier League title for the 13th time would amount to the greatest and most blatant scandal in sporting history. Football is a game of small moments and slight decisions but neither small nor slight enough to be manipulated on that scale.

  58. Walter, RefReviewers

    Please consider doing a table of right/wrong calls for as many of the last 10 ManUre games as possible. And add as much detail as possible on the most significant game-deciding calls.

    This documentation of the royal road to Rednose XX will be a priceless way to reign on His Parade. Surely you have something along these lines in the works? Perhaps even an illustrative unslated tackle or dive or two or three on video?

  59. Walter,
    If you excluded the incorrect Arsenal decisions from the ‘acceptable average’ calculation and recalculated what is the new average?
    This will give an indication of the true acceptable average for all non-Arsenal clubs against which the actuals could be compared.

  60. Sorry Walter that was rhetorical as I’ve done the calculation
    (1349-366)/(222-31) = 5.15 accepted incorrect decisions

    So I think the average for all teams is dragged up a whole decision by Arsenal’s figures.

    Therefore is it not more appropriate to say that only 5 decisions per game should go against a team?

  62. Jaroda,

    a very interesting remark and question.

    If I exclude the Arsenal games from this you get 888 wrong calls in 80 games. Meaning some 11 calls in each game in total and 5.5 wrong calls against each team. If all evens out of course.

    So in fact this is leading to a small difference of 1 wrong call more (or less depending on how you look at it) per game if we take the Arsenal games on board or not.

  63. Oh I think we were calculating at the same time. 😉

    And I see that I misunderstood you because I have excluded the total of all the wrong decisions in the Arsenal games and thus also the wrong calls against the opposition in those games.

  64. Actually your calculation is better as it eliminates the games that were influenced by both Arsenals incorrect decisions and the opponents at the time who may have been more favourably treated.

    I guess I was looking at the averages for each team and could see that 11 of the 19 teams (excluding Arsenal) appear in the range of 4-6 decisions, which would indicate a fairly acceptable deviation from the average of 5.

  65. @jaroda

    So does that mean that for 11 of the teams in the premier league (excluding Arsenal matches) the decisions even themselves out over the course of a season? On the basis of this calculation of course..

  66. @Anne,Walter: Having worked with computers for quite a while now and having tried to push ideas everywhere…there’s one thing I’ve learnt….and not that you guys don’t know it but I would like to talk a little abt it.

    The ref reviews are totally awesome and everything and I’m sure there is a huge amount of effort put in to it. And will stand if verified by Mike ‘ManU’ Riley. However just like this article..Walter…I think it will gain more attention and more buy in from refs of other teams if there are conclusions drawn like this one.

    @Anne: The reason THIS article has stood out because Walter has awesomely concluded (with help from others maybe) about something. This SOMETHING is what normal laymen like me who suck at data analysis and the math can easily understand.

    So I’m saying…lets try and simplify this even more and be ready with everything (as much as possible) before next season. I for one am throwing my hat in…if you want any help to make this better…you just have to Email me and I will try my best.

    But its all about KISS [Keep it sweet and simple]..there are variations but I’ll let that pass ; )

  67. Arvind,

    I understand what you try to say and I will be thinking aloud along what you are asking for. so please correct me if I misunderstood.

    In an ideal world we would and probably could present you the summmaries of each game we reviewed. And then link to the report itself where everyone can see from minute to minute how we reviewed each decision.

    Now I don’t know if this could be done technically but it could maybe be presented in a way that when we publish a review you only get the major decisions on your screen with a button to view the review completely.

    And in that way we could try to add more words to the reviews. Sometimes there is no need to say a lot (but in this way we could add and highlight great moments where the ref made a correct decision.

    And maybe but now I am thinking a lot further we could also add a link with each important decision with a video (as many people ask about this).

    So for those who want to check every decision they can do so if they want to. The others will only see the most important calls, a summary and the editorial words of the reviewer or my added words to a review and will have the chance to look at some clips explaining our decisions.

    Is it something like that Arvind you had in mind?

    Blimey, if it is and if we ever would be able to create something like that…. We would be making a giant step forward.

    But I don’t know if such a thing is possible. Because I think we also have to be careful because adding video clips might bring problems with publishing rights maybe??

  68. There are (almost) 2 duplicate columns in the reviews.

    The Correct/NotCorrect column is the same as the (binary) Points column: if Correct, Points=1; if NotCorrect, Points=0.

    The Type column is almost a duplicate of the Weight column. Weight=1 if Type is OFFSIDE, FOUL, THROWIN, GOALKICK, CORNER, REFCURIOSITY or OTHER; Weight=2 if Type is YELLOW; Weight=3 if Type is RED, GOAL or PENALTY. Which is a small table that is constant for all reviews.

    I would imagine most sources of video clips are copyrighted in some way. Whether any have a liberal enough license (such as one of the Creative Commons licenses), I don’t know.

  70. Gord, All,
    Since Gord does raise the legal point, does anyone know for Certain whether there are any “fair use” opportunities/loopholes/legal rights for using video clips of a certain duration on a non-profit basis? Their use is very widespread. Can it be done on UA under UK/EU law? Or on another non-UK server? It would be useful to know this going forward, in the event that video clips are ever considered for display in RefReviews, etc.

  72. @Arvind (others?)

    Ever heard of FMRD-Desktop? The entry at FreeCode.com (formerly Freshmeat.net) is:

    http://freecode.com/projects/fmrd-desktop-102

    This is a football dbase application for Windows or Linux, using either SQLite or PostgreSQL for the dbase backend. It is open source, using components in python, perl and Qt. It doesn’t look strictly applicable to what UntoldArsenal is doing with referee reviews, but the sponsoring company (Soccermetrics Research & Consulting, LLC) does:
    > Performance analysis of match officials
    http://www.soccermetrics.net/

    I’m up to my eyeballs in things to do at the moment, just finding time to try and keep up with new work by the reviewers is a struggle (I’m only at 34 or 35 games input).

  73. The BBC has an article about diving, nominall that the PFA is against it.

    http://www.bbc.co.uk/sport/0/football/17632778

    In that article, they mention that some formula exists to help in determining whether something is a dive. They don’t give the formula, nor do they reference it.

    Does someone have a reference to this formula?

  74. @Walter: The first thing I do when I see a ref review article is to jump to the summary of the First Half and Second Half and read what the reviewer has written. I then come back and look at specific rows for the data at that point. But the most valuable thing for me.. is your analysis at the end. Because, awesome as the data is…most people (its an assumption) are going to largely scroll through it and read only really controversial stuff.

    The data is valuable though at a later stage when you want to draw conclusions. Like in this article…if you hadn’t done all that work you wouldn’t have been able to write this article.

    So I’m saying…getting a lot of varied interesting “conclusions” .. where people can read and think… Okay this isn’t just data which is useful but hard for me to understand…due to complexity and volume..BUT someone who has explained WHY this data is important.

    So something like this article – People can see ‘Wrong calls’ – Direct impact on results…and get it immediately. I am working on something in my mind which might be helpful. But for now I hope I am clearer : )

  75. Hi Walter, I meant EPL also embracing a bit of video technology to help referees out. You are doing great Walter and I can only encourage you. keep it up

  76. To misquote from Xkcd.com: These arguments may contain advanced mathematics (which may be unsuitable for liberal-arts majors).
    On grounds of principle, not grounds of statistics, I would exclude the Arsenal results. Any teams with a low number of reviews I would give their averages a low confidence value to the average.
    People waving around statistics based arguments are fine but they need a seriously good understanding of statistical mechanics (not just plain old statistics) to avoid embarrassing themselves.

  78. If Steven Hawking said it evens itself out I may give it some credibility otherwise where is the science in it?

Leave a Reply

Your email address will not be published. Required fields are marked *