14 responses

  1. Stuart

    Good morning Walter,

    Having had a quick skim through (I’ll read properly later at lunch) I am just wondering if the 98% accuracy figure quoted by the FA only considers offsides which were given and doesn’t include offsides that should have been given but were missed? I can’t imagine the FA having a panel like yourselves to look into such matters.

  2. Arun

    In these two games, Dowd has been extremely unlucky to have such incompetent linesmen with him for these crucial fixtures.He has been one of the top referees in the league and the likes of the arsenal-newcastle game (a classic example of british game) are included in his resume. He is one of the strong contenders to follow Riley.
    Just because he got some wrong decisions in these two games, ruling him out as one of the best referees in the league a mistake only a novice will commit.

    That’s what would have been said by the media (if they bothered to publish this ).

  3. imagooner

    Nice article Walter 🙂 hope we see more articles of the referee’s who are in pole position for the riley’s job. I am sure it would be someone capable of working completely anonymous in the higher (corrupt) echelons.

  4. goonergerry

    This is a very good article. Referees have a 50% chance of getting decisions right by sheer chance alone.
    We need a completely neutral, independent body to monitor what is or is not given by refs and to review the decisions in each EPL game not a body like the FA that clearly have an interest in portraying referees as being always right. Given the level of incompetence of EPL referees who can look at a man being tripped in the penalty area in front of their eyes and wave play on as Lee Probert did at the weekend- it beggars belief that they are getting offsides right 92% of the time. You have to ask when they are seriously throwing around a stat like that- are they for real?

  5. Geezer

    Thanks for the good work!

  6. Arun

    Even 98% is a poor figure for an offside decision as only one wrong decision can change the result of match. The offside decisions must be 100% correct and that’s possible only through technology.

  7. Gord


    I believe the analysis (Walter’s white line) is something that Prozone Sports could do for PGMOL. Certainly they are providing some kind of service to PGMOL, as there have been announcements of arrangements between PGMOL and Prozone. But supposedly these arrangements with Prozone started before Poll retired. And yet, someone in another thread mentioned that all the feedback the referees get is a number, no “training”.


    I still don’t think technology is the savior you do. If you bring in video replay, and leave the system the same as it is now, it may help the assistants. If you bring in video, and then change how the offside is administered, the attackers will change what they are doing to “push the envelope” more, and it will result in little or no help to the assistants.

    It is likely that only the decisions that are called, are analysed. Certainly that is the “lazy” way to analyse something. Otherwise, someone has to decide what has to be analysed.

    If we look at 292 correct out of 320, we expect the standard deviation on 292 to be something proportional to 17. Which means that 309/320 is statistically the same as 292/320 (96.5%), at the 67% confidence level.

    If some analyst looked at every situation where an attacking ball passed the line of the second last defender to see if any attacking players were 1) in an offside position and 2) in a position to take advantage of that offside decision, we would probably see something like twice as many decisions? What is the ratio of called to uncalled? If we double the number of correct and total decisions, and look at what adding sqrt(correct) to correct brings, instead of getting 96.5%, we get 95%.

    A big part of the problem is that the league hasn’t properly defined what they mean by correct. And even if they did define it properly, they need to provide the data so that this 98% number can be verified.

  8. Gord


    I better go through my numbers again, I had only been awake 20 minutes or so, and no coffee. I’ll get a better answer as to how likely or unlikely that 292/300 is consistent with 98%.

  9. Gord

    Back again. If I treat how many decisions in total as a Poisson variable, and then draw a binomial variable from that total number of decisions that has a probability of success of 98% (what EPL claims), and generate 100,000 experiments, I see an actual correct fraction as low as 94% in 0.008% of the trials.

    If I drop the claimed rate to 97%, I see 0.002% of trials getting as low as 92% (just outside your 91.25%).

    In order to observe a fraction correct as low as 91.25% in 5000 trials out of 100,000 trials (95% confidence level), the “true” ability to judge offsides correctly (or however one words it) is about 94% (or less), not the claimed 98%.

    Another way to word it; if we observe 91.25% correct, at the 95% confidence level we expect the actual percentage correct to be between about 88 and 94%.

  10. Todd

    I agree its very important to check to make sure the statistical significance is appropriate, but I would take a different tact and look at the teams being evaluated and see if certain teams have higher total #’s of offside calls(due to style of play) and check that against the percentage of each team reviewed. That initial bias could distinctly impact/skew the calls. For example, fow many spurs games have been evaluated – Adebayor is notorious for being offside. Arsenal also tend to play through the last line leading to a relatively high number of offside situations compared to say Stoke who cross into the box.

    Maybe you’ve posted it and I missed it, but it would be helpful to see the percentage of games per team evaluated. So out of 60, probably 25-30 have been Arsenal games. As I mention above, this can skew the statistics due to a natural bias in the way Arsenal play. It would be interesting to see that rates for each of the top teams. Off the top of my head I would bet Arsenal and Spurs have the highest number of offsides, followed by city, then chelsea, just by the way they play.

  11. Gord


    Perhaps you could generate some data? 🙂 Being an engineer (with a minor in statistical mechanics), I’m good with numbers. There is a problem in doing this per team, in that the total number of decisions falls, which is going to increase the standard deviations of any analysis I do. The thing to do, would be to start from the teams for which the most data is available, and work down until it gets too noisy.

    I went to the gym, and then came back and did a bit more. Instead of just 1 binomial deviate for each Poisson total number of calls (and 100,000 of those), I did 100 binomial deviates for each Poisson. Nominally 10,000,000 trials per run.

    The EPL has no grounds to say that they are 98% accurate based on what referee reviews UntoldArsenal has done. Not even 1 in 10,000,000 will hit 91.25% accuracy if their “true” rate is 98%. With this expanded run of simulations, it is pretty obvious that the true rate for getting offside decisions correct, is somewhere between 88 and 94%.

  12. WalterBroeckx

    we had some 16 Arsenal games in the total of 60

  13. novicegooner

    pretty ‘sophisticated’ discussion happening here. My comment won’t contribute anything to the discussion. I just want to say how amazed I am with the work on this site about the refs. I learned a lot about rules of the game and the ‘truth’ of Premier League and I share what I get from here with my friends in Indonesia. I want them to be Arsenal supporters who do not only care about scoreline.

    super work, everyone!

  14. Gord

    Making statements based on statistics is not a good thing. It is sort of like standing on quicksand that has a crust on it. Is it going to break through?

    You cannot prove anything with statistics. If you are lucky you can disprove things. But advertisers the world over, continue to say that some clinical study proves some product has biological efficacy. The clinical study is there to gather statistics, a clinical study can’t prove a darned thing. Except that any company that says that some clinical study proves their product is effective, means that you should never buy their product ever again as clinical studies are not meant to prove anything (outside of advertiser stupidity, which knows no bounds anyway).

    Claiming 98% success on anything is good and bad. People like to believe that the 98% claimed is real. Being able to prove 98% can be difficult. And any proof is going to hinge on how the success is defined. Not just in general, the success may only be possible to do some very specific details in the definition.

    A referee in a game (or an assistant referee) only has a small interval of time, and only a single view of the incident, to make a decision as to offside or not. Reviewers from UntoldArsenal (or other places) have an indefinite amount of time and as many reviews as they want to make a decision. Assuming they can get views to look at.

    I think that technology is useful in training referees. Which is something that apparently Graham Poll has said recently, and wondered why EPL referees weren’t using. I’ve asked on UA about why Prozone Sports isn’t providing this, seeing as they have been providing services to PGMOL for many years and are seemingly capable of doing so based on what they advertise to provide. And nobody on UA seems to want to comment on this. Am I beating a dead horse? What services are Prozone Sports providing PGMOL, if they aren’t training referees using analytical methods and video? BJs?

    Walter wants to prove something about referees. I went to the trouble of doing something like 100 million experiments to see whether an EPL claimed offside effectiveness of 98% is reasonable. I think I’ve shown that 98% is not consistent with what UA reviewers have been seeing, and that any real offside detection rate in the EPL is somewhere between 88 and 94%. And no comment. Either online or private (as he has access to my email). Which I don’t understand.

    If further analysis of things is useful, having some kind of dbase access is useful. My suggestion, is CSV (comma separated value) is a useful format for everybody. UA has reviews (mostly for Arsenal) going back a year or three. Among other things, a person could look at decisions based on nationality, region of England, race or a few other things. But scraping HTML to make database entries to do something is not a time effective way to start things.

Back to top