By Walter Broeckx
As promised in the Untold Ref Review of the game between Arsenal – Newcastle I told that I would explain why I would open a virtual bottle of champagne to celebrate something.
Because this ref review has been the 100th review our ref reviewers have done this season.
Yes that is right 100 reviews of games in the EPL this season. I think an appropriate time to stop for a moment and think back at how it all came this far.
When I started doing the first reviews at the end of the season 2009-2010 I could only dream of doing the Arsenal games. And that is what I did in the season 2010-2011. Only doing Arsenal games.
We then got some people saying that we could not draw any conclusions based on such a small number of games. I did some 50 games in total last year on my own but as said before only Arsenal games. And only 38 games in the PL. And we then asked if there was any other referee that would like to help us. And so we now have a team of 3 ref reviewers here at Untold.
So I would like to take this moment to thank my colleagues ref reviewers. I know what it takes to review a game. I know how much time you have to put in to it. And I would like to apologise for all those times that I came up with a new model that we had to use. And from the rather simple model we used at the start of the season we know have a model that is even wider than most computer screens. I ask the impossible from them but they are doing their best to keep up with my demands. And if you would complain about it I can always point my finger at Dogface who is also very much responsible for what you get to see when a review is published. But later more on our favourite Dog.
I can tell you that our reviewers take their job very serious and that we communicate on things when we have doubt or disagree. And thinking back about those 100 games I think we could say that we agreed on about 95% of the games and reviews without any comment or remark. Sometimes it was just a remark on the method we should use to give points to a certain decision but we also had a few things when we didn’t agree on the decision. And then we did what democrats do and take the majority of votes and that was the decision we used in the review. But the longer the season has gone the less moments we needed to discuss things in fact.
And as this is an Arsenal blog I must also mention that it hasn’t been easy for the ref reviewers all the time. Hard to believe it maybe but we also had our injury troubles. One of our ref reviewers was out for a few weeks but is now getting back. I can also imagine that the wives of the reviewers will have asked to them: what are you again watching football? So let me tell to you my dear reviewers: don’t let the reviews get between you and your wife too much. I know it can be like a virus at times where you want to start or finish a review but we should always remember it is a hobby and not something that should take over our normal live.
But I really thank you my ref reviewers for the hard work. For all that time you have sacrificed for this Untold project. I whish I could do something back to you to show my gratitude but the only thing I can come up with is inviting you to a drink when I come to the game against Aston Villa next week.
Another word of thank should go to the man who goes through live on the internet with the name Dogface. We all have known him from his amazing refwatch articles on Untold where he comes up with the most amazing numbers about a game even before a ball has been kicked. This is the man responsible for the current look of our ref reviews and I think we all agree that it looks so much better than what we had at the start.
Dogface is also working on more things to come that we can use at the end of the season where we should be able to give you more details, graphics, numbers about teams, refs and maybe we could see some links between some teams and refs. And by seeing I don’t just refer to seeing it on the field but also seeing it in some numbers.
So thank you Dogface for helping me out with this and for helping when I need to import some things in to a ref review like images or videos. And for learning me the difference between a database and an Excel file. And for explaining it again when I use the words wrong. And again. It’s all the same for me but then again I just know how to work with it, never mind I even try to understand how it all comes to that. But a big thank to you Dogface for you commitment for the cause. And the invitation to have a drink goes to you too of course. Hope to see you next week.
And last but not least a thank you to the man who without him this would never have happened. I mean no other than Tony Attwood. The man who came up with the idea of Untold is also the founding father of the Untold Ref Review. Not that I think he knew about it when he started Untold. He wanted to start something different. And my boy did he made something different.
I don’t know if he envisaged anything as big as we have today but it was his initial work and his articles that drew us to the Untold Arsenal website. And from the one thing came another and now we have a website that is very unique in the Arsenal blog world. Even in the darkest hours of our Gooner being we can find here a source that will cheer us up. Bring a smile on our face. Make us angry about yet another ref screwing up. On a scale between 0 and 10 this Untold Arsenal blog is having a score of around 50 in supporting the players, the club and of course our manager. And on that same scale we score around -40 when it comes to our feelings for the “authorities that run football”. A bit of anarchists on the internet when it comes to that.
People can look at us, laugh at us but we will carry on doing what we think are good at: being Untold.
At a time like this we should look to the future. And once again we should try do better next season. I hope by the end of the season that we will have something around 130-150 ref reviews. (I still do have a lot to catch up with to be honest) and then we will try to analyse all the data in the Dogface database and look for the hidden things. (Did I use the words correct Dogface?)
But when we think of next season we will try to go further once again. With rather a few people from other clubs coming to our website lately and I would like to thank those who came over in peace and with reasoned comments. Maybe this could be the start of referees who support other teams to join us. Just imagine that one day we would be able to review all the games in the EPL. I know there are refs out there who would join us if they knew about us. The problem is for them to find out about us. So maybe this is something we should be working on towards next season. And if those visiting supporters know a ref just give them a note so they can apply for the job to become a ref reviewer.
So many thanks to all who have helped us. Many thanks to all the people who have asked us questions after the reviews. Because your input as a reader has been of great value for developing the ref reviews. Many thanks once again to the ref reviewers themselves. Many thanks to Dogface and Tony. And finally a very big thank you for the PGMOL, the FA and the refs. Because without those last 3 it wouldn’t be as interesting as it is now.
@Walter
I’m a bit behind on transcribing, only at game 21. While how the data for your team’s reviews is displayed on the website has changed over the course of the season, so far I haven’t had to restart how I transcribe data. Which suggests you had a pretty good design to start with.
I think this has been a very good project, and I thank you and your reviewers greatly.
Great effort guys. Keep up the good work.
Well done everyone. Keep up the good work.
congratulations and long may it continue. the best afc blog hands down.
Congrats. The Gooner world is deeply indebted to you. Thanks a lot and please keep up the good work.
Congrats again
I hope your hard work highlights what you want it too. I also hope that more Referee’s can join your cause and give it unquestionable validity.
Well played Untold, continually putting paid journalists to shame.
Some of your ref previews have been scarily accurate. Perhaps the refs themselves could learn about some of their unconscious failings by reading your reports..??
Thank you for this fantastic work. I always look out for these articles and I send the links to friends – some of whom aren’t even Arsenal supporters.
Some of these friends – even Ars fans – at first, don’t believe it’s all so bad, but they’re starting to see what a good job is done at Untold.
I’ve only disagreed with one decision made in a review, and that’s the one where Sagna’s leg was broken following a collision with Asso-Ekoto of sp*rs. But having questioned the incident in the comments, the ref reviewer for the game explained why it was that that challenge was OK. (I maintain that BAO went to stop Sagna passing him, with a horrible result, and that an Arsenal player would never get away with a challenge like that – broken bones/land beyond the sideline, or not).
Untold’s reviews are entirely better than the fictitious 92.3% spouted by “refs reviewing refs bollux lot”.
It’s true that without the PGMOL – the work wouldn’t be as interesting as it is now, but on the other hand, we might have won the league a couple of times?
Ultimately, I’m prepared to wait for honours because when Arsenal get them, I’ll know they’ve been gained honestly, and in the meantime – I’m seeing some sumptuous football – played by the footballistically intelligent players of my beloved Arsenal.
Very, very well done, and thanks, Untold.
Thank You Walter and team. You guys rock. Untold rocks.
Surely the next stage in these ref reviews must be to put them on a more official level. At the moment, mainly Arsenal fans read the results with (in order) interest, irritation and outright fury.
The inevitable outcome for the fans must be a permanent lack of confidence in referees’ ability to officiate, due either to inefficiency or simple corruption.
Walter, you say that the reviewers must always remember that their work is only a hobby. I would suggest that what they do is becoming so in-depth and sophisticated, that officialdom should rapidly consider the contribution the reviews could make
towards the integrity of our national game.
A can of worms may well be starting to open….slowly. 😆
I’ll give you some feedback based on what a scientific reviewer would say to you if you were thinking of submitting a paper for publication in a scientific journal.
1. You’re not yet at gold standard but I’m not putting you down for that, I’m merely saying that, for gold standard, you need more time and resources than you are able to provide currently. You need three independent reviewers for each game, one Arsenal fan, one opposition fan and one neutral. No reviewer must know the results of the others and their analysis must be collated by a fourth person who does not know who submitted the analyses, nor their bias, until analyses have been colllated and compliled over many games.
2. It’s unclear whether your three reviewers are all Arsenal fans and whether, as a result, there are any intrinsic biases in how you do the analyses. We all know that, when at games, our instincts are biased in favour of Arsenal. Particularly on the 50:50 decisions……
3. There’s no evidence that the three of you have each analysed the same match, independently, without knowing the results of the others, to see how much variability there is in the analysis. In other words, are you infallible in your retrospective analyses and, if not, what % of decisions vary?
4. There’s no formal analysis done on what ‘normal variability’ might be. I’ve won 25 bets in a row on Betfair before now and the percentage likelihood of that happening based on the odds I’ve backed is very small (< 0.2% in fact), but I've done it. I've had no inside knowledge and what I find is that I can 'zone in' and 'get hot'. Overall, though, I lose about as often as you would expect in the long run. So there's a question here about how many matches you need to analyse before you can be sure the stats aren't 'bad runs'. You've not yet presented any statistical analysis to back up your claims. Is it, in fact, possible to do that?
5. You have not yet come up with a 'predictor' which will compute, based on the data, what you expect the refereeing outcome for a match to be, knowing the sides and the referee. These algorithms are never perfect and announcing the prediction before the game is impossible as then the referee, if they are indeed 'bent' can change their behaviour to discredit your method. But for internal purposes, such an algorithm adds weight to your results and you can arrange for independent arbitrators to validate the predictions prior to the game without communicating them to the big bad world. At the moment, you have hunches which are right sometimes.
6. You have not yet factored in the position that certain clubs are more likely to have more games with 'game changing decisions' simply because their scorelines are closer more often. Man City and Man Utd this season will have fewer games like that as evinced by their vastly superior goal differences. If Arsenal, Chelsea and Spurs are less likely to win 4-0, clearly far more games may be susceptible to one big wrong decision and this may not be a function of bias, more of game evolution. The best way, after all, for Arsenal to avoid the effects of bad refereeing decisions is to win games more easily! Or get thrashed 8-2, which of course is not what anyone hopes for…….
You'll note I've not said I disagree with the analyses you've done, I've simply said that these are the sorts of justifiable questions that will be raised.
If I were you, I'd set up a Company Limited by Guarantee, try and secure the services of 30 qualified referees representing all 20 EPL clubs each season and get each to agree to analyse 1 match each round of fixtures and they do X as a partisan referee and 38 – X as a neutral. You'd need 30 people to compare decisions of each game by the three examiners to build the picture of natural variability in analysis (this might only be necessary for 1 season if it turned out there was little, if any, difference) and you'd need some ICT guru who can produce standard templates and turn the analyses into meaningful data in as automated a manner as possible. It would also be useful to have some football people inputting what it would be useful to analyse/look out for.
If you get the pledges to do that, I'd go to Richard Scudamore of EPL, David Bernstein of the FA and Michel Platini of UEFA and try and get funding to run it for 3 years. I don't know how much it'd cost to do, but it wouldn't be huge sums. If they won't back it, ask them if they have anything to hide!! They are supposed to upholding the integrity of the game, after all…..
That would give you the evidence base to really flush out what is going on and keep all the referees on their toes.
Short of confessions from managers, Directors and players about what is going on, I wonder if you'd manage to secure convictions using what you've currently got. Since rich folks usually have the ability to hire high powered lawyers who can bring in high powered statisticians for hire to challenge your assumptions………which may hang a jury and lead to acquittals…..
congrats walter and your team..we really appreciate all the hard work and dedication you guys put into these reviews..you gave us a way to not only see but also prove just how biased a certain ref was against us..keep up the good work and i really do hope you find another refs to join your team
Congratulations for all ref reviewers and Dogface, may I give you all virtual bottle of your favorite drink as thank you fo your hard work. Oh and I’m thankful for your wives letting you do this 🙂
Maybe a certain person could give us the funds to do what you suggest Rhys.
For the moment I still have to see that my wife and children have something to eat every day… and I think this goes the same for the other ref reviewers
@Rhys
I don’t know that it is necessary to set things quite like you envisage. Yes, it would probably be useful to test if bias is present, but I don’t know if it is necessary to expect that it is present and significant. The reason I am transcribing this data (manually), is that I want to do some other kinds of analysis on the data in the time domain and in the frequency domain. There might be some interesting runs in the data as well (similar to your Betfair example).
I don’t think statistics is the answer if court of law is involved, the only court UA needs to convince is the court of public opinion. If the people no longer trust referees to be behaving in a professional manner, the FA, the Premier League and PGMOL will need to change how they operate.
But, who knows what will be found in the data?
I would like to add my voice to the many that have already expressed their gratitude for what the Untold blogging team produce. I recognise that their is sacrifice of time and energy to produce the quality stuff you do and I don’t say it often enough. So thanks, it is appreciated.
Walter this is a magnificent site and I certainly find the articles informative and interesting. With so many anti Arsenal comments in the media and on certain blog sites you provide much needed balance – and balance based on fact.
I think you have expanded the Ref reviews extremely well making good use of the resources available. You have put together a store of information which can be the basis of questions which the PGMOL would find hard to answer – should they have the guts to attempt to do so.
Unlike Rhys the Rambler I have no objection to you having a sence of pride in a Belgian player in an Arsenal shirt – to me that is a good emotion, not a weakness. The important thing is what the player does when he pulls on the Arsenal shirt and that he gives his all for it.
Off subject – but I would also extend my very deepest sympathy to those who lost loved ones in the Swiss coach crash.
Actually, I can’t believe that it has been 100 reviews already this season. I have been reading every single one of them with a lot of interest.
Keep up the good work. It is amazing to see how far you have come over the past two years. The ref reviews are solid analyses of the ref performances. Never mind what Rhys is saying. Within the scope of what you’re trying to achieve, you are doing a great job.
Congrats Walter and the team…all working people would agree with the fact that all the ref reviews are written after painstaking hardwork and all without a care for personal reward and renumeration! My heartfelt thanks to you all and Walter need not worry about the wives not liking it…women know how to have their way!!!
Great job guys – well done for all your hard work .Hope these reviews get a wider coverage at other friendly sites or at popular blogs of other clubs .You may even be elevating their profile !
I do hope you’re sending these reviews to the PGMOL ,the FA ,the League Managers Association and the Players Association, basically to educate them .
Thank you! Great stuff, please keep it coming.
This is amazing guys, i have to say a big well done. Let’s hope your hard work can eventually get taken seriously by the powers that be and the game can be rescued back from the corrupted wastelands upon which it sits. My only fear is that they might be worried of changing anything for fear of this being labelled an admittance of guilt!
Great complements to our Ref-Reviewers. Congrats for a job welldone withing the confines of your limitations. I know that you guys would have done a lot more if you had the wherewithal. Kudos to you, Walter and the team.
Walter, RefReviewers, Tony,
The quote that I love most is from Shakespeare King John: “So foul a sky clears not without a storm.” Thank god, you are that storm!
Dogface, Walter,
Is there any possibility of being able to offer links to video clips that illustrate key points in the ref reviews: like game changing non-calls? That is, links to youtube or elsewhere where these, our own bite-sized retrospective video replays would dwell and be linked to? Or perhaps a better technical solution, but it would be awesome to complement the numbers with the exact video that displays the key moment. Developing this capability for reviewers to bring a qualitative dimension – a needed qualitative dimension – would vastly increase the scope of the recruitment drive for ref reviewers and bring in hordes of new visitors to this site. The marriage of your numbers with video clips of key calls/non-calls would be state of the art. Perhaps a reader or a friend of a reader with these video editing skills could be recruited to make this possible.
Gord,
Yes, to court the the court of public opinion: where enough, not even 50 per cent, can constitute a critical mass for change – demanding and bringing about video replay for fair play. Bringing about that outcome would circumvent the siren lure of the quest for quantitative truth that Mr. Jagged poses as the only evidence. Imo, it’s getting to fair play that trumps all and any fetishizing of the numbers. No, the earnest numbers effort hereabouts are especially worthwhile because they hold a moral mirror up to help expose the bent/incompetent calls and callers that are ruining this game. Forget Mr. Jagged’s dream scenario of spending years to get what he would consider actual evidence and present it to the powers that be with his dream payoff: being able to legitimately say “gotcha!” to them, when they predictably refuse to fund the project. Are you kidding, Mr. Jagged? What sheer idiocy and crackpot realism to even suggest such a path. Spending years of number crunching just to get to a climax: a point where we present it (in hair shirts and on bended knee) to our satanic majesties so that they can say “No,” if that much. Sheer masochism.