Why is Data So Unreliable?

This post is about the information that is used by a supercomputer we all carry around with us (no, dummy, not your mobile phone. Not quite yet). It is about the idea that we all suffer from innumerable distortions and agendas. Even those who claim to have no agenda…well, that is the agenda. To pretend they are above it or haven’t got one is disingenuous and indicates a worrying lack of self-awareness.

This supercomputer is an organic machine, part of a larger organism with a wide range of sensors for input, it requires very careful maintenance, is easily damaged and easily corrupted, benefits from being made to regularly solve complex problems, has seemingly limitless storage capacity and is carried around on your shoulders.

In our rush to worship at the altar of AI, data, quantum computers, and machine learning we step neatly over the weirdness and utter unpredictability that this supercomputer (for ease of typing it shall henceforth be known as the brain) brings.

optical illusion
Two straight lines, bent by the brain.

Sure, you can reduce all the external inputs into a measurable thing. The number of photons that hit the retina, the loss caused by an aged lens, the sensitivity of our fingers, sense of balance, the speed at which we solve problems and so on. Despite possessing a brain in all its judgemental and unpredictable glory we seem desperate to quantify and measure everything possible. For with the surety that comes from turning every conceivable bit of input into a number, then surely it must be within our ken to calculate the output? And if we don’t get it right then we can re-examine the computational processing algorithms and refine them until we come to as close an output that the programmer(s) expect. You see, there will be some parameters set for an acceptable/realistic/likely outcome somewhere in the brief, and usually, the aim is that the output matches the expectation.

The brain starts to gallop ahead, and I suppose this is what intrigues the scientists trying to create AI when the capacity for the random social variables and the filters they create comes in. The ability of the brain to make the weirdest associations from two apparently random bits of data always astonishes me.

For example: 30 years ago I shared a flat with Oliver Reed’s nephew, who apparently strove to exceed or at least replicate the lifestyle of his uncle. I was persuaded to take a tab of acid (LSD), which I tore in half because it scared me but, hey, peer pressure. This had a startling effect on me and I had to sit alone as it felt as if snooker balls of thoughts were cannoning into one another and going off at funny angles. To this day, if I see a snooker game I am reminded of that moment. I imagine it will be a very very long time before there will be a computer that can make those sort of weird cognitive leaps.

Data can be as much social and experiential as pure numbers fed into STATA, R or SPSS and manipulated in various approved ways. The data coming from the brain’s sensors is reliably distorted through one or several social lenses. Are you rich, poor, foreign, insecure, angry, a victim of something, in a wheelchair, aspirational, impaired, with a neurological condition (I have MS – have had for 26y)? Perhaps you are clinging to the notion that you are utterly impartial and free from an agenda and thus right? That is a powerful filter, often producing feelings of self-righteous indignation that can’t always be adequately expressed in 280 characters.

It doesn’t stop this fellow from trying though.

When you are designing research, analysing research, presenting research, having research presented to you then try to remember that your brain and the recipient’s brain have different filters. They may seem externally similar, but at some point, that same information will hit a unique filter and the gap between intent and understanding soon becomes apparent.

The human desire to avoid cognitive dissonance is strong and mismanaged leads people to do terrible things to try and ‘fix’ it. Thankfully, we have a great ability to bend data to fit our pre-conceived notions of what feels right or to fill voids with made-up data. We ALL do it. I believe that the most we can do is to open up the analysis to others who manifestly do not share a similar agenda. As independent as possible and are trained to look for inconsistency, in the accrual of data or the motives of those who have handled it before you see/hear it.

So-called facts in newspapers are a good place to start asking why and how. In a commercial environment, people bandy around poor data and try to cover it with the force of personality of seniority. BBC Radio 4 has an excellent series (available for download) called Thought Cages that deals with the vagaries of the human brain in an amusing and engaging way.

We all lie and deceive all the time. It is in our nature. Sometimes you need a different variety of deceiver to look into your world and help you identify the deceits. I can help you, so contact me via LinkedIn.


Why is data dangerous?

In the words of @RorySutherland: “The data made me do it” is the 21st Century equivalent of “I was only obeying orders”. The growing power and influence of Data Science touches everyone’s lives. Sutherland also remarks: “Markets are complex and there can be more than one right answer. People in business prefer the pretence of ‘definitive’ because if you can show you’ve done the ‘only right thing’ you have covered yourself in event of failure”. These are all attempts at Plausible Deniability, and they are weak.

For the record, plain old data is not dangerous, you are unlikely to be hit by an errant Spearmans Rho, or a rogue Control variable that detached itself from an analysis. Data is just a record of the measurable values of something that has happened in the past. Digital exhaust, if you will. Like speed in a car, it is the inappropriate use of it that causes issues.


Doing the right thing often sees people becoming  enslaved to Type 1 and Type 2 data, because they are the easy parts. You can hire experts, who can count well, use the software and understand how to tease out knowledge from the data points. What the majority can’t do, or may even do intentionally, is to manipulate the presentation, context and language used when presenting their findings. This is the Type 3 data I talk about, that isn’t traditional data as we know it.

Type 3 data is the really dangerous stuff. The reason for this is our complete fallibility as human beings. This is nothing to be ashamed of, it is how we are made and conditioned. It is in fact, entirely, boringly, and ordinarily normal. I was recently told by a lawyer – I say this because she is pretty well-educated – that all statistics are a lie. She then cited the famous Mark Twain (nicked from Disraeli) saying of, “There are lies, damn lies and statistics”, as if this were all the proof she required. Interestingly, when I challenged her on this and made a case for accurate uses of statistics she refused to even acknowledge this. She was wedded to her belief and I must be wrong. Case closed.


I think immersion in courtroom rhetoric may have been getting the better of her. However, this goes to show the just how dangerous we humans can be. Imagine being a client with a lawyer whose dogmatism may cause them to overlook or be able to question relevant statistical evidence? All stemming from a strongly held view that all statistics are lies. Professor Bobby Duffy recently wrote an excellent book called Perils of Perception and on p.100 he shows just how problematic this view can be.

My point is: If a person who is well-educated, and practising in a profession like law, can hold such a position, then it is not beyond any of us to do so, quite unwittingly. Until one is more familiar with the behavioural biases that we are all susceptible to, the way Type 1 and Type 2 data can be mis-represented (Type 3 data) and how that uses our in-built foibles to generate a reaction.

This is where someone who understands both of these areas, and can blend that knowledge into an expertise which is useful, can help you. When important decisions on strategy, direction and spending  are conditional on interpreting data from others, you want to get it right first time. If not, you’ll be forced into, “The data made me do it”, and that rarely ends well.

burning money





Let’s break down today’s bad data usage – Yes Guardian newspaper, I mean you!

Overnight there was a report published in The Guardian newspaper entitled “Met police’s use of force jumps 79% in one year”. I see the hysteria on Twitter – being whipped up and added to by the usual suspects who revel in the dog-whistle approach to political discourse – about the use of force by the Metropolitan Police being used disproportionately against black people.

“The Metropolitan police’s use of force has risen sharply in the last year, with black people far more likely to be subjected to such tactics than anyone else, the Guardian can reveal.”

Firstly: this is not an attempt to take sides. The police may be guilty of the accusation. Without correct and fair anaslysis of data it is impossible to tell. See a previous post about how to approach stories like this.

Secondly: the purpose of this article is to interrogate the findings of the Guardian’s reporting of this story. If this undermines the story then so be it. Do not conflate that with an endorsement of the police in London, for I do not know enough to comment about them. This is about the use of data.

The main thrust of the article is the “79% in a year” claim. It is what has been seized upon and retweeted with vigour. Nowhere does it appear that the people getting all worked up over this selective quote  have actually looked into the data.

“On 39% of occasions in which force was used by Met officers in the first five months of the financial year, it was used on black people, who constitute approximately 13% of London’s population.”

The first thing that struck me about this piece was the language and imagery used. Whilst the language is not the data, the way it is used certainly serves to alert you to the fact that they may be glossing over details in the pursuit of shock value. The Guardian is (was?) a credible broadsheet with a left-of-centre bias. Nevertheless, now they are giving away their content for free, they seem to be leaning towards the ‘clickbaity’ style of reportage, and that is a pity. Look at the graphic they have used. It is fairly emotive stuff. A white man pointing a weapon at you.

taser front view

In the first paragraphs the article uses words and phrases like, “jumps, risen sharply, most likely, on average, approximately, raised alarm, receiving end, stark figures, police culture” and the like.

These are written efforts to engage the enraged response (metaphorically speaking) part of our brains rather than the rational analysis. The System 1 reaction and not System 2 as Daniel Khanemann calls it in his book Thinking Fast and Slow.

Arguably, the alleged disproportionate use of force by police officers against black people is so serious an allegation that it warrants slowing down, taking a deep breath and analysing correctly?

Let’s break down the critical analysis a bit by asking some questions, and making some observations.

  • The only reference to the data used is “Guardian analysis of official figures“. This alone should sound the loudest alarms ringing in your head and set you into sceptical analysis mode. What figures, analysed by whom, what is their expertise, what were the controls used, compared to what, is there (perish the thought that a journalist is anything other than scrupulously impartial) an agenda on the part of the presenter of these figures?  [I think I may have found the data being used. See the bottom of the article for links. It certainly isn’t acknowledged in the article. This might lead a cynic to wonder if it may be being taken out of context and the journalists don’t want this easily checked for fear of undermining their credibility.]


  • Many people think of the word black being interchangeable (perhaps incorrectly) with people of colour. It turns out that the Guardian even mentions that ‘Asians’ and ‘Other’ are not part of this classification.


  • Are the figures generated by the ethnicity initially recorded by police codes for radio use or the self-defined ethnicity – 16+1 versus 9 – codes used by the subjects, even if they differ from the officer assesment. It doesn’t say.


  • In the last paragraph of the piece the most convoluted attempt at figures is used;  we witness the groups ‘Asian’ and ‘Other’ being rolled together to make a ‘52%’ claim sound more shocking. They need to decide how they portray things and stick to it. Previously the paper excluded ‘Asian’ and ‘Other’ from the ‘Black’ category and instead let them sit outside along with ‘White’ in order to use the five month and 79% figure on which the outrage is based.


  • There is no indication if these figures are split between reactive (responding to calls from the public), or proactive (the officers see something that they decide to investigate further). Proactive interventions are carefully considered by officers, they rarely steam in like you see in the movies. Things like back-up availability, whether they are single-crewed (and far more vulnerable), priorities like previous calls, outstanding paperwork (yes, really, there is a lot), their caseload and so on. Proactive policing is where a racist would shine as they would be able to target black people if that was their aim. From there they would need to engage and at least claim a veneer of credibility for their choice to use force. That wouldn’t last long as everyone would need to be in on it. These days that is very difficult.


  • Debra Coles from the charity Inquest is reported as saying: “This also provides yet more evidence about the overpolicing and criminalisation of people from black and minority communities. It begs important questions about structural racism and how this is embedded in policing practices.” – From other remarks in the article, when the Metropolitain Police were approached and asked for their view, it sounds like after losing some 20k officers the police are rarely proactive, mostly reactive. If only they had the time and resources to ‘overpolice’ anywhere.


  • What if, and I am trying to steer away from political and social commentary here for it is not my intention, the police respond to more incidents in places where there is a greater proportion of black people? I X amount of interactions involve use of foprce then is stands that the use of force against blacks is more likely. There is no doubt there is historical antipathy towards the police amongst much of the black community, especially in London. Previous generations of the Met (and other forces) have not been known for their even-handed approach towards the black community. Young men (for it is predominantly males)  in groups often feel that their masculinity is being challenged if an authority figure like a police officer lawfully requires them to do something. What if this leads to more physical  resistance which in turn leads to force having to be used? What if the white people, the Asians and the Others are more compliant when dealing with the police? What if, what if, what if? The fact is that these figures do not seem to be presented in a holistic manner. By that I mean controlling for variables such as age, gender, location, time, weather, changed police priorities, changed dynamics of interaction due to cuts in resources and so on.


  • The phrase ‘use of force’ is misused by the journalists and politicians. The police use a very specific definitionb, and it is not what the ordinary person thinks it may be. A voluntary handcuffing is a use of force. You know, the kind where the officer says something like, “for my own protection I am going to handcuff you.” and the subject complies. Perhaps a single-crewed female arresting a large male and having to drive him to custody herself. Merely drawing Captor (CS) spray needs to be recorded as a use of force. No one was sprayed, situation calmed down. Same as the drawing of a baton. Force is also shooting someone dead. There is a wide definition of force. Force, in police recording terms, does not mean taking the suspect to the ground in a violent bundle.


  • The whole method of recording has changed, a fact the paper skips neatly over. Too complicated to explain I imagine. The simple fact is that comparing these new figures generated and recorded one way with the past where they were not recorded in the same way, if at all, is simply invalid. It is far too soon to tell.


  • The politician, David Lammy MP, famous for trying to whip up stories like this to create indignation – I say this merely because he is a public figure who regularly tortures data or chooses to use tortured data –  betrays a lack of understanding when he talks about the criminal justice system and the police. The police in London are merely one small part of this national system. Saying there is systemic racism at each stage of the system in a piece targeted at the police in London does smack of trying to score wider points and not, in my opinion, worthy of inclusion. It weakens any point trying to be made. It is good to have Lammy on board for a bit more clickbait type appeal though. He has a large Twitter following and retweeted the article almost immediately. Surely not because he is mentioned in it.


  • As Matt Twist of the Met Police said, “…the figures should not be compared with population demographics. He said: “The collation of these figures is still in its early stages, and as this is new data, there are no previous benchmarks to compare it to. Therefore any conclusions drawn from them must be carefully looked at against this context, and should only be compared with those individuals who have had contact with officers, rather than the entire demographic of London.” You may think he is a police stooge but it does not make his statement incorrect.


  • The paper even says it is comparing  FY 2017/18 to FY 2018/19. This means that from April 6th 2017 to April 5th 2018 and similarly for 18/19. This is important because the new recording system was introduced from April 2017. The data being quoted is April to August 2017. It is being compared with April to August 2018. What happened to the seven months in between? Does this show a steady rise instead of a jump? Has anything else changed in this time? For example: the new system may not have started well and overlooked items or officer engagement was not what it should be, resulting in pressure from Borough Commanders down to record more accurately, leading to an apparent jump in incidents when it is actually a rise in adherence. Just ignoring a seven month gap is concerning. Why? An oversight or intentional?


Images are worth a thousand words. Misleading images are still far more impactful than poor descriptions. I reproduce this because it is a howler of a poor and misleading graphic.  The article is using the Financial Year for measurement, hence the mention of 2019 whilst we are in 2018.  Laying out the same images of London side-by-side implies that a comparison is about to be demonstrated.  However, the left hand image uses Westminster and mentions five other boroughs, none of which are referenced in the right hand image. This makes the image of questionable value, other than adding to the devaluation of credibility.  The source attribution should say that this is where the data was sourced, not presented in this way. I rather implies that this graphic is from the Met Police. It isn’t.

I think this was rather twisted to produce the graphics: Met Police Use of Force information.

howler of a bad graphic

Interestingly, the Met Police data gives this caveat, albeit buried on the third tab of their use of force stats page and not linkable by a URL. It explains areas where the data may be misinterpreted. The journalists don’t bother to tell us if they have taken this into account or not. We’ll never know.

CoversheetSo you can see that this article – and many stories on many topics – is riddled with inconsistencies. To me, I just dismiss it because it hasn’t got some of the basics right. It may be speaking a degree of truth, but that truth is devalued by the poor presentation.

Data is fine. Data is useful. Data is just digital exhaust. Data without context is just numbers and means nothing.

As I said previously, “Data only helps you take a problem apart and understand its pieces. It is not suited to put them back together again. The tool needed for that is the brain.”

Try to dissect stories quoting numbers. Be they in the press or someone making a commercial claim in order to influence your actions.


Here are some likely data sources for the story and for you to use when reading these type of stories using numbers to give credibility to their assertations.:

UK Government crime statistics

Metropolitan Police data

Office for National Statistics

UK Data Service video


PS: Anecdotally: I have known many types of officer. From 6’3″ tall Senegalese immigrants, who started as a PCSO in London and is now policing rural Oxfordshire, to short white people that are Reading natives who are born and bred and police there. They vary in their attitudes and actions because they are people. The huge majority want to make their communities better places. I do have no doubt that amongst them there are a few racist thugs, albeit a tiny and ever-decreasing amount. A bit like regular people I suppose. 


PPS: There may be typos. I try very hard to proofread. I am a terrible typist though.