Long read: the dangers of decision making and inertia in the age of big data

If forced to draw up a list of the top ten buzzwords of the 21st century, ‘big data’ would likely be on the list alongside ‘post truth’, ‘millennial’ and ’selfie’. But what does big data really mean, why has it become so important and what are its potential dangers?

A huge amount of our life is digitised and recorded, from our financial transactions to our train, bus and even car journeys. Depending on which mobile device you use the number of steps you take, the elevation you climb and even your heart rate, is constantly being measured and uploaded to a distant sever farm that few of us will even see. It may seem obvious, but we have more and more recorded measurements than ever before in history.

For the purpose of this essay, the exponential growth in the collection of metrics serves as a base definition of big data. Gerald Ashley, author of Two Speed World says “the data has always been there, it’s just that we’ve never been able to access it and people have become very excited. There’s a feeling that we should be able find out all sorts of new information.”

How big data changed advertising and media

A very simple example of the early success of big data can be found in the publishing industry and the measurement of advertising spend on print versus its online counterpart. The decline of print media has not only been caused by the proliferation of the internet and mobile adoption, advertisers themselves have hastened print’s demise by moving much of their advertising spend online, becoming far more reluctant to advertise in newspapers where, in comparison to the web, advertising can’t be effectively measured.

For example if an organisation was to advertise in a printed edition of a newspaper they would be able to find out how many copies of the paper were sold and could potentially measure how many people called an advertised phone number or website but other than that, data would be scant.

By contrast, advertising online will reveal how many people clicked on the advert, how long they stayed on the page and what they did next as well as collecting a plethora of demographic information – age, gender, nationality, etc. Some web measurement tools even record the exact movements of a user’s mouse across the page. It’s an obvious example but even at a very basic level such as media buying, the upsides of having lots of data are glaringly obvious.

The dangers of big data

But with big data comes an obvious danger too. Let’s say we have a database of several thousand people and two of the metrics we’ve recorded are one, the colour of a person’s eyes and two, whether they prefer cats or dogs. From this we might extrapolate ‘proof’ that people with green eyes are more likely to own a cat and people with brown eyes are more likely to own a dog.

The result of course would be nonsense. Gerald put it like this: “It’s quite dangerous to take a whole heap of data and just try and find patterns in it. Human beings love patterns. We’ll see faces on a piece of toast or on the surface of the planet Mars, so we have to be careful we don’t torture the data just to find a pattern that could be misleading.”

The dangers of the deliberate misuse of big data in the political and corporate world therefore are potentially frightening with organisations twisting data to arrive at a pre-specified destination. Gerald Ashley agrees.

“Politicians naturally frame questions to get the answer they want or the conclusion they want you to get to. So what looks like a very solid and correct way of doing a survey or opinion poll is very dependant on how you ask the question. It frightens me that people think big data is a silver bullet that is going to solve everything. The chances are that quite a lot of egregious mistakes will be made.”

To prove the point Gerald uses the following example. In 1981 Prince Charles got married, Liverpool won the European Cup, Australia lost the Ashes and the Pope died. In 2005 Price Charles got married, Liverpool won the European Cup, Australia lost the Ashes and the Pope also died. By this logic, if Prince Charles should ever decide to remarry, the Pope might be justified to worry.

I couldn’t help but fact check the anecdote and can reveal that although Pope John Paul II was wounded in an assassination attempt in 1981 he lived until 2005, although the rest of the incidents are indeed true. The fact that this story has entered the annals of folklore, despite having such a glaring factual error further underscores the human need to see patterns and dismiss outliers.

Asking the wrong questions could lead to disaster

Whether we like it or not big data is increasingly going to impact our lives and careers. What we need to do is to learn how to ask the right questions and learn to interpret data that doesn’t fit our pre-existing ideas so we don’t extrapolate misleading or useless answers.

A good real world example is the financial crash of 2008. The banks had no shortage of data, in fact they were swimming in it with armies of analysts and risk managers with PhDs in Physics, Economics and Maths all employed to maximise returns and avoid risk. Of course we all know what happened next.

In fact the crash of 2008 didn’t come as a surprise to everyone. There were a significant group of people who predicted the events which transpired but the banks failed to listen. If anyone has seen the Hollywood movie The Big Short (or even better read the non-fiction book), it reveals a cast of characters who identified the issues that would cause the banking system to collapse years before it happened.

What’s striking about the real life events of The Big Short is that while many of the individuals used the information to get rich, there were a few who attempted to warn the banks of the oncoming meltdown and publicly challenged them.

Elsewhere economists such as Nouriel Robini were laughed at when they pointed out the catastrophic risks lurking in credit swaps. As the saying goes – people who tell the truth aren’t popular. Big data then, can be completely and utterly useless if institutions fail to ask the right questions or fail to act on ‘inconvenient’ information. The banks had been asking themselves how to make more money, when really they should have been investigating the enormous risks of collateralised debt obligations.

Asking the right questions is crucial

“Early on in your career you need to know all the answers. When you get close to the top of your career you need to think – ‘what are the right questions?’. Answers are always there, it’s finding the right questions to ask which I think is very much more difficult ” says Gerald. Typically, having a plan is perceived as being much better than not having a plan but herein lies the danger.

Gerald makes a parallel with criticisms levelled at the government over the lack of concrete plans for Brexit. “When Mrs May said ‘Brexit means Brexit’ people say ‘what the hell does that mean?’ I think this is a very good example where people can’t actually even agree what we’re trying to solve let alone what the methodology might be.”

It’s quite easy to see that Brexit does indeed mean many different things to many different people. For some it was a deeply entrenched dislike of EU bureaucracy, for others a protest over immigration and for many it was simple economic argument.

So rather than the need for a concrete Brexit plan and timeline, we actually need to understand what the nation wants to achieve and what the best means to achieve those goals are. Only once we understand the multitude of differing issues can we begin to attempt action and it may well be that we can achieve the aims of the nation with or without triggering Article 50.

Reassessing normality and avoiding group think

“I tend to only like people who agree with me” says Gerald. It’s a danger in many organisations that our emotions and biases play a big role in our decision making. We like people who agree with us and dislike those who disagree or challenge us. This is a mistake. If management ignore conflicting views in the face of well presented data, institutionalisation and group think can soon set-in.

There are numerous historical examples of this. Perhaps the most recent was the UK phone hacking scandal that blew up in 2011. The practice of hacking phones had been reported since the mid 1990s yet most media owners failed to take it seriously.

Many editors even encouraged the practice whilst former editor of The Mirror Piers Morgan even admitted it in his diaries whilst hacking was still taking place. Despite all the evidence available, the warnings, the imprisonments of journalists and the reams of copy dedicated to the subject, the practice was still carried out well into the late 2000s.

The allegations and convictions mounted-up and when it was revealed that journalists had hacked the phone of dead schoolgirl Milly Dowler, the victims of the 7/7 terror attack and the relatives of dead British soldiers the affair finally exploded.

High profile media figures were put on trial, several were imprisoned and The News of The World, a newspaper which had been in publication for 168 years, closed for good. Whilst the incident could be considered ‘little data’ in comparison to recent scandals such as the Panama Papers it’s a classic example of the failure of senior executives to act in the face of overwhelming risk, despite being given numerous opportunities and warnings to do so. Rather than ask the question ‘is what we are doing ethical and is it legal?’ they instead focussed on fighting the journalists and organisations who brought the story to light.

When CEO of News International Rebekah Brooks was asked how she wanted the affair to end, she was reported as saying with “Alan Rusbridger on his knees, begging for mercy” (Alan Rusbridger is the former Editor of The Guardian). In retrospect it’s staggering the practices were so widespread and maintained for so long with so many journalists, editors and senior executives invested in the enterprise. It’s a lesson for all of us. If you don’t like what you see and management won’t listen get out or blow the whistle. In the words of author and former Quantitative Analyst Nassim Taleb “If you see fraud and do not say fraud – you are a fraud.”

Big data – the opportunities for accountants and finance professionals

Last year we published a piece called ‘Are Accountants tomorrow’s dinosaurs’ which discussed the challenges presented to the accounting sector by emerging technologies. It’s clear that many back-end accounting functions will disappear in the coming years. Accountants and finance professionals need to develop a new set of skills which will take them far beyond analysing data and keeping up with legislation.

They will need to make sure businesses are asking the right questions of the data to develop meaningful insights, not just answers that confirm pre-existing beliefs. All of us will need to make sure we don’t ignore outliers just to fit our model and perhaps most importantly we need to be aware of the different dimensions of risk. Finally, don’t let emotions, relationships or your own ego get in the way. Providing the data is there, we should listen to people who disagree with us – they might be right.

You can listen to Gerald Ashley’s talk – Messes, Problems and Puzzles by using the player below.

Benjamin Berry is the Content Manager for AAT  The Association of Accounting Technicians and the former editor of Yahoo

The content team are the owners of AAT Comment.