Big Data: 98 per cent as good as gut feel

The Human Face of Big Data.Via Brian's Books

Not so long ago I was given a demonstration of a very clever bit of kit to analyse big data. By gathering evidence about your expressed preferences shared through social media it could make predictions about the preferences you don't articulate.

Scary clever.

The person presenting proudly told us how their analysis had identified the right kind of music to attract the target market to a new energy drink.
Let's say they picked Artist X as the ideal brand amabassador.

Funnily enough, Artist X was exactly the person the product's brand manager had come up with from their own 'analysis'. I'm sure they had some evidence. Some charts with brand positions. Some brand truths. All that malarky. But they had much more. They had stuff no one had written down, no one could easily define. They had what they felt.

And where was all this processed? In the 'gut' of the brand manager - someone living and breathing and believing the brand.

The vendors of the clever big data cruncher marvelled at how Artist X could be proven to be a 98% fit with the requirements of the brand.

I fear they may be wrong.

I suspect the brand manager was 100% right and that the dimensions of data captured by the tool has gaps - gaps that only guts can currently fill.

You process big data every day - with staggering accuracy. When you drive a car or walk along a crowded street you are processing huge amounts of information and making decisions on it. In real time. If only big data could do that? It's getting better. Our weather forecasts are one example. But it's not 100%. There's still room for your gut - and it's important room.

One final example. When I started out as a local newspaper man, my first sub-editing job was to go through what were called the village columns. We'd call it user generated content today. Reports from local correspondents of village fetes, whist-drives, bowls matches, bring and buys, church services - that kind of micro-local news. It was mostly hand-written (semi-scrawled) and often on unlined paper. And it was my task to correct, amend, headline, and guestimate. I'd mark it all up with instructions for the guys who would set the pages (using ems and picas, font names and point sizes).

And then I'd have to estimate how much I needed to cut or add, what depth I should set images to (for example) to fill the number of pages I had been allocated, accounting for the adverts booked on them. It was a task that was not unlike estimating the number of individual straws in a decent sized haystack.
Yet, within a few weeks, I could mark up all that separate, different-looking, miscellaneous copy and have the output fit within 5 or 6cms of the final column each week. All done without the aid of any big data analysis.

Big data has a hugely important role for organisations (and for governments) moving forward. Who controls it and to what ends it is used will occupy us for years to come. But throughout that process let's not forget the extra value that something human adds.

The more digital we become, the more human we become

The more digital we become the more connected we become, the more human we demand.

And the more human we become.

I spoke at Mobile Content 08 in London yesterday. It was a great event with a stella line-up: fjord's Christian Lindholm, flirtomatic's Mark Curtis (author of Distraction), msearchgroove's Peggy Salz, Sean Kane from Bebo, Melissa Goodwin from ITV, Ray de Silva from Vodafone and Graham Thomas from T-mobile. Not to mention Day One's chair JonathanMacdonald (ogilvyone and chief node in the Every Single One of Us project - which you MUST check out) Day Two's Tomi Ahonen (Communities Dominate Brands and mobile legend) and in attendance, Day Three's Andrew Grill (London Calling). And many more, of course.

I opened with a talk on mobile's role in the eighth mass media - which is why the line about the more digital we become the more human we become was so appropriate.

Here's the accompanying slide deck - though you won't find the line in it.

Mobile's Role in Eighth Mass Media

View SlideShare presentation or Upload your own.

But the fact that The Power of The Network equates to the power of humanity is written through the idea of the eighth mass media.

As we become the connections and the way the connections are made we are reconnecting with our human selves.

We connect in a network - in adhoc self-forming communities of purpose. There is nothing from the centre. And the centre has a tendancy to dehumanise.

• The government line is very different from an MP's opinion.
• Company policy is very different from an employee doing what seems right and reasonable.
• Brand voice can't compare to a human conversation with someone who cares about what they are doing.
• Search engine optimisation can't beat my friends' recommendation (and of course their friends' recommendations)?

Now that we have the ability to organize ourselves through the tools to connect, what we are demanding is more human interaction.

Our connectedness in, through and beyond silos is tearing off the hard corporate masks and revealing the soft human faces. And we're ready to reach out and touch that warm flesh.

The digital future is a very human place to be. And for this, it is ever truer to who we are.

Clay Shirky on how algorithms caused the credit crunch

I had the great good fortune to meet and interview Clay Shirky (author of Here Comes Everybody) yesterday at Incisive Media in London.
He was in town for the AOP conference among other keynotes - and meetings with Government Ministers, no less.
So to get a one-to-one for any length of time was fantastic. We had a scheduled 20mins... but things got interesting so we stretched it...

The result is the video you see below. At least, that's part of the result. It's part one of seven. The interview lasts over 35mins in total and Clay answers not only my questions, but all those put through comments on this blog or via twitter over the last few days.

I'll be adding the next six parts over the next week or so - and for those who can take it all in one go - I'll add the full and unedited (and higher quality) version to google video before too long (youtube limits me to 10mins a pop, anyway).

In part one we talk about Clay's journey - what is in his background that makes him place the human element so high in his consideration of group forming network theory. (image by matlock via flickr)

And the new ways value are created in self-forming communities of purpose leads us to Clay's fascinating analysis of what the bankers did wrong and what they can learn (how social capital mitigates risk) from reconnecting with humans.

Hope that's whetted your appetite: Now hear from the man himself. And if you've missed Clay on his current trip to the UK, you can catch him again here. (blog of the event here) And you can buy his book, here.