Tuesday, May 28, 2013
Big data... big ideas
Foreign Affairs magazine is not usually a tome of project management knowledge, but an essay on 'big data' by Cukier and Mayer-Schoenberger caught my eye.
In internet years, I think 'big data' is somewhere around 1994. So, there's a bit to go yet. But, forgive me -- did they say: "forget economically viable samples, accept messiness over accuracy, and be happy not understanding causation from correlation"?
One wonders if this pair have ever had to define "DONE" to the project sponsor?
I've done some pseudo-big data projects: data warehouse for a multi-billion $(revenue) company and multiple ERPs with zillions of millions of records. I never got away with 'messiness'. Somehow, the CFO, CEO, and all the other C's and VPs wanted to have assurance that business data was accurate... how quaint! When I was a business unit vice-president, I wanted accurate reports also. Who knew this wasn't needed?
However, I get it. Business is changing so project metrics may have to change as well. There may be a challenge fitting 'messiness' into the "technical performance measures" (TPM) or taking earned value on it.
And, certainly correlations instead of causality will start many arguments about "who shot John" and whether an improvement or a disaster can be laid at the foot of the project.
And, the economics of ever larger samples -- perhaps the whole population -- is going to come out of somebody's budget: there's no free lunch (data), even if user generated (See: Facebook)
KENNETH CUKIER is Data Editor of The Economist. VIKTOR MAYER-SCHOENBERGER is Professor of Internet Governance and Regulation at the Oxford Internet Institute. They are the authors of Big Data: A Revolution That Will Transform How We Live, Work, and Think
Check out these books I've written in the library at Square Peg Consulting