Sunday, August 20, 2017

Average is not always the answer

Doesn't all data sets have an average?
Actually, not
  • Yes, data taken from a common distribution along a common scale  has a parametric value* that emerges from a long run of repetitions

    The repetition-weighted sum we commonly call the average, even if it's not the most frequent outcome, and even if it's not an allowable outcome (the average of integers is often not an integrer, as the average value of the roll of one die is 3.5)
  • Data that cluster's about a central value always has an average. This clustering idea we call the "central limit theorem"
Here's a couple of issues
  • No, not every data set we run across in PM are from a common distribution, or has a common scale, and so the concept of average doesn't apply. Example: big projects and little projects don't have a common scale: big numbers for the former; small numbers for the latter
  • Not every data set clusters
Three ideas you may not have thought about:
  1. Clustering: When there's clustering, there will be an average. Usually, data taken from a common scale clusters You can't average feet and inches; nor apples and oranges. Common scale required!
  2. Not clustering: Data taken from many different scales doen't cluster, but rather follows a "power law". This gives rise to the so called "80/20" rule and other other ideas commonly shown on a Pareto Chart of descending values... so, no cluster effect, and NO AVERAGE.

    Almost all issues dealing with money is power law stuff.
    There's just no meaningful average of big projects (on one scale) with little projects (on another scale). The 80/20 rule is more the way to look at it since a Pareto Chart is scale-free!
  3. Independent events: Something happens; when will it happen again? You're doing something; when will it be finished? This isn't clustering, and it's not 80/20 stuff. You read this page; when will you read it again? Or, how long will it take you to finish reading?

    This leads to the so-called "additive rule": just add a constant to what you know to get the next outcome. Where does the constant come from? You estimate it! This leads to: "just give me five more minutes to get this done" sort of thing.

    Of course, there's no average because there's no common scale because everything is independent, memory-less (the present does not depend on the past, though the past may figure into your experience to give an estimate of the constant).
* Parametric values, like average, are calculated; often they are not observed or even reliazable in the data set
This discussion adapted from the chapter on Bayes Rule in "Algorithms to Live By" by Christia and Griffiths

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!
Read my contribution to the Flashblog