Bayes Thinking Part II

---

In Part I of this series, we developed the idea that Thomas Bayes was a rebel in his time, looking at probability problems in a different light, specifically from the proposition of dependencies between probabilistic events.

In Part I we posed the project situation of 'A' and 'B', where 'A' is a probabilistic event--in our example 'A' is the weather--and 'B' is another probabilistic event, the results of tests. We hypothesized that 'B' had a dependency on 'A', but not the other way 'round.

Bayes' Grid

The Figure below is a Bayes' Grid for this situation. 'A+' is good weather, and 'B+' is a good test result. 'A' is independent of 'B', but 'B' has dependencies on 'A'. The notation, 'B+ | A' means a good test result given any conditions of the weather, whereas 'B+ | A+' [shown in another figure] means a good test result given the condition of good weather. 'B+ and A+'  means a good test result when at the same time the weather is good. Note the former is a dependency and the latter is a intersection of two conditions; they are not the same.

  

The blue cells all contain probabilities; some will be from empirical observations, and others will be calculated to fill in the blanks. The dark blue cells are 'unions' of specific conditions of 'A' and 'B'. The light blue cells are probabilities of either 'A' or 'B'.

Grid Math

There are a few basic math rules that govern Bayes' Grid.

• The dark blue space [4 cells] is every condition of 'A' and 'B', so the numbers in this 'space' must sum 1.0, representing the total 'A' and 'B' union
• The light blue row just under the 'A' is every condition of 'A', so this row must sum to 1.0
• The light blue column just adjacent to 'B' is every condition of 'B' so this column must sum to 1.0
• The dark blue columns or rows must sum to their light blue counter parts

Now, we are not going to guess or rely on a hunch to fill out this grid. Only empirical observations and calculations based on those observations will be used.

Empirical Data

First, let's say the empirical observations of the weather are that 60% of the time it is good and 40% of the time it is bad. Going forward, using the empirical observations, we can say that our 'confidence' of good weather is 60%-or-less. We can begin to fill in the grid, as shown below.

In spite of the intersections of A and B shown on the grid, it's very rare for the project to observe them. More commonly, observations are made of conditional results.  Suppose we observe that given good weather, 90% of the test results are good. This is a conditional statement of the form P(B+ | A+) which is read: "probability of B+ given the condition of A+".  Now, the situation of 'B+ | A+' per se is not shown on the grid.  What is shown is 'B+ and A+'.  However, our friend Bayes gave us this equation:

P(B+ | A+) * P(A+) = P (B+ and A+)  = 0.9 * 0.6 = 0.54

Take note: B+ is not 90%; in fact, we don't know yet what B+ is.  However, we know the value of 'B+ and A+' is 0.54 because of Bayes' equation given above.

Now, since the grid has to add in every direction, we also know that the second number in the A+ column is 0.06, P(B- and A+).

However, we can go no farther until we obtain another independent emprical observation.
 

To be continued

In the next posting in this series, we will examine how the project risk manager uses the rest of the grid to estimate other conditional situations.

Share this article with your network by clicking on the link.

 

---

Like this blog? You'll like my books also! Buy them at any online book retailer!

Bayes thinking, Part I

---

---

Our friend Bayes, Thomas Bayes, late of the 18th century, an Englishman, was a mathematician and a pastor who's curiosity led him to ponder the nature of random events.

There was already a body of knowledge about probabilities by his time, so curious Bayes went at probability in a different way. Until Bayes came along, probability was a matter of frequency:

"How many times did an event happen/how many times could an event happen". In other words, "actual/opportunity".

To apply this definition in practice, certain, or "calibrated", information is needed about the opportunity, and of course actual outcomes are needed, often several trials of actual outcomes.

Bayes' Insight
Recognizing the practicalities of obtaining the requisite information, brother Bayes decided, more or less, to look backward from actual observations to ascertain and understand conditions that influenced the actual outcomes, and might influence future outcomes.

So Bayes developed his own definition of probability that is not frequency and trials oriented, but it does require an actual observation. Bayes’ definition of probability, somewhat paraphrased, is that probability is...

The ratio of expected value before an event happens to the actual observed value at the time the event happens.

This way of looking at probability is really a bet on an outcome based on [mostly subjective] evaluations of circumstances that might lead to that outcome. It's a ratio of values, rather than a frequency ratio.

Bayes' Theorem
He developed a widely known explanation of his ideas [first published after his death] that have become known as Bayes' Theorem. Used quantitatively [rather qualitatively as Bayes himself reasoned], Bayesian reasoning begins with an observation, hypothesis, or "guess" and works backward through a set of mathematical functions to arrive at the underlying probabilities.

To use his theorem, information about two probabilistic events is needed:

One event, call it 'A', must be independent of outcomes, but otherwise has some influence over outcomes. For example, 'A' could be the weather. The weather seems to go its own way most of the time. Specifically 'good weather' is the event 'A+', and 'bad weather' is the event 'A-'. 

The second event, call it 'B', is hypothesized to have some dependency on 'A'. [This is Bayes' 'bet' on the future value] For example, project test results in some cases could be weather dependent. Specifically, 'B+' is the event 'good test result' and 'B-' is a bad test result;  test results could depend on the weather, but not the other way 'round.

Project Questions
Now situation we have described raises some interesting questions:

• What is the likelihood of B+, given A+? 
• What are the prospects for B+ if A+ doesn't happen? 
• Is there a way to estimate the likelihood of B+ or B- given any condition of A? 
• Can we validate that B indeed depends on A?

Bayes' Grid
Curious Bayes [or those who came after him] realized that a "Bayes' Grid", a 2x2 matrix, could help sort out functional relationships between the 'A' space and the 'B' space. Bayes' Grid is a device that simplifies the reasoning, provides a visualization of the relationships, and avoids dealing directly with equations of probabilities.

Since there's a lot detail behind Bayes' Grid, we'll take up those details in Part II of this series.

Photo credit: Wikipedia

Like this blog? You'll like my books also! Buy them at any online book retailer!

Is it alright to guess in statistics?

---

Is guessing in statistics like crying in baseball? It's something "big people" don't do.

Or is it alright to guess about statistics? 

The Bayesians among us think so; the frequency guys think not. 

Here's thought experiment: I postulate that there are two probabilities influencing yet a third. To do that, I assumed a probability for "A" and I assumed a probability for "B", both of which jointly influence "C". But, I gave no evidence that either of these assumptions was "calibrated" by prior experience.

I just guessed

What if I just guessed about "A" and "B" without any calibrated evidence to back up my guess? What if my guess was off the mark? What if I was wrong about each of the two probabilities? 

Answer: Being wrong about my guess would throw off all the subsequent analysis for "C".

Guessing is what drives a lot of analysts to apoplexy -- "statisticians don't guess! Statistics are data, not guesses."

Actually, guessing -- wrong or otherwise -- sets up the opportunity to guess again, and be less wrong, or closer to correct.  With the evidence from initial trials that I guessed incorrectly, I can go back and rerun the trials with "A" and "B" using "adjusted" assumptions or better guesses.

Oh, that's Bayes!

Guessing to get started, and then adjusting the "guess" based on evidence so that the analysis or forecast can be run again with better insight is the essence of Bayesian methodology for handling probabilities.

 

And, what should that first guess be?

• If it's a green field -- no experience, no history -- then guess 50/50, 1 chance in 2, a flip of the coin
• Else: use your experience and history to guess other than 1 chance in 2
  

According to conditions

Of course, there's a bit more to Bayes' methodology: the good Dr Bayes -- in the 18th century -- was actually interested in probabilities conditioned on other probable circumstances, context, or events. His insight was: 

• There is "X" and there is "Y", but "X" in the presence of "Y" may influence outcomes differently. 
• In order to get started, one has to make an initial guesses in the form of a hypothesis about not only the probabilistic performance of "X" and "Y", but also about the the influence of "Y" on "X"
  
• Then the hypothesis is tested by observing outcomes, all according to the parameters one guessed, and 
• Finally, follow-up with adjustments until the probabilities better fit the observed variations. 

Always think Bayesian!

• To get off the dime, make an assumption, and test it against observations
• Adjust, correct, and move on!

---

Like this blog? You'll like my books also! Buy them at any online book retailer!

The THREE things to know about statistics

---

Number One: It's a bell, unless it's not
For nearly all of us when approaching something statistical, we imagine the bell-shape distribution right away. And, we know the average outcome is the value at the peak of the curve.

Why is it so useful that it's the default go-to?  

Because many, if not most, natural phenomenon with a bit of randomness tend to have a "central tendency" or preferred state of value. In the absence of influence, there is a tendency for random outcomes to cluster around the center, giving rise to the symmetry about the central value and the idea of "central tendency". 

To default to the bell-shape really isn't lazy thinking; in fact, it's a useful default when there is a paucity of data. 

In an earlier posting, I went at this a different way, linking to a paper on the seven dangers in averages. Perhaps that's worth a re-read.

Number Two: the 80/20 rule, etc.

When there's no average with symmetrical boundaries--in other words, no central tendency, we generally fall back to the 80/20 rule, to wit: 80% of the outcomes are a consequence of 20% of the driving events. 

The Pareto distribution, which gives rise to the 80/20 rule, and its close cousin, the Exponential distribution, are the mathematical underpinnings for understanding many project events for which there is no central tendency. (see photo display below) 

Jurgen Appelo, an agile business consultant, cites as example of the "not-a-bell-phenomenon" the nature of a customer requirement. His assertion: 

The assumption people make is that, when considering change requests or feature requests from customers, they can identify the “average” size of such requests, and calculate “standard” deviations to either side. It is an assumption (and mistake)...  Customer demand is, by nature, an non-linear thing. If you assume that customer demand has an average, based on a limited sample of earlier events, you will inevitably be surprised that some future requests are outside of your expected range

What's next to happen?
A lot of stuff that is important to the project manager are not repetitive events that cluster around an average. The question becomes: what's the most likely "next event"? Three distributions that address the "what's next" question are these:

• The Pareto histogram [commonly used for evaluating low frequency-high impact events in the context of many other small impact events], 
• The Exponential Distribution [commonly used for evaluating system device failure probabilities], and 
• The Poisson Distribution, commonly used for evaluating arrival rates, like arrival rate of new requirements

Number three: In the absence of data, guess!

Good grief! Guess?! Yes. But follow a methodology (*):

• Hypothesize a risk event or risky outcome (this is one part of the guess, aka: the probability of a correct hypothesis)
• Seek real data or evidence that validates the hypothesis (**)
• Whatever you find as evidence, or lack thereof, modify or correct the hypothesis to come closer to the available evidence.
• Repeat as necessary

(*) This methodology is, in effect, a form of Bayes' reasoning, which is useful for risk analysis of single events about which there is little, if any, history to support a Bell curve or Pareto analysis. Bayes is about uncertain events which are conditioned by the probability of influencing circumstances, environment, experience, etc. (Your project: Find the Titanic. So, what's the probability that you can find the Titanic at point X, your first guess?)

(**) You can guess at first about what the data should be, but in the absence of any real knowledge, it's 50/50 that you're guessing right. Afterall, the probability of evidence is conditioned on a correct hypothesis. Indeed, such is commonly called the Bayes likelihood: the probability of evidence given a specific hypothesis.

---

Like this blog? You'll like my books also! Buy them at any online book retailer!

The principle of 'calculated risk' (updated)

---

"In carrying out the task assigned .... you will be governed by the principle of calculated risk ... which you shall interpret as avoidance of [risk] exposure ... without good prospect ... as a result of such exposure ...  of greater [benefit]" (*)

Admiral Chester Nimitz
to his subordinate admirals,
Battle of Midway, June, 1942

"You will be governed by the principle of calculated risk"
What does that really mean?

"You will be governed ..." means you've been handed the governance task; there has been a passing of (command) authority and decision-making. Whatever the strategic and most far reaching business and project considerations there are, and whomever has the responsibility for articulating the strategic vision, all that has been compressed into an instruction to "you". "You" have the authority to assess cost and benefit in the moment, and pull the trigger ... so to speak ... to take a risk or not.

"Calculated risk" is intended to convey a defensive tactic to protect a scarce or endangered asset or outcome; unless the loss of that asset begets a larger offsetting advantage. To that end, there are these constituents:

• First, a risk assessment based upon what it means for the loss of a scarce asset or a debilitating outcome vs the benefit or advantage of a favorable outcome. Call this a cost-benefit analysis and calculation
  
• Second, the quality of your knowledge base, assumptions of knowledge accuracy and timeliness, and a heads-up that there may be knowledge gaps. This is really the epistemic component of the risk: how much can the risk be mitigated by better knowledge?
• Next, doctrine, ideology, or rules-of-thumb are made subordinate to the calculation. Now to walk away from doctrine for a calculated risk takes some intellectual flexibility and maturity, to say nothing of walking out on the limb.
• If there is an adversary or nemesis, game-theory may be useful to estimate reactions (walk a mile in their shoes, etc .... )
  
• And last, if there is no scarcity which requires the protection of a risk calculation, then the principle is unnecessary.

Fair enough

What happens when it comes to actually facing the risk in a real project situation?

• Intellectual flexibility may be required. 
• Cool heads will be needed; emotion will be left at the door, hopefully
  
• Random effects will almost certainly intervene and perturb the knowledge-base calculations. You probably can't do much about the randomness; that's the nature of the it ... more knowledge doesn't help.
• Updates to game theory assumptions we be needed along the way.
  
• And, revisits to the knowledge base to update the risk calculation (the calculation may be rather dynamic in the doing ...) will be needed. (**) 

Plan B:

In spite of cool calculations, in the heat of the moment managers may blunder through the guardrails. Then what?

There should be a framework for Plan B on the shelf. Facts at the time will fill in the framework.

Ah, but who's in charge of Plan B? 

Someone should always be hanging back to grasp the strategic picture while tacticians deal with the here-and-now. And, that someone should have supreme executive authority to step in and make corrections.

_____________

(*) A Naval War College essay dissecting the Nimitz principle of calculated risk is found here.

(**) This idea of revision in the face of new information is the Bayesian approach risk calculations. That is: you have a hypothesis, "A", based on your going-in knowledge; then there's a change or there is observed evidence "B". Now you know more, and so the likelihood changes ("likelihood" being the probability that the observed evidence will support the original hypothesis, P(B/A)

 

---

Like this blog? You'll like my books also! Buy them at any online book retailer!

Guessing and Bayes

---

In my posting prior to this one, I gave an example of two probabilities influencing yet a third. To do that, I assumed a probability for "A" and I assumed a probability for "B", both of which jointly influence "C". But, I gave no evidence that either of these assumptions was "calibrated" by prior experience.

I just guessed

What if I just guessed about "A" and "B" without any calibrated evidence to back up my guess? What if my guess was off the mark? What if I was wrong about each of the two probabilities? 

Answer: Being wrong about my guess would throw off all the subsequent analysis for "C".

Guessing is what drives a lot of analysts to apoplexy -- "statisticians don't guess! Statistics are data, not guesses."

Actually, guessing -- wrong or otherwise -- sets up the opportunity to guess again, and be less wrong, or closer to correct.  With the evidence from initial trials that I guessed incorrectly, I can go back and rerun the trials with "A" and "B" using "adjusted" assumptions or better guesses.

Oh, that's Bayes!

Guessing to get started, and then adjusting the "guess" based on evidence so that the analysis or forecast can be run again with better insight is the essence of Bayesian methodology for handling probabilities.

 

And, what should that first guess be?

• If it's a green field -- no experience, no history -- then guess 50/50, 1 chance in 2, a flip of the coin
• Else: use your experience and history to guess other than 1 chance in 2
  

According to conditions

Of course, there's a bit more to Bayes' methodology: the good Dr Bayes -- in the 18th century -- was actually interested in probabilities conditioned on other probable circumstances, context, or events. His insight was: 

• There is "X" and there is "Y", but "X" in the presence of "Y" may influence outcomes differently. 
• In order to get started, one has to make an initial guesses in the form of a hypothesis about not only the probabilistic performance of "X" and "Y", but also about the the influence of "Y" on "X"
  
• Then the hypothesis is tested by observing outcomes, all according to the parameters one guessed, and 
• Finally, follow-up with adjustments until the probabilities better fit the observed variations. 

Always think Bayesian!

• To get off the dime, make an assumption, and test it against observations
• Adjust, correct, and move on!

---

Buy them at any online book retailer!

Plan v Objective

---

"In war [projects] nothing goes according to plan, but always remember your objective"

Israeli general

Good advice.

And, of course, your objective always is -- or always should be:

Apply your resources to maximize their value added while taking the least risk to do so.

But our general's admonition begs the question:

Is "nothing goes according to plan" the same as there's "no value in planning"? And, if so, why plan in the first place? Why not maximize agility?

Or, why not be Bayesian about it: Make an educated guess to begin, and then replan with new information or circumstances?

The usual answer

The usual answer to those questions is that the value of the plan is in the planning, that is: discovering one or more paths to victory! One or more ways to accomplish the objective.

And, if there is more than one way to get there, then whatever plan is adopted is not totally fragile; an alternative is available if things go really wrong.

That all said, planning is about doing these tasks and investing intellectually in their development:

• Establishing the scope detail that fills out the objective .... or narrative
• Anticipating the risks and devising mitigations .... or not (some risks can be ignored)
  
• Assembling resources; training staff and robots [AI is in the training frame these days]
• Establishing a sequence for doing the work
  

Such that, when the plan goes awry, it can be reconfigured -- perhaps on the fly -- and re-baselined, always with the most strategic objective in mind.

And, did I mention that the foregoing is Bayes-style planning methodology: always update your first estimate with new information, even it makes the first estimate look a bit foolish or optimistic

Yogi said:

Yogi said a lot of things, but he said this that seems to apply:

"If you don't know where you are going, you may be disappointed when you get there."

In our business, you might write it thus:

"If you don't [or won't] plan what you are going to do, you may be disappointed in what you wind up doing"

And, you might miss the objective altogether. You spent all the money -- presumably other people's money [OPM] and you didn't do the job! [That's usually a challenge to your career]

---

Buy them at any online book retailer!

Small data is the project norm

---

I've written before that the PMO is the world of 1-sigma; 6-sigma need not apply. Why so? One-time projects don't generate enough data for real statistical process controls to be valid.  To wit: projects are the domain of small data. (Usually)

And so, small data drives most projects; after all, we're not in a production environment. Small data is why we approximate, but approximation is not all bad. You can drive a lot of results from approximation.

Sometimes small data is really small.  Sometimes, we only have one observation; only one data point. Other times, perhaps a handful at best.

How do we make decisions, form estimates, and  work effectively with small data? (Aren't we told all the magic is in Big Data?)

Consider this estimating or reasoning scenario:

First, an observation: "Well, look at that! Would you believe that? How likely is that?"
Second, reasoning backward: "How could that have happened? What would have been the circumstances; initial conditions; and influences?"
Third, a hypothesis shaped by experience: "Well, if 'this or that' (aka, hypothesis) were the situation, then I can see how the observed outcome might have occurred"
Fourth, wonderment about the hypothesis: "I wonder how likely 'this or that' is?
Fifth, hypothesis married to observation: The certainty of the next outcome is influenced by both likelihoods: how likely is the hypothesis to be true, and how likely is the hypothesis -- if it is true -- to produce the outcome?

If you've ever gone through such a thought process, then you've followed Bayes Rule, and you reason like a Bayesian!

And, that's a good thing. Bayes Rule is for the small data crowd. It's how we reason with all the uncertainty of only having a few data points. The key is this: to have sufficient prior knowledge, experience, judgment to form a likely hypothesis that could conceivably match our observations.

In Bayes-speak, this is called having an "informed prior".  With an informed prior, we can synthesize the conditional likelihoods of hypothesis and outcome. And, with each outcome, we can improve upon, or modify, the hypothesis, tuning it as it were for the specifics of our project.

But, of course, we may be in uncharted territory. What about when we have no experience to work from? We could still imagine hypotheses -- probably more than one -- but now we are working with "uninformed priors". In the face of no knowledge, the validity of the hypothesis can be no better than 50-50.  

Bottom line: Bayes Rule rules! 

---

Buy them at any online book retailer!

Uncertainty is non-negotiable

---

"Uncertainty is an essential and nonnegotiable part of a forecast. .... sometimes an honest and accurate expression of the uncertainty is what has the potential to save [big things].... However, there is another reason to quantify the uncertainty carefully and explicitly. It is essential to scientific progress, especially under Bayes’s theorem."

"The Signal and the Noise: Why So Many Predictions Fail-but Some Don't" 

by Nate Silver

Now, some say: "we don't estimate; we don't forecast".
Of course, that's nonsense. Everyone estimates, if even only in their head

• How long will it take me to write this blog?
• How long will it take me to go to lunch?
• How long will it take me to do almost anything I can think of? 

But, Mr Silver leads all discussion of uncertainty, estimates, and forecasts around to Bayes Theorem, which can be laid out this way as a process:

• Formulate an issue or question or hypothesis
• Make an early guess as to outcome
• Experiment to gather evidence as to whether or not the guess is reasonable
• Re-formulate based on evidence -- or lack thereof
• Repeat as necessary 

In fact, in another part of Silver's book, he says this -- a cautionary statement for project managers:

"In science, one rarely sees all the data point toward one precise conclusion. Real data is noisy—even if the theory is perfect, the strength of the signal will vary. And under Bayes’s theorem, no theory is perfect. Rather, it is a work in progress, always subject to further refinement and testing. This is what scientific skepticism is all about."

And, one last caution from author Silver -- which reinforces the ideas of the Bayes process and also makes the point -- often ignored or overlooked --  that there is often little enough data inside one-time projects to support textbook statistical approaches:

"As we have learned throughout this book, purely statistical approaches toward forecasting are ineffective at best when there is not a sufficient sample of data to work with." 

---

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

Of facts and theories

---

"I have no data yet. It is a capital mistake to theorize before one has data. Insensibly, one begins to twist facts to suit theories, instead of theories to suit facts"

Sherlock Holmes

Yes, Mr Holmes has hit upon the dilemma of various reasoning strategies:

• Inductive reasoning: from a specific observation to a generalization of causation
• Deductive reasoning: from a generalized theory to a predicted set of specifics

As he correctly posits, inductive reasoning is hazardous. Just a slight error in facts, or in fashioning causation, or most frequently confusing causation with correlation, may lead to quite incorrect theories.

Thus, the strength of Bayes reasoning (*), a form of deductive reasoning. Aren't we all Bayesians?

-------------------
Look up Bayes Theory on Wikipedia; it's means of deducing conditions which are predictive of facts, a form of statistical reasoning.

---

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

Precision without accuracy

---

I've often said the game of American football is a game of precision informed by inaccuracy.

From sometimes 25 yards away, in the split of a second, among the tangle of potentially 20+ actors all scrambling, a referee makes a judgment (hypothesis) about where the tip of an oblong ball happened to be when the action stopped.
Then, a steel chain is laid down to measure off the distance from the tip to the end of the chain (facts). The latter is precise; the former is often wildly a guess, but nonetheless is transformed instantly into "fact"

And, you may ask, what has this to do with project management, or management at all?
It's all about two schools of thought about managing with uncertainty -- managing what we know about risk (or, what we don't know, which is trickier even)

1. School of objectivity: facts come from what we observe; estimates are derived from experience and projected onto to knowable future circumstances.
   Risk is a matter of estimating impact (see: experience) and then applying a probability derived from the facts about how often such has occurred before.
   This is the "frequency-based" view of assessing risk
2. School of subjectivity: Nothing wrong with observable facts, to be sure. But in the school of subjectivity, it's what you believe to be the case that carries the day. The set of beliefs, however, is not set in stone; in fact, to be a good subjectivist, you have to be willing to update your beliefs in the context of new facts (observations)

Not that the objectivist would ignore new facts; they wouldn't.
But, the subjectivist begins where an objectivist never would: with a guess (gasp! ... a guess! shocking!)

• But, a guess about what? Not "facts", of course, but rather a guess (conjecture) about a hypothesis of the circumstances that might lead to facts (observables).

• Or, if there are observables, then a guess about the circumstances (hypothesis) that led to those observables.

• Either way, if the hypothesis is discoverable, or verifiable, then new facts are predictable.

And, so that sets up a set of conditional relationships:

1. Hypothesis, and 
2. Hypothesis (conjectured or validated) given some observables
3. Facts, and 
4. Facts (or new facts) given that a hypothesis is actually valid

These four make up the eco-system of the subjectivist. There is constant working about of a "a priori" hypothesis; facts, and do they fit the "a priori"; and either a modified hypothesis, or if its valid, then predicted new facts.

If you are one that wrestles with the four above, then you are not only a subjectivist, you are a Bayesian, and you reason according to Bayes theorem

And, you may not have known that you were either a Bayesian or a subjectivist!

---

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

Small data

---

I've written before that the PMO is the world of 1-sigma; 6-sigma need not apply. Why so? Not enough data, to wit: small data.

Small data drives most projects; after all, we're not in a production environment. Small data is why we approximate, but approximation is not all bad. You can drive a lot of results from approximation.

Sometimes small data is really small.  Sometimes, we only have one observation; only one data point. Other times, perhaps a handful at best.

How do we make decisions, form estimates, and  work effectively with small data? (Aren't we told all the magic is in Big Data?)

Consider this estimating or reasoning scenario:

First, an observation: "Well, look at that! Would you believe that? How likely is that?"
Second, reasoning backward: "How could that have happened? What would have been the circumstances; initial conditions; and influences?"
Third, a hypothesis shaped by experience: "Well, if 'this or that' (aka, hypothesis) were the situation, then I can see how the observed outcome might have occurred"
Fourth, wonderment about the hypothesis: "I wonder how likely 'this or that' is?
Fifth, hypothesis married to observation: The certainty of the next outcome is influenced by both likelihoods: how likely is the hypothesis to be true, and how likely is the hypothesis -- if it is true -- to produce the outcome?

If you've ever gone through such a thought process, then you've followed Bayes Rule, and you reason like a Bayesian!

And, that's a good thing. Bayes Rule is for the small data crowd. It's how we reason with all the uncertainty of only having a few data points. The key is this: to have sufficient prior knowledge, experience, judgment to form a likely hypothesis that could conceivably match our observations.

In Bayes-speak, this is called having an "informed prior".  With an informed prior, we can synthesize the conditional likelihoods of hypothesis and outcome. And, with each outcome, we can improve upon, or modify, the hypothesis, tuning it as it were for the specifics of our project.

But, of course, we may be in uncharted territory. What about when we have no experience to work from? We could still imagine hypotheses -- probably more than one -- but now we are working with "uninformed priors". In the face of no knowledge, the validity of the hypothesis can be no better than 50-50.  

Bottom line: Bayes Rule rules! 

---

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

Actually, you can't measure it

---

We're told repeatedly: you can't manage what you can't measure. Or, you can measure anything -- everything

Actually, you can't

Measurements do the changing
There are may project domains where measurements change the thing being measured, so that the results are incorrect, sometimes dramatically so:

• Many chemical reactions or chemistry attributes
• Some biological effects
• Most quantum effects
• Most very-high or ultra-high frequency systems (VHF and UHF, to extend to micro and millimeter wave systems)
• Some optical effects

 And, of course, many human behaviors and biases are themselves biased by measurement processes

Intangibles et al
Not be left out: the affects and effects of intangibles, like leadership, empathy, the art of communication, and others. Not directly measureable, their impact is a matter of inference. Typically: imagine the situation without these influences; imagine the situation with them. The difference is as close to a measurement -- if you can call it that -- that you'll get.
 

Which all leaves the project where?

• Inference and deduction based on observable outcomes which are downstream or isolated or buffered from the instigating effects 
• Statistical predictions that may not be inference or deduction
• Bayes reasoning, which is all about dependent or conditioned outcomes
• Simulations and emulations

Bottom line: don't buy into the mantra of "measure everything". Measuring may well be more detrimental than no measurements at all

---

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

Weight of evidence

---

If you are into risk assessments, you may find yourself evaluating the data

Evaluating the data usually means testing its applicability with a hypothesis. The process usually cited in project management chapters about risk is this: 

A hypothesis is formed. And then the evidence against the hypothesis—observable data—is evaluated. If the evidence against is scant, the hypothesis is assumed valid; otherwise: false.

Guessing?

Is this guessing? The hypothesis is true because no one seems to object? After all, how much evidence is against? Enough, or have you taken the time to really look?

Most of us would agree: the evidence-against-the-hypothesis does not always fit the circumstances in many project situations.

There are many cases where you've come to the fork in the road; what to do? Famed baseball player Yogi Berra once posited: "When you come to the fork in the road, take it!"

In the PM context, Yogi is telling us that with no data to evaluate, either road is open. In Bayes* terms: it's 50:50 a-priori of there being an advantage of one road over another.

Weight of Evidence

Enter: weight-of-evidence**, a methodology for when there is some data, but yet there is still a fork in the road. In this case, we consider or evaluate each "road"—in project terms, models or suppositions—and look at the ratio of probabilities. 

• Specifically, the probability of model-1 being the right way to go, given the data, versus the probability of model-2, given the data.

Each such ratio, given the observable data, conditions, or circumstances, is denoted by the letter K: K equals the ratio of probabilities

Philosophers and mathematicians have more or less agreed on these strength ratios:

Strength ratio, aka “K”	Implied strength of evidence favoring one over the other
1 - 3	Not really worth a mention
3 -20	Positive
20 – 150	Strong
> 150	Very strong

Why form a ratio?

It's somewhat like a tug-of-war with a rope:

• Each team (numerator team and denominator team) pulls for their side.
• The analogy is that the strength of the pull is the strength or weight of evidence. Obviously, the weight favors the team the pulls the greatest. Equal weight for each team is the case for the rope not moving.
• K is representative of the strength of the pull; K can be greater than 1 (numerator team wins), less than 1 (denominator team wins), or equal to 1 which is the equal weight case.

More data

The importance and elegance of the methodology is felt when there are several data sets—perhaps from different sources, conditions, or times—and thus there are many unique calculations of "K". 

You might find you have a set of K’s: K1 from one pair of teams, but K2 from another, and so on. What to do with the disparate K’s?

Sum the evidence
The K’s represent the comparative weight of evidence in each case. Intuitively, we know we should sum up the "evidence" somehow. But, since "K" is a ratio, we really can't sum the K’s without handling (carefully) direction.

• That is: how would you sum the odds of 2:1 (K = 2) and 1:2 (K = 2, but an opposite conclusion)? We know that the weights are equal but pulling in opposite directions. Less obvious, suppose the opposing odds were 2:1 and 1:5?

Add it up

Fortunately, this was all sorted some 70 years ago by mathematician Alan Turing. His insight: 

• What really needs to happen is that the ratio's be multiplied such that 2:1 * 1:2 = 1. 
• To wit: evidence in opposite direction cancels out and unity results.

But, hello! I thought we were going to sum the evidence. What's with the multiplying thing?

Ah hah!  One easy way to multiply—70 years ago to be sure—was to actually sum the logarithms of K, 

It's just like the decibel idea in power: Add 3db to the power is the same as multiplying power by 2

Is it guessing?

You may think of it this way: when you have to guess, and all you have is some data, you can always wow the audience by intoning: "weight of evidence"!

Appendix:
Geeks beyond this point

Does everyone remember the log of 2 is 0.3? If you do, then our example of summing odds of 2:1 and 1:2 becomes: Log(2) + Log(1/2) = 0.3 – 0.3 = 0.

Of course the anti-Log of 0 is 1, so we are back at the same result we had by intuitive reasoning.

On first examination, this might seem an unusual complication, but it's a take-off on the slide rule and the older decibel (dB) measurement of power. They both multiply by summing: a 3db increase in power (or volume) means the power has doubled.

An example

What if we had four sets of evidence: odds of {10:1; 2:1; 1:2; and 1:20}. What’s the weight of evidence?

Using logarithms: log(10) = 1, log(2) = 0.3, log(1/2) = -0.3; log(20) = log(10) + log(2),
{1 + 0.3 – 0.3 – 1.3} = - 0.3 or odds of 1:2
Not all that obvious

_____________________________________________

*Bayes refers to a protocol of evaluating probabilities, given some a-priori conditions, with the idea of discovering a-posterior an underlying probable “truth”. Such discovery depends on an opportunity to gather more data, perhaps with other conditions attached.

**A practical application of the weight of evidence method is cryptanalysis. It was first developed into a practical methodology in WWII by famed mathematician Alan Turing working at storied Bletchley Park. Something of it is explained in the book "Alan Turing: The Enigma"

---

Bookmark this on Delicious

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog