3-point estimates ... good for you!

---

Here's my three point estimate paradox:

We all know we must estimate with three points (numbers)... so we do it, reluctantly
None of us actually want to work with (do arithmetic with) the three points we estimate

In a word, three point estimates suck -- not convenient, thus often put aside even if estimated -- and most of all: who among us can do arithmetic with three point estimates?

But hey! Three point estimates are good for you ... they are the broccoli of projects

First question (are you whining? ): where does this really matter in the domain of project management?

• When adding up a budget or calculating the duration of a schedule
• When doing a risk register and someone says to multiply impact x probability

Second question: how do you go about it? Actually, you can build a couple of simple templates in a spreadsheet to do the arithmetic. Then fill in the 3 points, and presto! -- a summation or product (add or multiply) pops out.

The budget
To understand this, start with the basics:

• Suppose I have two work package budgets, one is fixed and one is uncertain. I use a 3-point estimate for the latter. I need the total budget
• As an example, with numbers:
• Fixed budget = 10
• Uncertain budget 3-pts: 5, 8, 12
• Possible totals: 10+5; 10+8; 10+12
• With no other information, pretend you are a Bayesian. You would say your "a priori" is that all 3 totals are equally possible. Thus: 1/3 * (15+18+22) = 18.3

Now, you can see it's easy to extend this to both numbers being uncertain: in that case, you would then do 9 possible totals and divide by 9.

The risk register
Multiplication is the issue with the risk register, typically I x P. Say what you will, but both I and P are uncertain, and usually P is the most uncertain. We usually have no real basis for assuming a probability since we usually have insufficient history to establish a reasonable frequency (1/P)

As an example with numbers, suppose I've been told that with some risk my budget is going to be cut by $50. For our purposes, we can take this impact as a single point "certainty". What's uncertain is whether or not there's much chance it will happen.

• Fixed impact = 50
• Uncertain probability (better thought of as a confidence of a some % or less): 10%, 30%, 50%
• Possible IxP products (this figure, or less): $5, 15, 25 (P is dimensionless, so IxP has the dimensions of I)
• Again we can do our Bayes thing (no information for the 'a priori') and come with the average IxP: 1/3 (45) = $15
• This extensible to uncertainty of both numbers wherein there will be 9 products

Monte Carlo
Is anybody going to do this on a real project with a spreadsheet? Likely not. But, this is the way a Monte Carlo simulation works and the MC Sim is common among projects run quantitatively.

---

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

A brief history of project scheduling

---

Patrick Weaver gave a talk at Primavera06 about the history of scheduling. His talk is captured in an interesting paper: "A brief history of scheduling: back to the future"

Patrick is somewhat of a historian and prognosticator on matter such as these. He also has written:

• The Origins of Modern Project Management: www.mosaicprojects.com.au/Resources_Papers_050.html
• ‘Trends In Modern Project Management - Past Present & Future’www.mosaicprojects.com.au/Resources_Papers_061.html
• ‘Seeing The Road Ahead – The art of presenting schedule data effectively’www.mosaicprojects.com.au/Resources_Papers_106.html   

So, what is the history of scheduling? I certainly remember the days of yester-year before MSProject and its adult cousin Primavera; I remember when the only scheduling tool I had was graph paper, and I remember when the mainframe scheduling tools began to replace hand-drawn bar chart schedules and simple networks.

Weaver writes:

Modern schedule control tools can trace their origins to 1765. The originator of the ‘bar chart’ appears to be Joseph Priestley (England, 1733-1804); his ‘Chart of Biography’ plotted some 2000 famous lifetimes on a time scaled chart “…a longer or a shorter space of time may be most commodiously and advantageously represented by a longer or a shorter line.”

Priestley’s ideas were picked up by William Playfair (1759-1823) in his ‘Commercial and Political Atlas’ of 1786. Playfair is credited with developing a range of statistical charts including the line, bar (histogram), and pie charts

We learn these additional nuggets:

• The science of ‘scheduling’ as defined by Critical Path Analysis (CPA) celebrated its 50th Anniversary in 2007.
• In 1956/57 Kelly and Walker started developing the algorithms that became the ‘Activity-on-Arrow’ or ADM scheduling methodology for DuPont.
• The PERT system was developed at around the same time for the US Navy's Polaris missile program 
• PERT lagged CPM (Critical Path Analysis) by 6 to 12 months (although the term ‘critical path’ was invented by the PERT team).
• Later the Precedence (PDM) methodology was developed by Dr. John Fondahl; his seminal paper was published in 1961 describing PDM as a ‘non-computer’ alternative to CPM.

Of course, one of the most profound developments was the arrival and market penetration of the low cost PC-based personal scheduling tools like MSProject. In Weaver's view that made schedulers out of everyone, but everyone is not a scheduler, or can even spell the word.

In my personal opinion, the integration of Monte Carlo tools with low cost scheduling applications like MSProject was equally profound. The MC simulation "fixes" some of the most egregious problems with PERT and the simulation idea has largely run PERT into obsolescence. The main thing that PERT does not handle well is schedule joins or merge, and the statistical merge bias that goes with it.

On the dark side, MSProject has profoundly muddled the whole idea of the WBS. The fact that one of the fields in MSP is called "WBS" is most unfortunate. There are a whole generation of project analysts who believe they created a WBS by the simple act of making that field visible. Not so: schedule is the "verbs" of the project; WBS is the "nouns". Together, WBS + schedule = project narrative.

Now, in Weaver's view, we return to the future:

The changing role of the scheduler has been almost as interesting:
• The mainframe era saw scheduling as:

• A skilled profession
• Central to the success of projects

• Then came the PCs…… everyone and no-one was a ‘scheduler’ in the 1980s and 90s
• However, in the 21st century, the new ‘enterprise’ era sees scheduling as:

• A skilled profession
• Central to the success of projects

---

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

Ask a SME... or Ask a SME?

---

It seems like the PM blog sphere talks constantly of estimates. Just check out #noestimates in Twitter-land. You won't find much of substance among thousands of tweets (I refrain from saying twits)

Some say estimates are not for developers: Nonsense! If you ask a SME for an estimate, you've done the wrong thing. But, if you ask a SME for a range of possibilities, immediately you've got focus on an issue... any single point estimate may be way off -- or not -- but focusing on the possibilities may bring all sorts of things to the surface: constraints, politics, biases, and perhaps an idea to deep-six the object and start over with a better idea.

How will you know if you don't ask?

Some say estimates are only for the managers with money: Nonsense re the word "only". I've already put SMEs in the frame. The money guys need some kind of estimate for their narrative. They aren't going to just throw money in the wind and hope (which isn't a plan, we all have been told) for something to come out. Estimates help frame objectives, cost, and value.

To estimate a range, three points are needed. Here's my three point estimate paradox:

We all know we must estimate with three points (numbers)... so we do it, reluctantly
None of us actually want to work with (do arithmetic with) or work to (be accountable to) the three points we estimate

In a word, three point estimates suck -- not convenient, thus often put aside even if estimated -- and most of all: who among us can do arithmetic with three point estimates?

One nice thing about 3-points and ranges, et al, is that when applied to a simulation, like the venerable Monte Carlo, a lot washes out. A few big tails here and there are no real consequence to the point of the simulation, which is find the central value of the project. Even if you're looking for a worst case, a few big tails don't drive a lot.

But, here's another paradox:

We all want accurate estimates backed up by data
But data -- good or bad -- may not be the driver for accurate estimates

Does this paradox let SMEs off the hook? After all, if not data, then what? And, from whom/where/when?

Bent Flyvbjerg tells us -- with appropriate reference to Tversky and Kahneman -- we need a reference class because without it we are subject to cognitive and political maladies:

Psychological and political explanations better account for inaccurate forecasts.
Psychological explanations account for inaccuracy in terms of optimism bias; that is, a cognitive predisposition found with most people to judge future events in a more positive light than is warranted by actual experience.
Political explanations, on the other hand, explain inaccuracy in terms of strategic misrepresentation.

So that's it! A conspiracy of bad cognition and politics is what is wrong with estimates. Well, just that alone is probably nonsense as well.

Folks: common sense tells us estimates are just that: not facts, but information that may become facts at some future date. Estimates are some parts data, some parts politics, some parts subjective instinct, and some parts unknown. But in the end, estimates have their usefulness and influence with the SMEs and the money guys.

You can't do a project without them!

---

Bookmark this on Delicious

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

The GIGO thing... revisited

---

Garbage in; garbage out... GIGO to the rest of us

Re GIGO, this issue is raised frequently when I talk about Monte Carlo simulation, and the GIGO thing is not without merit. These arguments are made:

• You have no knowledge of what distribution applies to the uncertainties in the project (True)
• You are really guessing about the limits of the three point estimates which drive the simulation (partly true)
• Ergo: poor data information in; poor data information out (Not exactly! the devil is in the details)

Here are a few points to consider (file under: Lies, damn lies, and statistics):

First: There's no material consequence to the choice of distribution for the random numbers (uncertain estimates) that go into the simulation. As a matter of fact, for the purposes of PM, the choices can be different among tasks in the same simulation.

• Some analysts choose the distribution for tasks on the critical path differently than tasks for paths not critical.
• Of course, one of the strengths of the simulation is that most scheduling simulation tools identify the 'next most probable critical path' so that the PM can see which path might become critical.

Re why the choice is immaterial:  it can be demonstrated -- by simulation and by calculus -- that for a whole class of distributions, in the limit, their infinite sum takes on a Normal distribution.

• X(sum) = X1 + X2 + X3 +.... +XN, where N is a very large number, X(Sum) is Normal, no matter what the distribution of X is

As in "all roads lead to Rome", so it is in statistics: all distributions eventually lead to Normal. (those that are practical for this purpose: single mode and defined over their entire range [no singularities]),

To be Normal, or normal-like, means that the probability (more correctly, the probability density function, pdf) has an exponential form. See: Normal distribution in wikipedia

We've seen this movie before in this blog. For example, few uniform distributions (not Normal in any respect), when summed up, the sum took on a very Normal appearance. And, more than an appearance, the underlying functional mathematics also became exponential in the limit.

Recall what you are simulating: you are simulating the sum of a lot of budgets from work packages, or you are simulating the sum of a lot of task durations. Therefore, the sim result is summation, and the summation is an uncertain number because every element in the sum is, itself, uncertain.

All uncertain numbers have distributions. However, the distribution of the sum need not be the same as the distribution of the underlying numbers in the sum. In fact, it almost never is. (Exception: the sum of Normal is itself Normal) Thus, it really does not matter what distribution is assumed; most sim tools just default the Triangular and press on.

And, the sim also tends to discount the GIGO (garbage in/garbage out) problem. A few bad estimates are likewise immaterial at the project level. They are highly discounted by their low probability. They fatten the tails a bit, but project management is a one-sigma profession. We largely don't care about the tails beyond, certainly not six sigma!

Second: most folks when asked to give a 3-point simply take the 1-pointer they intended to use and put a small range around it. They usually resist giving up anything so the most optimistic is usually just a small bit more optimistic than the 1-pointer; ....  and then they put something out there for the most pessimistic, usually not giving it a lot of thought.

When challenged, they usually move the most-likely 1-pointer a bit to the pessimistic side, still not wanting to give up anything (prospect theory at work here). And, they are usually reluctant to be very pessimistic since that calls into question the 1-pointer (anchor bias at work here). Consequently, you get two big biases working toward a more optimistic outcome than should be expected

Third: with a little coaching, most of the bias can be overcome. There is no real hazard to a few WAG because unlike an average of values, the small probability of the tails tends to highly discount the contribution of the WAGs. What's more important than reeling a few wild WAGs is getting the 1-pointer better positioned. This not only helps the MC Sim but also helps any non-statistical estimate as well.

Bottom line: the garbage, if at the tails, doesn't count for much; the Most likely, if a wrong estimate, hurts every methodology, whether statistical or not.

---

Bookmark this on Delicious

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

Fussing about PERT and Monte Carlo methods

---

A note from a reader of some of my stuff on slideshare.net/jgoodpas:

"... I just read your slide presentation on using stats in project management and am a bit confused or perhaps disagree ... that using Monte Carlo will improve the accuracy of a schedule or budget based on subjective selections of activity or project
a) optimistic, realistic and pessimistic ($...Time) values,
b) a probability distribution (3 point triangular) and
c) arbitrary probability of occurrence of each value.

If I understand your presentation Central Limit Theory (CLT) and the Law of Large Numbers (LLN) when applied using Monte Carlo simulation states that accuracy is improved and risk more precisely quantified.

It seems to me that this violates a law that I learned a long time ago...garbage in-garbage out. Is this true? ...."

I replied this way about accuracy:

• Re 'accuracy' and MC Sim (Monte Carlo simulation): My point is (or should have been) the opposite. The purpose of MC Sim is to dissuade you that a single point estimate of a schedule is "accurate" or even likely. Indeed, a MC Sim should demonstrate the utility of approximate answers
• Project management operates, in the main, on approximations... ours is the world of 1-sigma (Six Sigma is for the manufacturing guys; it doesn't belong in PM).
• Re the precision/accuracy argument: My favorite expression is: "Measure with a micrometer, mark with chalk, and cut with an axe", implying that there is little utility in our business (PM) for precision, since most of our decisions are made with the accuracy of chalk or axes.
• A MC Sim gives you approximate, practical,and actionable information about the possibilities and the probabilities (all plural) of where the cost or schedule is likely to come out.
• And, to your point, the approximate answers (or data) should be adequate to base decisions under conditions of uncertainty, which is for the most part what PMs do.

So, what about MC Sim itself?

• Re your contention: " ... selecting the distribution, optimistic, pessimistic and realistic values must be accurate for Monte Carlo to provide more accurate results." Actually, this contention is quite incorrect and the underlying reason it is incorrect is at the core of why a MC Sim works.
• In a few words, all reasonable distributions for describing a project task or schedule, such as BETA (used in PERT), Normal (aka Bell), Rayleigh (used in many reliability studies, among others), Bi-nominal (used in arrival rate estimates), and many others are all members of the so-called "exponential family" of distributions. You can look up exponential distributions in Wikipedia.
• The most important matter to know is that, in the limit when the number of trials gets large, all exponentials devolve to the Normal (Bell distribution).
• Thus, if the number of trials is large, the choice of distribution makes no difference whatsoever because everything in the limit is Normal.
• If you understand how to do limits in integral calculus, you can prove this for yourself, or you can look it up on Wikipedia

How large is large?
It depends.

• As few as five sometimes gives good results (see image) but usually 10 or more is all you need for the accuracy needed for PM.
• Consequently, most schedule analysts pick a triangular distribution, which does not occur in nature, but is mathematically efficient for simulation. It is similar enough to an exponential that the errors in the results are immaterial for PM decision making purposes.
• Some other picks the uniform distribution like shown in the image; again for mathematical convenience

Should I worry about 'when' and 'where'?

• Now the next most important matter to know is that a sum of exponentials (BETA, Rayleigh, whatever) -- like would be the case of a sum of finish-to-start tasks -- has the same effect as a number of trials.
• That is, if the project is stationary (does not matter when or where you look at it), then a string of repeated distributions (as would be in a schedule network) has the same characteristics as a single distribution tried many times.
• And, each member of the string not be the same kind of distribution, though for simplicity they are usually assumed to be the same.
• Thus, whether it's one distribution tried many times, or one distribution repeated many times in a string of costs or tasks, the limiting effect on exponentials is that the average outcome will itself be a random variable (thus, uncertain) with a distribution, and the uncertainty will be Normal in distribution.

And, about that garbage

• The usual statement about GIGO assumes that all garbage has equal effect: Big garbage has big effects; and little garbage has little effects; and too much garbage screws it all up.
• This is certainly the case when doing an arithmetic average. Thus, in projects, my advice: never do an arithmetic average!
• However, in a MC SIM, all garbage is not equal in effect, and a lot of garbage may not screw it up materially. The most egregious garbage (and there will be some) occurs in the long tails, not in the MLV.
• Consequently, this garbage does not carry much weight and contributes only modestly to the results.
• When I say this I assume that the single point estimates, that are so carefully estimated, are one of the three estimates in the MC Sim, and it is the estimate that is given the largest probability; thus the MLV tends to dominate.
• Consequently, the egregious garbage is there, has some effect, but a small effect as given by its probability.

What about independence and correlations?

• If the distributions in the MC Sim are not independent then results will be skewed a bit -- typically, longer tails and a less pronounced central value. Independence means we assume "memoryless" activities, etc so that whether a string of tasks or one task repeated many times, there is no memory from one to the other, and there is no effect on one to the other other than in finish-to-start there will be effects on the successor start time.
• Correlations are a bit more tricky. In all practical project schedules there will be some correlation effects due to dependencies that create some causality.
• Again, like loss of independence, correlation smears the outcome distribution so that tails are longer etc.
• Technically, we move from the Normal to the "T" distribution, but effects are usually quite inconnseqential. 

Where to get the three points if I wanted them?

• My advice is in the risk management plan for the project, establish a policy with two points:
  1. All tasks on the calculated single point critical path will be estimated by experts and their judgement on the task parameters go in the sim
  2. All other tasks are handled according to history re complexity factors applied to the estimated MLV (most likely value, commonly the Mode): that is, once the MLV for a non-critical task is estimated, factors set by policy based on history, are applied to get the estimates of the tails.
• Thus, a "complex task" might be: optimistic 80% of MLV and pessimistic 200% of MLV, the 80% and 200% being "policy" factors for "complex" tasks.
• This is the way many estimating models work, applying standard factors taken from history. In the scheduling biz, the issue is not so much accuracy -- hitting a date -- as it is forecasting the possibilities so that some action can be taken by the PM in time to make a difference.
• Providing that decision data is what the MC Sim is all about. No other method can do that with economic efficiency

PERT v MC Sim
Now, if the schedule is trivial, there is no real difference in the PERT and MC Sim. Take, for example, a dozen tasks in a tandem string of finish-to-start.

• The PERT answer to the expected value of the overall duration and the MC Sim will be materially identical.
• The sum of the expected values of each task is the expected value of the sum. So, the SIM and the PERT give identical answers insofar as an "accurate" expected value.
• But that's where it ends. PERT can give you no information about the standard deviation of the outcome (the standard deviation contains the reasonable possibilities about which you should be worried), nor the confidence curve that lets you assess the quality of the answer. 
• If your sponsor asks you run the project with an 80/20 confidence of finishing on time, where are you going to go for guidance? Not to PERT for sure.
• And, if your sales manager says bid the job at 50/50, you have the same issue... where's the confidence curve going to come from?
• And, most critically: PERT falls apart with any parallelism in the schedule in which there is no slack.

Thus, some 30 years ago PERT was put on the shelf, retired by most scheduling professionals

I'm almost out of gas on this one, so I'll end it here.

---

The image shows the outcome of 5 independent uniform distribution summed, as in a finsh-to-start schedule sequence.

---

Bookmark this on Delicious

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

Who knew? Estimates are not free!

---

The title of this blog says it all: Who knew? Estimates are not free!

In a recent email blast from Mike Cohn, who -- by the way, wrote the book: "Agile Estimating and Planning" -- we learn this wisdom -- hopefully, not hearing it for the first time:

I have no problem with a boss who asks a team to provide an estimate for how long a huge set of product backlog items will take to deliver.

There could be hundreds of items – perhaps even thousands, and I have no issue with the boss asking for an estimate.

I will be happy to provide a boss with that information.

So long as the boss understands that information has a cost.

And, so there you have it: No free lunch!
And, hear this: real agilists do estimates! (Mike is as real as they come)

From whence they come
Estimates are not something one has on the tip of their tongue, like that oft-practiced elevator speech about the benefit of this project.

No, common sense -- which we all have some measure of -- tells us good estimates take a bit of time, and that time should be set aside in the project schedule and budget. And estimating, and its close cousin forecasting -- which some define as assembling and integrating estimates -- should not be expected to be "other duties as assigned", only to be handled in the "white space" of the project day.

Nope, estimating and forecasting are legitimate tasks, and there's none better to do them than some of the project's finest -- our corps of SMEs.

And more
And, as if all this is not enough to spin heads, real agilists also use Monte Carlo simulation to develop and validate estimates!

Gasp! Statistics and agile... can this be real?

But yes, it's real! Tony Magennis tells us so in his small book: "Forecasting and Simulating Development Projects" (in spite of the fact he tries to get your attention with #NoEstimates Forecasting)

---

Bookmark this on Delicious

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

About micrometers, chalk, and axes

---

I'm nearing my one year anniversary working on a volunteer project to restore this Lockheed PV-1, circa 1942, short-range bomber and reconnaissance aircraft.

Some assembly is required!

Frankly, I'm trained as an electrical engineer -- Georgia Tech, et al -- but there's not much electrical/electronic on this aircraft. So, I've had to recall 7th grade metal shop: brakes, shears, rivets, etc.

One thing I've learned from my (mechanical engineer) team leader -- also a Georgia Tech wreck as it turns out -- is the 3-step process of mechanics (who knew? You can't make this stuff up!):

1. Measure with a micrometer
2. Mark it with chalk (in our case, Sharpie marking pens), and
3. Cut it with an axe! (in our case, shears of one kind or another, or (gasp!) the band saw)

Ooops! This very process shows up in project management. See: accuracy vs resolution and the spreadsheet -- estimates that are just a bit better than a guess are entered into cells with zillions of digits of resolution. Nonsense.

And, we see it as domain distortion when the defined process crowd with six sigma control limits wants to port their ideas into the domain of the one sigma program office. Again, nonsense (except for the problems defining process in six sigma that is quite portable and worthy)

So it is that I have to return repeatedly in my risk management courses to why/how the precision of a estimate in a work package -- or the less frequent but more troublesome estimate outlier -- more or less washes out at the PMO level in a Monte Carlo simulation of all the work package effects.

Monte Carlo is the 3-step process writ large:

1. Agonize over every work package estimate (micrometer, but the WP manager does this step)
2. Enter all the WP data into a simulation package using 3-point estimates and benchmarks (chalk, likely applied by the project analyst)
3. Run the simulation to get the "big picture" (axe)

What I find is those in the PMO wielding the axe seem to get inordinately excited about an outlier estimate here or there. If you're the axe-person, don't sweat the micrometers!

---

Bookmark this on Delicious

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!

Read my contribution to the Flashblog

How old is the wine? Exposing risk

---

Mike Cohn has a posting on a "risk exposure burn down chart".

It's a take off on the more familiar object and task burn down chart applied universally in agile. Mike's chart is pretty straight forward: Estimate the risk impact (as modified by likelihood) and call this 'risk exposure'. When the risk is mitigated, or passes by without impact, the exposure is burned off.

Nice. But is this old wine in a new bottle?

Exposure
Some years ago -- actually, many years ago -- I took over a department of about 85 engineers, mostly system engineers, with a budget (then) of about $10M annually. One of the first things my new boss asked me to do was to figure out what my (our) exposure was.

Exposure? I understood this to be a question about risk, but what's the metric here, and how do you measure for data? For instance, the metric could be scope exposure, schedule exposure (as in Mike's chart), quality, or budget. Or, it could be some other "measure of effectiveness" -- MoE -- that was applied in the various programs.

In the moment what my boss was asking for was an estimate of the budget risk (impact and likelihood) that was lurking in the work package assigments to which my department engineers had committed themselves.

That $10M department budget was all signed out to various projects for work that had to be done, but what if the work could not be done for $10M? How much risk was there for me as department director and for the various program managers who needed the work done?

So, I had my metric: monetized budget; and I had a way to measure it.

Management and Monte Carlo
What I did, of course, was build a version of a risk register, much like Mike's but with some differences that account for risk management: for each work package, I asked my WP managers to give me a 3-point estimate (no single points allowed!) of the  remaining effort required to get 'done' (cost-to-complete according to a completion standard more or less understood by all -- a similiar idea exists in agile, of course).

A Monte Carlo simulation of the WP estimates gave me -- actually, my business analyst -- a single point measurement (expecgted value statistic) to compare with the single point budget. Each statistic value is a deterministic number so I can do arithmetic on them. Of course, the optimistic, ML, and pessimistic numbers are random numbers -- values in a distribution -- so I can't do arithmetic with them, except by simulation. Thus, their columns are not added.The difference in these two points was my (our) exposure measured as the risk weighted expected value of the portfolio.

The spreadsheet below illustrates how it was done:

---

Bookmark this on Delicious

What's wrong with risk management?

---

Here's one of those provocative titles you see from time to time. This one, however, is from Matthew Squair (formerly DarkMatter and now Critical Uncertainties), and so it carries a bit of cache:

All You Ever Thought You Knew About Risk is Wrong

And, so getting to the points, there are two:

• Point 1 is The St Petersburg Paradox, and
• Point 2 is the error between time-parallel and time-sequenced risks (See also: John von Neumann's et al Monte Carlo simulation)

Point 1
In a word or two, it's a matter of utility (that is, perceived value vs risk) and the extremity of risk vs affordability

The St Petersburg Paradox, first posed by 18th century mathematician Daniel Bernoulli, is that in the face of constant expected value, we can not expect gamblers (or decision makers) to be immune to the potential for catastrophe between one risk scenario and another. The fact that scenario expected values are perceived differently, even though they are not, is the bias behind the idea of utility.

Example: if your project has a 10% chance of costing the business a million dollar loss on a project failure, is that any different than a project with a 1% chance of costing the business a ten million dollar loss? Or, another project with 0.1% chance of putting the business out of business with $100M in losses? At some point, there is capitulation: enough is enough. Sponsors won't take the risk, even though the expected value is constant--($100K)--and modest and equal in all three situations.

Thus, someone makes a utility judgment, applying their perceived value (fear in this case) to an otherwise objective value and coming up STOP! Expected value, as a calculated statistic, obscures the extreme possibility that may render the project moot.

Point 2:
We've all been taught that rolling a die 100 times in sequence is statistically equal to rolling 100 die one time. This property is called ergodicity--meaning statistics are stationary with time...it doesn't matter when you do the rolling, the stats come up the same.

This idea that parallel and sequential events are statistically equivalent underlies the validity of the Monte Carlo simulation (MCS). We can do a simulation of a hundred project instances in parallel and expect the same results as if they were done in sequence; and, the average outcome will be the same in both cases.

But, what about the circumstances that afflict projects that are not time stationary: those circumstances where is does matter when in time you do the work? There's always the matter of resource availability, timing of external threats (budget authorization, regulatory changes), and perhaps even maturity model impacts if the project is long enough.

Consequently, when doing the MCS, it's a must to think about whether the circumstances are ergodic or not. If not, and if material to the outcome, then the MCS must be leavened with other reserves and perhaps major risk strategies must be invoked.

Summary
Maybe everthing you know about risk management is not quite right!

---

Bookmark this on Delicious

Monte Carlo: Garbage in; garbage out?

---

For many years, I've preached the benefits of the Monte Carlo simulation (MCS). For many reasons, it's superior to other analysis paradigms. In network analysis, for instance, it handles the parallel join or 'gate' as no other method will.

In fact, it's this very capability that renders the Monte Carlo superior to other risk adjusted ideas, like the PERT method in scheduling. PERT suffers from an inability to handle the 'merge bias' at gates because PERT does not handle the mathematics of multiplying distributions (required at gates); PERT only provides a means to calculate a statistic for each task.

Architecturally, gates are the weakest construct in schedules, so any failure to handle them is a show stopper in my mind

Quantitative methods schedule from John Goodpasture

But, my students ask me invariably: how can the MCS be any better than the 3-point estimates that go into it; if the 3-pointers are guesses, isnt' the whole MCS thing just a crap shoot?

And, the second thing they ask is: who's got the time to triple the estimating problem (from the most likely single point estimate to the trio of optimistic, pessimistic, and most likely) especially in a network of many hundreds, if not thousands, of tasks?

My answer is this: it depends; it depends on whether or not your are the work package leader concerned for a handful of tasks, or the project manager concerned for a network of hundreds (thousands in some cases) of tasks.

If the former, then you should not guess; you should take the time to consider each estimate, based on benchmarks, so that each estimate is has some auditable calibration to the benchmark.

But if the latter, then let the Central Limit Theorem do the work. A few bad estimates, indeed a lot more than a few in most cases, have negligle impact on the MCS results. In other words, the good, the bad, and the ugly tend to cluster around a central value--a phenomenon called central tendency. Thus, at the PM level, you can live without completely solid calibration. Only a few estimates need to be pretty well thought out vis a vis the O,P,ML triplett.

This may sound like heresey to the calibration crowd, but as a politician recently said: arithmetic! Actually, calculus is the better word, since we have to deal with functions. And it's calculus that gives us the tools to understand that a few bad estimates, even really bad estimates, are largely discounted for effect by the integrated effects of all the other estimates. So, let's not get uptight about a few bad eggs!

Nevertheless, to do a MCS, you do need estimates of O, P, and ML. Where do they come from?  All the tools allow for defaulting in a trio to every task. Is that a good idea?

Here's my counsel:

• Take the time to estimate the most likely critical path. The MCS tool will identify other near-critical paths, and may even find an alternate path that is critical (not the one you thought)
• Establish, in the risk management plan, a policy about the defaults values (for % of ML)
  The policy may have several elements: hardware, software, administration, and process tasks. Each of these will have hard, medium, and easy tasks.
  The policy can be a matrix of O, ML, and P defaults for each (Example: for hard software, the policy is to estimate O = 80% ML and P = 200% ML).
  These generally come from experience, and that means from actual results elsewhere, so the defaults are not just picking values out of thin air....there's usually back-up

---

Bookmark this on Delicious

The marriage of Monte Carlo and Earned Value

In my risk management classes, I position EVM as a risk management tool because of its usefulness as a forecast tool. There're no facts about the future according to Dave Hulett, only estimates. And estimates are where the risks are.

I also instruct Monte Carlo simulations [MCS] as a forecast tool.

I get this question from students a lot: 'How do I integrate the forecast results from these two tools?', or 'How do I use both of these tools in my project; do I have to chose between them?'

It's a reasonable question: EVM is a system of linear deterministic equations; Monte Carlo is a simulation of a stochastic process described by random variables in functional relationships. How should analysts span the deterministic and the stochastic, and the process model and the linear equation model?

The answer lies in the fact that both systems, EVM and MCS, can be used to predict the EAC--estimate at complete. And, EVM practices do allow for re-estimation of remaining work as an alternate approach to the linear equation forecast. Running a simulation is one way to do the re-estimation.

Reconciliation of the calculated EAC from the EVM and the simulation EAC from the MCS means reverse engineering the required efficiencies for utilization of cost and schedule going forward.

The equation at work is this one:
EAC = AC + [BAC - EV] / CPI

The facts are AC (actual cost), BAC (budget at completion), and cumulative EV. The cumulative CPI is a historical fact, but it's only an estimate of future performance. That's where the MCS comes in. MCS results may shape our idea about this estimate when compared to the EVM linear equation calculations.

Of course the MCS results are not a single point number that fits conveniently into a linear equation; the results are a distribution of possibilities. How to deal with this?

There are two approaches:
First, the expected value of the MCS distribution could be used as a good estimate of the EAC. Expected value is deterministic, so it can be used in a linear equation with other deterministic values

Second, the MCS usually provides a cumulative probability curve, the so-called "S Curve", from which a single point number can be picked according to a project policy or doctrine about how to pick.

Here's how the second approach might look. The project policy about risk aversion--that translates into picking a point on the confidence curve--is usually documented in the risk management plan.  Using the policy guidance, an EAC is picked from the MCS confidence, and then compared to the EAC calculated by EVM equations.

Once the MCS value is determined, the equation above is reworked to solve for the future CPI. Now you have two CPI's: one from the EVM estimate, and one from the MCS re-engineering. What to do now?

The conservative thing is to pick the worst case. The management thing is to determine what needs to be done or changed to bring the CPI into an acceptable range, and then do it.

 Bookmark this on Delicious  

Are you on LinkedIn?    Share this article with your network by clicking on the link.

Throwing darts at Monte Carlo

Kailash Awati writes the blog Eight to Late; in a recent post, he makes a humorous but quite informative explanation of the Monte Carlo process by analogy with a drunk throwing darts at a target.  Entitled "A drunkard's dartboard: an intuitive explanation of Monte Carlo methods", he addresses a paradox that may have occurred to many, to wit:

.... that one can get accurate answers via random numbers. 

Permit me to comment:
'accurate' only in the sense that a Monte Carlo will provide distribution of possible outcomes [answers] weighted by the probability [strictly: the probability density.  That is, probability per unit of output], and the accuracy of this range of answers is highly dependent on the inputs provided to the simulation tool.  So, it's still a matter of guarding against "garbage in/garbage out"

Nevertheless, in his clever post, Awati uses simple geometric shapes to demonstrate the answer to that paradox. In the end, I was convinced!

  Bookmark this on Delicious  

Are you on LinkedIn?    Share this article with your network by clicking on the link.

All things bell shaped

Chapter 14 from the GAO Cost Estimating Manual on "Cost Risk and Uncertainty" is a good read, easily understood, and very practical in its examples.  Here's one illustration that I particularly like.  When you look at it, it's understood in a moment that the repeated random throw of two dice generates a probability density function [PDF] that has a bell-shape curve that is tending towards a true Normal distribution.

Statisticians call this phenomenon the Central Limit Theorem: random occurrences over a large population tend to wash out the asymmetry and uniformness of individual events.  A more 'natural' distribution ensues.  The name for it is the Normal distribution, more commonly: the bell curve.

Here's what it looks like to a project manager.  Notice that regardless of the distribution of cost adopted by  work package managers for each individual work package, in the bigger picture at the summation of the WBS there will tend to be a bell-shaped variation in the WBS budget estimate.  In part, use of these ideas addresses the project manager's need to understand the parameters of variation in the project budget as evidenced by the esitmates of WBS.  This diagram is (again) from Chapter 14 of GAO's manual:

If the risk analyst generates these data from a simulation, like a Monte Carlo simulation, then the numeric statistics like variance and standard deviation are usually reported, along with the cumulative probability more commonly called the "S" curve.  In the diagram, on the right side, we see the cumulative curve plotted and labeled on the vertical axis as the confidence level.  With a little inspection, you will realize that the cumulative curve is just the summation of the probabilities of the bell curve that is adjacent on the left.

The GAO manual, and especially Chapter 14, has a lot more information that is well explained.  Give it a read.

Are you on LinkedIn? Share this article with your network by clicking on the link.

Top Five Ideas -- Statistics for Project Management

Do you love statistics? Probably not!

Nevertheless, they are present in every project and so a few minutes to grasp the top five ideas is time well spent.

In my presentation (at the slideshare link), adapted from my book "Quantitative Methods in Project Management", I talk about Monte Carlo simulation, the Central Limit Theorem, the Law of Large Numbers, probability distributions, and the merge bias problems in schedules.


Click here for a slideshare to show you the ropes on the top five ideas for statistics that help the project manager.

Are you on LinkedIn? Share this article with your network by clicking on the link.