Monday, April 30, 2018

Uncertainty is non-negotiable

"Uncertainty is an essential and nonnegotiable part of a forecast. .... sometimes an honest and accurate expression of the uncertainty is what has the potential to save [big things].... However, there is another reason to quantify the uncertainty carefully and explicitly. It is essential to scientific progress, especially under Bayes’s theorem."
"The Signal and the Noise: Why So Many Predictions Fail-but Some Don't" 
by Nate Silver
Now, some say: "we don't estimate; we don't forecast".
Of course, that's nonsense. Everyone estimates, if even only in their head
  • How long will it take me to write this blog?
  • How long will it take me to go to lunch?
  • How long will it take me to do almost anything I can think of? 
But, Mr Silver leads all discussion of uncertainty, estimates, and forecasts around to Bayes Theorem, which can be laid out this way as a process:
  • Formulate an issue or question or hypothesis
  • Make an early guess as to outcome
  • Experiment to gather evidence as to whether or not the guess is reasonable
  • Re-formulate based on evidence -- or lack thereof
  • Repeat as necessary 
In fact, in another part of Silver's book, he says this -- a cautionary statement for project managers:
"In science, one rarely sees all the data point toward one precise conclusion. Real data is noisy—even if the theory is perfect, the strength of the signal will vary. And under Bayes’s theorem, no theory is perfect. Rather, it is a work in progress, always subject to further refinement and testing. This is what scientific skepticism is all about."
And, one last caution from author Silver -- which reinforces the ideas of the Bayes process and also makes the point -- often ignored or overlooked --  that there is often little enough data inside one-time projects to support textbook statistical approaches:
"As we have learned throughout this book, purely statistical approaches toward forecasting are ineffective at best when there is not a sufficient sample of data to work with." 

Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!
Read my contribution to the Flashblog