Monday, January 28, 2013
Ask a SME?
There are lot's of answers given, but invariably "ask a SME" (subject matter expert) is included in the answer set.
Some recents posts have been up and down on this approach, so I submit: it matters what you ask, as well as who you ask.
Afterall, an expert is stipulated to have expertise, so it's wasteful not draw them into the discussion. They certainly get the big bucks. On the other hand, opinion -- as a general matter, and like eye witnesses to history -- is not very reliable, and if that's all the SME has to offer, then I suggest you look elsewhere for information.
What the SME should bring is an expert interpretation of benchmark data that is to be adjusted and applied to the current situation.
There are tools for helping: usually we call them configurable models. To assist with the interpretation, these models have 'knobs' and 'switches' to tune the configuration and impact of one or more parameters on the current project.
Who knows how to set the knob or the switch? By and large we look to the SME for an expert opinion about the properties and parameters of the model that are under our control. However...
Without benchmark data to guide the model adjustments, the model is "uncalibrated" and thus hardly better than a guess.
If you really are going to guess, and not use calibrated estimating parameters, it's best to adopt a Bayesian methodology. That's what Bayes is all about: working with uncalibrated a priori guesses with an eye towards developing the calibration.