Though it's now almost 18 months old, this paper (15-page pdf file) by Bernardo Huberman and others at HP Labs lays out a clever layer of innovation that purports to improve the accuracy of 'pure' prediction markets. Their introduction is intriguing in that it implies real potential for scientific discipline in prediction and knowledge management:
"While in the physical and biological sciences the discovery of strong laws has enabled the prediction of future scenarios with uncanny accuracy, in the social sphere no such accurate laws are known. To complicate matters further, in social groups the information relevant to predictions is often dispersed across people, making it hard to identify and aggregate it. Thus, while several methods are presently used in forecasting, ranging from committees and expert consultants to aggregation techniques such as the Delphi method... the results obtained suffer in terms of accuracy and ease of implementation... If [prediction] markets are large enough and properly designed, they can be more accurate than other techniques for extracting diffuse information, such as surveys and opinions polls."
But pure prediction markets, they note, suffer from some well-known flaws. (But as I've noted before, these don't necessarily make them any more expensive or less valid than the flawed methods they replace.) They go on:
"...information markets... tend to suffer from information traps... illiquidity... manipulation... and lack of equilibrium... These problems are exacerbated when the groups involved are small and not very experienced at playing in these markets. Even when possible, proper market design is very expensive, fragile, and context specific."
They then get to the essence of the approach:
"...a method of harnessing the distributed knowledge of a group of individuals by using a two-stage mechanism. In the first stage, an information market is run among members of the group in order to extract risk attitudes from the participants, as well as their ability at predicting a given outcome... In the second stage, individuals are simply asked to provide forecasts about an uncertain event, and they are rewarded according the accuracy of their forecasts. These individual forecasts are aggregated using [a] nonlinear function and used to predict the outcome... this [approach] vastly outperforms both the imperfect market and the best of the participants."
They add to this another function designed to separate out the effect of public vs. private information, improving prediction results even further.
My take: While such accuracy may be essential for some applications (e.g., forecasting demand for a well-established product), it appears to require significant time and expertise to do right. (The math is dense, to say the least.) An even more important implication - especially for resilient strategic planning - is that it appears to lack a dynamic component: a way for a group to continuously seek, refine and assimilate new information about a complex and changing market or competitive environment. Finally, it seems to rely on a closed system, where 'good forecasters' are identified up-front on an assumption that their forecasting ability is generic and not likely to be found in otheres. Not enabling new, marginal participants with correct (but possibly 'heretical') information to enter and influence the process can be deadly. In other words, the approach, while clever and useful for static applications, appears to lack precisely the openness and adaptability essential for anticipating the emerging likelihood and potential impact of discontinuous change, (i.e., sudden, unprecedented surprises). Being able to quickly see inflection points developing in a measure of conventional wisdom is often more important than achieving the last decimal point of forecast probability.
Comments