Dark Data: Why What You Don't Know Matters

Description

A practical guide to making good decisions in a world of missing data In the era of big data, it is easy to imagine that we have all the information we need to make good decisions. But in fact the data we have are never complete, and may be only the tip of the iceberg. Just as much of the universe is composed of dark matter, invisible to us but

David J. Hand is emeritus professor of mathematics and senior research investigator at Imperial College London, a former president of the Royal Statistical Society, and a fellow of the British Academy. His many previous books include The Improbability Principle, Measurement: A Very Short Introduction, Statistics: A Very Short Introduction, and Principles of Data Mining.

"[A] penetrating study of missing ('dark') data and its impacts on decisions-skewing stats, enabling fraud, embedding inequity and triggering preventable catastrophes. Advocating 'data science judo,' Hand offers expert training, from recognizing when facts are being cherry-picked to designing randomized trials. A book illuminating shadowed corners in science, medicine and policy."---Barbara Kiser, Nature "A tour de force. . . . Hand is a good and able guide to take us through the many aspects of dark data that are potentially skewing our understanding of real world observations and potential scientific breakthroughs. He writes in an accessible and understandable way too."---Simon Cocking, Irish Tech News "Well-written and accessible."---Tim Harford, Undercover Economist "You need to read [Dark Data], and be convinced by David's reasoning and his examples of cases in which unseen or unreported data play a critical and sometimes even a fatal role. You are likely to walk away with the feeling that the term dark data is indeed a very effective one to arouse both curiosity and suspicion, mixed with happiness that finally a great term was coined by a statistician-and sadness that the statistician is not you."---Xiao-Li Meng, IMS Bulletin "An exploration of a major problem in data analysis with an attempt of classification, analysing causes, mechanisms, and to some extent also suggest mitigations."---Adhemar Bultheel, European Mathematical Society "An excellent guide to the many reasons for caution in interpreting data."---Diane Coyle, Enlightened Economist