Monday, February 25, 2013

Becoming Bayesian: Three Books To Read

Intuitively speaking, being Bayesian is easy.  Prognosticator Nate Silver suggests that it requires following two general rules:  First, think probabilistically – evaluate different outcomes by considering their likelihoods of occurring.  Second, adapt to new information – as new data arrives, update your probabilities.

Academically speaking, being Bayesian is more difficult.  It requires a solid mathematical understanding of Bayesian inference.  After all, an academic Bayesian must perform such inference, or derive proofs concerning it.  Fortunately, Bayesians have produced a large and distinguished literature.  Unfortunately, statisticians, mathematicians and physicists have written much of it, intending it (reasonably) for consumption by other statisticians, mathematicians and physicists.  For a cognitive scientist like me this literature is intimidatingly difficult.  How does a cognitive scientist become Bayesian?

Recent research developments in my lab require that I become Bayesian, and must quickly train my students to become Bayesian as well.  After extensively searching the literature, and buying a number of classic texts, I believe that the fastest way for a cognitive scientist to become Bayesian is to read three key books.  More reading of the technical literature is required for a cognitive scientist to be Bayesian, but this additional reading will be much more rewarding if one begins as follows:

First, get excited about becoming Bayesian.  To do so, read The Theory That Would Not Die by Sharon Bertsch McGrayne.  (Full references for books are provided at the end of this blog entry.)

McGrayne’s book gives the history of Bayesian inference, beginning with Presbyterian minister Thomas Bayes’ 18th century rule for calculating, and updating, conditional probabilities.  McGrayne traces the evolution of this rule to its modern usage, providing a dizzying array of case studies.  She describes the role of Bayes’ rule for such real world problems as cracking the German WWII enigma code, searching for missing submarines and atomic bombs, and evaluating medical diagnoses.  Cognitive scientists may be particularly interested in her chapter on Alan Turing.  She also describes the continuing controversy surrounding Bayesian inference – Bayesianism is repeatedly abandoned because of concerns about some of its core assumptions (such as its subjective definition of probability) or because of the difficult calculations it entails when applied to real world problems.  Experimentally minded cognitive scientists will be interested in the biographical portraits that McGrayne provides of famous statisticians like Fischer, Pearson, and Savage.  McGrayne also chronicles the continual re-adoption and re-discovery of Bayes’ rule, because of its practical significance and because of the advent of new computational approaches. It is a theory that refuses to die – and that makes it very enticing!

Second, develop a working understanding of Bayesian inference.  To do so, read Doing Bayesian Data Analysis by John Kruschke.

McGrayne’s book gets one excited about Bayes, but provides very few technical details about what Bayesian inference involves.  Kruschke’s book solves this problem, by beginning with some basic elements of probability theory, then developing Bayes’ rule, introducing Bayesian inference for a coin-flipping scenario, and moving on to Bayesian alternatives for a number of core non-Bayesian statistical tasks.  Kruschke modestly notes that his book “is definitely not a mathematical statistics textbook in that it does not emphasize theorem proving, and any mathematical statistician would be totally bummed at the informality, dude.” However, this lack of formality is the book’s strength – it is informal, but informative.  Kruschke does not assume any statistical background at all, and builds up an understanding of Bayesian inference from first principles.  Furthermore, he replaces theorem proving with working computer programs, for which he provides generous comments.  The result: hands-on experience with core Bayesian concepts, a deep practical understanding of what Bayesian inference is all about, and an ability to actually apply Bayesian techniques to real data.  Kruschke’s book lays the foundation – and provides the courage – for moving on to classic, formal treatments of Bayesian inference.

Third, reflect on why being Bayesian might be an important foundation for cognitive science.  To do so, read Bayesian Rationality by Mike Oaksford and Nick Chater.

Classical cognitive science has evolved from logicism, which views thinking as carrying out operations in some kind of mental logic, a logic that ultimately defines judgments as being true or false.  Oaksford and Chater argue that such logicism is a mistake.  They instead argue that a better formalism for classical cognitive science is probability theory: “Logic-based approaches to cognition appeared to be viable for mathematical theorem proving and simple formal game playing, but seemed fundamentally ill-suited to representation and reasoning with real-world, common sense knowledge.”  Their book begins by considering logicism in order to contrast it with the Bayesian approach that they favor.  They develop their Bayesian move – which they name ‘the probabilistic turn’ – over several early chapters.  They then proceed to consider a number of core paradigms in the study of deductive (i.e. logicist) reasoning from the Bayesian perspective, making the case that the probabilistic approach is more appropriate.  This book is stimulating, because it links Bayesian inference to foundational assumptions in cognitive science – a link that is now leading to a lively and growing debate in the literature.  It also provides a satisfying tie-in to the Bayesian view of probability in statistics – which subjectively and controversially defines probability in terms of degree of confidence in beliefs.

To recap: if you are a cognitive scientist who is considering becoming Bayesian, then I recommend that you read these three books.  To be Bayesian you will need to read more – key authors are Cox, Jaynes, Jeffreys, and Savage – but rest assured that you will be well prepared to do so.

The Three Key Books To Read:

Below are the bibliographic details for the three books that I recommend, as well as links to information about them on

No comments:

Post a Comment