Bayesian Inference in Scala

A very short introduction on Bayesian inference with a simple Scala implementation.

Originally presented as a ShiftForward Tech Talk, a series of weekly talks presented by ShiftForward employees.

Transcript

1. Bayesian Inference in Scala Rui Gonçalves ShiftForward 2015-06-19
2. What is a probability?
3. Probability Frequentist definition A probability P(A) is the proportion of outcomes where the fact A is true after we repeat the experience or observed the situation a near infinite amount of times.
4. P(A|B) = P(B|A) ⇥ P(A) P(B) Bayes’ Theorem
5. Web Images Videos News Maps Search tools More SafeSearch conditional probabilities Rui
6. Probability Bayesian definition A probability P(A) is the degree of belief in the fact or prediction A.
7. Bayes’ Theorem P(H|E) = P(E|H) ⇥ P(H) P(E)
8. Bayes’ Theorem P(H|E) = P(E|H) ⇥ P(H) P(E) Hypothesis Evidence
9. Bayes’ Theorem P(H|E) = P(E|H) ⇥ P(H) P(E) Hypothesis Evidence Prior
10. Bayes’ Theorem P(H|E) = P(E|H) ⇥ P(H) P(E) Hypothesis Evidence Prior Posterior
11. Bayes’ Theorem P(H|E) = P(E|H) ⇥ P(H) P(E) Hypothesis Evidence Prior Posterior Likelihood
12. Bayes’ Theorem P(H|E) = P(E|H) ⇥ P(H) P(E) Hypothesis Evidence Prior Posterior Likelihood Normalizing constant
13. The Dice Problem “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?”
14. The Dice Problem “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?” “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?”
15. The Dice Problem “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?” “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?”
16. The Dice Problem “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?” H 4 6 8 12 “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?”
17. The Dice Problem “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?” H 4 6 8 12 Prior P(H) 1/4 1/4 1/4 1/4 “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?”
18. The Dice Problem “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?” H 4 6 8 12 Prior P(H) 1/4 1/4 1/4 1/4 Likelihood P(E|H) 0 1/6 1/8 1/12 “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?”
19. The Dice Problem “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?” H 4 6 8 12 Prior P(H) 1/4 1/4 1/4 1/4 Likelihood P(E|H) 0 1/6 1/8 1/12 P(H) * P(E|H) 0 1/24 1/32 1/48 “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?”
20. The Dice Problem “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?” H 4 6 8 12 Prior P(H) 1/4 1/4 1/4 1/4 Likelihood P(E|H) 0 1/6 1/8 1/12 P(H) * P(E|H) 0 1/24 1/32 1/48 Posterior P(H|E) 0 4/9 3/9 2/9 “Suppose I have a box of dice that contains a 4-sided die, a 6-sided die, an 8-sided die and a 12-sided die. Suppose I select a die from the box at random, roll it, and get a 6. What is the probability that I rolled each die?”
21. The Dice Problem “Now suppose I roll the same die from before again, and this time I get an 8. What are the probabilities now?”
22. The Dice Problem “Now suppose I roll the same die from before again, and this time I get an 8. What are the probabilities now?” H Prior P(H) Likelihood P(E|H) P(H) * P(E|H) Posterior P(H|E) 4 0 0 0 0 6 4/9 0 0 0 8 3/9 1/8 1/24 9/13 12 2/9 1/12 1/54 4/13
23. In Scala: Pmf type Pmf[H] = Map[H, Double] object Pmf {
 def apply[H](hypos: Seq[H]): Pmf[H] =
 hypos.map { h => (h, 1.0 / hypos.length) }.toMap
 }
24. In Scala: Suite trait Suite[H, E] {
 def pmf: Pmf[H]
 def likelihood(ev: E, hypo: H): Double
 
 def observed(ev: E): Suite[H, E] = {
 val newPmf = pmf.map {
 case (h, prob) => (h, prob * likelihood(ev, h))
 }.normalized
 
 Suite(newPmf)(likelihood)
 }
 }
25. In Scala: Dice case class Dice(hypos: Seq[Int]) extends Suite[Int, Int] {
 val pmf = Pmf(hypos)
 
 def likelihood(data: Int, hypo: Int) =
 if (hypo < data) 0 else 1.0 / hypo
 } val prior = Dice(List(4, 6, 8, 12))
 val posterior = prior.observed(6)
26. The Euro Problem “Lukas was playing with a Belgian one-euro coin. He spun it on its edge 10 times and it ended with the heads side up 8 of those times. Intrigued, he proceed to repeat the experience 240 more times. At the end, the coin came up heads 140 times and tails 110. Is this data evidence enough to state that the coin is biased rather than fair?”
27. The Euro Problem “Lukas was playing with a Belgian one-euro coin. He spun it on its edge 10 times and it ended with the heads side up 8 of those times. Intrigued, he proceed to repeat the experience 240 more times. At the end, the coin came up heads 140 times and tails 110. Is this data evidence enough to state that the coin is biased rather than fair?” “Lukas was playing with a Belgian one-euro coin. He spun it on its edge 10 times and it ended with the heads side up 8 of those times. Intrigued, he proceed to repeat the experience 240 more times. At the end, the coin came up heads 140 times and tails 110. Is this data evidence enough to state that the coin is biased rather than fair?”
28. In Scala: Euro type CoinSide = Boolean
 val Heads = true
 val Tails = false
 
 case class Euro(pmf: Pmf[Double]) extends Suite[Double, CoinSide] {
 
 def likelihood(data: CoinSide, hypo: Double) =
 if (data == Heads) hypo else 1.0 - hypo
 } val data1 = Seq.fill(8)(Heads) ++ Seq.fill(2)(Tails) val data2 = Seq.fill(140-8)(Heads) ++ Seq.fill(110-2)(Tails)
 
 val prior = Euro(Pmf(0.0 to 1.0 by 0.001))
 val postData1 = prior.observedSet(data1)
 val postData2 = postData1.observedSet(data2)
29. Euro Posteriors Prior After 10 spins After 250 spins 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.000 0.001 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010 0.011 0.012 0.013 probability
30. Choosing a Prior So far, we’ve chosen uninformative priors - we started with no knowledge or assumptions about the hypotheses’ probabilities. What if we actually know more or believe in something else about them?
31. An Informative Prior Uniform Triangle 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.00000 0.00025 0.00050 0.00075 0.00100 0.00125 0.00150 0.00175 0.00200 probability
32. Uniform Triangle 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.00000 0.00025 0.00050 0.00075 0.00100 0.00125 0.00150 0.00175 0.00200 0.00225 0.00250 0.00275 0.00300 0.00325 probability After 10 Spins
33. Uniform Triangle 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.000 0.001 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010 0.011 0.012 0.013 probability After 250 Spins
34. Summary • Bayesian inference provides a rational way to combine prior beliefs with new evidence • Beliefs can be updated iteratively as evidence arrives • With enough data, different initial beliefs tend to converge on the same posterior
35. The End Based on Think Bayes, by Allen B. Downey http://greenteapress.com/thinkbayes https://github.com/ruippeixotog/think-bayes-scala