Statistical Rethinking - Lecture 03

Lecture 03, Chapter 4, for "Statistical Rethinking: A Bayesian Course with R Examples".

Transcript

1. Week 2: Linear Models Richard McElreath Statistical Rethinking
2. Predictive checks • Posterior probability never enough • Even the best model might make terrible predictions • Also want to check model assumptions • Predictive checks: Can use samples from posterior to simulate observations • NB: Assumption about sampling is assumption
3. 0 1000 3000 number of water samples Frequency 0 3 6 9 0.89 0 1000 3000 number of water samples Frequency 0 3 6 9 0.38 0 1000 3000 number of water samples Frequency 0 3 6 9 0.64 0 1000 3000 number of water samples Frequency 0 3 6 9 0.000 0.010 0.020 probability of water probability 0 0.5 1 (A) p = 0.38 (B) p = 0.64 (C) p = 0.89 A B C Merged Figure 3.4 0 1000 3000 Frequency 0 3 6 9 0.89 0 1000 3000 number of water samples Frequency 0 3 6 9 0 1000 3000 number of water samples Frequency 0 3 6 9 0.64 0 1000 3000 number of water samples Frequency 0 3 6 9 0.000 0.010 0.020 probability of water probability 0 0.5 1 (B) p = 0.64 (C) p = 0.89 A B C Merged
4. Posterior predictions • One line of code • Will get harder, later. But strategy remains the same. 0 1000 3000 number of water samples Frequency 0 3 6 9 0.89 number of water samples 0 3 6 9 0 1000 3000 number of water samples Frequency 0 3 6 9 0.64 0 1000 3000 number of water samples Frequency 0 3 6 9 0.000 0.010 0.020 probability of water probability 0 0.5 1 (B) p = 0.64 (C) p = 0.89 A B C Merged 'ĶĴłĿIJ Ƌƌ 4JNVMBUJOH QSFEJDUJPOT GSPN UIF UPUBM QPTUFSJPS -Fę ćF GBNJMJBS QPTUFSJPS EFOTJUZ GPS UIF HMPCFUPTTJOH EBUB ćSFF FYBNQMF QBSBNFUFS WBMVFT    BSF NBSLFE CZ UIF number of water samples 'ĶĴłĿIJ Ƌƌ 4JNVMBUJOH QSFEJDUJPOT GSPN UIF UPUBM QPTUFSJPS -Fę ćF GBNJMJBS QPTUFSJPS EFOTJUZ GPS UIF HMPCFUPTTJOH EBUB ćSFF FYBNQMF QBSBNFUFS WBMVFT    BSF NBSLFE CZ UIF WFSUJDBM MJOFT .JEEMF DPMVNO &BDI PG UIF UISFF QBSBNFUFS WBM VFT JT VTFE UP TJNVMBUF PCTFSWBUJPOT 3JHIU $PNCJOJOH TJNV MBUFE PCTFSWBUJPO EJTUSJCVUJPOT GPS BMM QBSBNFUFS WBMVFT OPU KVTU   BOE  FBDI XFJHIUFE CZ JUT QPTUFSJPS QSPCBCJMJUZ QSPEVDFT UIF QPTUFSJPS QSFEJDUJWF EFOTJUZ ćJT EFOTJUZ QSPQB HBUFT VODFSUBJOUZ BCPVU QBSBNFUFS UP VODFSUBJOUZ BCPVU QSFEJD UJPO 0CTFSWFE WBMVF  IJHIMJHIUFE 3 DPEF  )2 ʄǤ -$)*(ǭ Ƽ ƿ ǐ .$5 ʃDŽ ǐ +-*ʃ.(+' . Ǯ ćF TZNCPM .(+' . BCPWF JT UIF TBNF MJTU PG SBOEPN TBNQMFT GSPN UIF QPTUFSJPS EFOTJUZ UIBU ZPVWF VTFE JO QSFWJPVT TFDUJPOT 'PS FBDI TBNQMFE WBMVF B SBOEPN CJOPNJBM PCTFSWBUJPO JT HFOFSBUFE 4JODF UIF TBNQMFE WBMVFT BQQFBS JO QSPQPS UJPO UP UIFJS QPTUFSJPS QSPCBCJMJUJFT UIF SFTVMUJOH TJNVMBUFE PCTFSWBUJPOT BSF BW FSBHFE PWFS UIF QPTUFSJPS :PV DBO NBOJQVMBUF UIFTF TJNVMBUFE PCTFSWBUJPOT KVTU MJLF ZPV NBOJQVMBUF TBNQMFT GSPN UIF QPTUFSJPS‰ZPV DBO DPNQVUF JOUFSWBMT BOE
5. Predictive checks • Something like a significance test, but not • No universally best way to evaluate adequacy of model-based predictions • No way to justify always using a threshold like 5% • Good predictive checks always depend upon purpose and imagination “It would be very nice to have a formal apparatus that gives us some ‘optimal’ way of recognizing unusual phenomena and inventing new classes of hypotheses [...]; but this remains an art for the creative human mind.” —E.T. Jaynes (1922–1998)
6. So far... • The program • Make a model • Approximate the posterior • Use posterior to describe uncertainty • Use posterior-based predictions to check model
7. Philosophy • Inference in language of probability • The “best” parameter value is not the focus • The whole posterior is the focus • Even the “best” may be terrible From Breath of Bones: A Tale of the Golem
8. Triumph of Geocentrism • Claudius Ptolemy (90–168) • Egyptian mathematician • Accurate model of planetary motion • Epicycles: orbits on orbits • Fourier series http://facultyweb.berry.edu/ttimberlake/copernican/
9. Geocentrism • Descriptively accurate • Mechanistically wrong • General method of approximation • Known to be wrong Regression • Descriptively accurate • Mechanistically wrong • General method of approximation • Taken too seriously
10. Linear regression • Simple statistical golems • Model of mean and variance of normally (Gaussian) distributed measure • Mean as additive combination of predictors • Constant variance
11. 1809 Bayesian argument for normal error and least-squares estimation  0$7+(0$7,&60$*$=,1( ]D[LV 3DODQHWHVGVUEQ K OQWUUL SHULKHOLRQ LVWKHSRLQFORVHVWLWWDQH HL L RQKLVHOOSWLDORUEW L GHHULRQH E\aWHWP ISHULKHOLRQ SVDH ROFLH\ FRPSXWHGYDOXHV ILQ HUO\VWXLHVRI ODQHWUPRWLQ 6SRHQZWD HNR WZ K R QHQ ),*85( 3DUDPHWHUV GHVFWLELQJ WKH SODQHWDU\ RUELW GLIIHUHQW SODQHV  ZKLFK GHWHUPLQHV WKH RULHQWDWLRQ RI WKH HOOLSVH ZLWKLQ WKH RUELWD SODQH 1RWH WKDW DSKHOLRQ LV WKH SRLQW RQ WKH RUELW IXUWKHVW IURP WKH VXQ ZKHUHD SHULKHOLRQ LV WKH SRLQW FORVHVW WR WKH VXQ 7KH HOOLSVH LWVHOI LVGHWHUPLQHG E\ DWKH OHQJWK RI LWV VHPLPDMRU D[LV DQGHLWV HFFHQWULFLW\ )LQDOO\ WKH SRVLWLRQ RI WKH SODQHW RQ WKLV HOOLSWLFDO RUELW LV GHWHUPLQHG E\ WKH WLPH RI SHULKHOLRQ SDVVDJH &ROOHFWLYHO\ WKH VL[ TXDQWLWLHV L 4O 7 DH DQG US DUH UHIHUUHG WR DV WKH HOHPHQWV RI WKH RUELW )RU IXWXUH UHIHUHQFH ZH QRWH WKDW WKH DQJOH LQ )LJ ,LV NQRZQ DV WKH WUXH DQRU!DO\ DQG GHILQH Y 4O FR DJDLQDVXPRIDQJOHV LQWZRGLIIHUHQW SODQHV 7KHZRUG DQRLQDO\ ZDV DSSDUHQWO\ FKRVHQ EHFDXVH RI WKH GLVFUHSDQF\ EHWZHHQ WKH DFWXDO DQG
12. Why normal? • Why are normal (Gaussian) distributions so common in statistics? 1. Easy to calculate with 2. Common in nature 3. Most logical assumption 0.0 0.1 0.2 0.3 0.4 x density −4σ −2σ 0 2σ 4σ 95%
18. 0 4 8 12 16 -6 -3 0 3 6 step number position -6 -3 0 3 6 0.0 0.2 0.4 position Density 4 steps -6 -3 0 3 6 0.00 0.15 0.30 position Density 8 steps -6 -3 0 3 6 0.00 0.10 0.20 position Density 16 steps 0 4 8 12 16 -6 -3 0 3 6 step number position -6 -3 0 3 6 0.0 0.2 0.4 position Density 4 steps -6 -3 0 3 6 0.00 0.15 0.30 position Density 8 steps -6 -3 0 3 6 0.00 0.10 0.20 position Density 16 steps Figure 4.2
19. Why normal? • Processes that produce normal distributions • Addition • Products of small deviations • Logarithms of products Francis Galton’s 1894 “bean machine” for simulating normal distributions
20. Normal distribution 1S(Y|µ, σ) =  √ πσ FYQ − (µ − Y) σ 1S(Y|µ, σ) =  √ πσ FYQ − (µ − Y) σ MJLFMJIPPE = TUBOEBSEJ[FS × FYQ ⎛ ⎝ MPDBUJPOTIBQF TDBMF ⎞ ⎠
21. 0.0 0.1 0.2 0.3 0.4 x density −4σ −2σ 0 2σ 4σ Normal density 1S(Y|µ, σ) =  √ πσ FYQ − (µ − Y) σ 95%
22. Why normal? • Ontological perspective • Processes which add fluctuations result in dampening • Damped fluctuations end up Gaussian • No information left, except mean and variance • Can’t infer process from distribution! • Epistemological perspective • Know only mean and variance • Then least surprising and most logical (maximum entropy) distribution is Gaussian • Nature likes maximum entropy distributions
23. Linear models • Models of normally distributed data common • “General Linear Model”: t-test, single regression, multiple regression, ANOVA, ANCOVA, MANOVA, MANCOVA, yadda yadda yadda • All the same thing • Learn strategy, not procedure
24. Language for modeling • Questions to answer 1. What are the outcomes? 2. How are the outcomes generated (what is likelihood)? 3. What are the predictors, if any? 4. How do predictors relate to likelihood? 5. What are the priors? From Breath of Bones: A Tale of the Golem
25. Language for modeling IF NPEFM EFĕOJUJPOT 8F XJMM BMTP CF BCMF UP TFF OBUVSBM XBZT UP DIBOHF UIFTF BT VNQUJPOT JOTUFBE PG GFFMJOH USBQQFE XJUIJO TPNF QSPDSVTUFBO NPEFM UZQF MJL FHSFTTJPO PS NVMUJQMF SFHSFTTJPO PS "/07" PS "/$07" PS TVDI ćFTF BSF B IF TBNF LJOE PG NPEFM BOE UIBU GBDU CFDPNFT PCWJPVT PODF XF LOPX IPX UP UBM CPVU NPEFMT BT NBQQJOHT PG POF TFU PG WBSJBCMFT UISPVHI B QSPCBCJMJUZ EJTUSJCV PO POUP BOPUIFS TFU PG WBSJBCMFT  3FEFTDSJCJOH UIF HMPCF UPTTJOH NPEFM *UT HPPE UP XPSL XJUI FYBNQMFT FDBMM UIF QSPQPSUJPO PG XBUFS QSPCMFN GSPN QSFWJPVT DIBQUFST ćF NPEFM JO UIB BTF XBT BMXBZT O8 ∼ #JOPNJBM(O, Q) Q ∼ 6OJGPSN(, ) XIFSF O8 XBT UIF PCTFSWFE DPVOU PG XBUFS QPJOUT O XBT UIF UPUBM OVNCFS P PJOUT BOE Q XBT UIF QSPQPSUJPO PG XBUFS 3FBE UIF BCPWF TUBUFNFOU BT ćF DPVOU O8 JT EJTUSJCVUFE CJOPNJBMMZ XJUI TBNQMF TJ[F O BOE QSPCBCJMJUZ Q ćF QSJPS GPS Q JT BTTVNFE UP CF VOJGPSN CFUXFFO [FSP BOE POF 0ODF XF LOPX UIF NPEFM JO UIJT XBZ XF BVUPNBUJDBMMZ LOPX BMM PG JUT BTTVNQ • Revisit globe tossing model: outcome parameter to estimate likelihood prior distribution parameters “is distributed”
26. Language for modeling FHSFTTJPO PS NVMUJQMF SFHSFTTJPO PS "/07" PS "/$07" PS TVDI ćFTF BSF B IF TBNF LJOE PG NPEFM BOE UIBU GBDU CFDPNFT PCWJPVT PODF XF LOPX IPX UP UBM CPVU NPEFMT BT NBQQJOHT PG POF TFU PG WBSJBCMFT UISPVHI B QSPCBCJMJUZ EJTUSJCV PO POUP BOPUIFS TFU PG WBSJBCMFT  3FEFTDSJCJOH UIF HMPCF UPTTJOH NPEFM *UT HPPE UP XPSL XJUI FYBNQMFT FDBMM UIF QSPQPSUJPO PG XBUFS QSPCMFN GSPN QSFWJPVT DIBQUFST ćF NPEFM JO UIB BTF XBT BMXBZT O8 ∼ #JOPNJBM(O, Q) Q ∼ 6OJGPSN(, ) XIFSF O8 XBT UIF PCTFSWFE DPVOU PG XBUFS QPJOUT O XBT UIF UPUBM OVNCFS P PJOUT BOE Q XBT UIF QSPQPSUJPO PG XBUFS 3FBE UIF BCPWF TUBUFNFOU BT ćF DPVOU O8 JT EJTUSJCVUFE CJOPNJBMMZ XJUI TBNQMF TJ[F O BOE QSPCBCJMJUZ Q ćF QSJPS GPS Q JT BTTVNFE UP CF VOJGPSN CFUXFFO [FSP BOE POF 0ODF XF LOPX UIF NPEFM JO UIJT XBZ XF BVUPNBUJDBMMZ LOPX BMM PG JUT BTTVNQ POT 8F LOPX UIF CJOPNJBM EJTUSJCVUJPO BTTVNFT UIBU FBDI TBNQMF HMPCF UPTT JOEFQFOEFOU PG UIF PUIFST BOE TP XF BMTP LOPX UIBU UIF NPEFM BTTVNFT UIB • Revisit globe tossing model: The count nW is distributed binomially with sample size n and probability p. The prior for p is assumed to be uniform between zero and one.
27. Some data: Kalahari foragers DFOTVT EBUB GPS UIF %PCF BSFB ,VOH 4BO DPNQJMFE GSPN JOUFSWJFXT DPO EVDUFE CZ /BODZ )PXFMM JO UIF MBUF ÔT 'PS UIF OPOBOUISPQPMPHJ SFBEJOH BMPOH UIF ,VOH 4BO BSF UIF NPTU GBNPVT GPSBHJOH QPQVMBUJP PG UIF UI DFOUVSZ MBSHFMZ CFDBVTF PG EFUBJMFE RVBOUJUBUJWF TUVEJFT C QFPQMF MJLF )PXFMM -PBE UIF EBUB BOE QMBDF UIFN JOUP B DPOWFOJFOU PCKFDU XJUI 3 DPEF  OLEUDU\ UHWKLQNLQJ GDWD +RZHOO G  +RZHOO 8IBU ZPV IBWF OPX JT B EBUB GSBNF OBNFE TJNQMZ G * VTF UIF OBN G PWFS BOE PWFS BHBJO JO UIJT CPPL UP SFGFS UP UIF EBUB GSBNF XF BS XPSLJOH XJUI BU UIF NPNFOU * LFFQ JUT OBNF TIPSU UP TBWF ZPV UZQJO " EBUB GSBNF JT B TQFDJBM LJOE PG PCKFDU JO 3 *U JT B UBCMF XJUI OBNF DPMVNOT DPSSFTQPOEJOH UP WBSJBCMFT BOE OVNCFSFE SPXT DPSSFTQPOEJO UP JOEJWJEVBM DBTFT *O UIJT FYBNQMF UIF DBTFT BSF JOEJWJEVBMT *OTQFD height weight age male 1 151.765 47.82561 63 1 2 139.700 36.48581 63 0 3 136.525 31.86484 65 0 4 156.845 53.04191 41 1 5 145.415 41.27687 51 0 6 163.830 62.99259 35 1 ... 544 158.750 52.53162 68 1
28. Gaussian model • A first model: 140 150 160 170 180 0.00 0.02 0.04 0.06 height (cm) Density UIF QMPUUFE PVUDPNF WBSJBCMF MPPLT (BVTTJBO UP ZPV (BXLJOH BU USZ UP EFDJEF IPX UP NPEFM UIFN JT VTVBMMZ OPU B HPPE JEFB ćF NJYUVSF PG EJČFSFOU OPSNBM EJTUSJCVUJPOT GPS FYBNQMF BOE JO UIBU CF BCMF UP EFUFDU UIF VOEFSMZJOH OPSNBMJUZ KVTU CZ FZFCBMMJOH UIF IUT BSF BQQSPYJNBUFMZ OPSNBMMZ EJTUSJCVUFE CVU XIJDI OPSNBM EJT F BSF BO JOĕOJUF OVNCFS PG UIFN XJUI BO JOĕOJUF OVNCFS PG EJG OE WBSJBODFT 8FSF SFBEZ UP XSJUF EPXO UIF HFOFSBM NPEFM BOE QBSBNFUFST UIBU NBYJNJ[F UIF MJLFMJIPPE 5P EFĕOF UIF IFJHIUT SJCVUFE XJUI B NFBO µ BOE TUBOEBSE EFWJBUJPO σ XF XSJUF IJ ∼ /PSNBM(µ, σ). ZPVMM TFF UIF TBNF NPEFM XSJUUFO BT IJ ∼ N(µ, σ) XIJDI NFBOT ćF TZNCPM I SFGFST UP UIF MJTU PG IFJHIUT BOE UIF TVCTDSJQU J JWJEVBM FMFNFOU PG UIJT MJTU *U JT DPOWFOUJPOBM UP VTF J CFDBVTF JU Y ćF JOEFY J UBLFT PO SPX OVNCFST BOE TP JO UIJT FYBNQMF DBO SPN  UP  UIF OVNCFS PG IFJHIUT JO ƽɠ# $"#/  "T TVDI UIF TBZJOH UIBU BMM UIF HPMFN LOPXT BCPVU FBDI IFJHIU NFBTVSFNFOU
29. Gaussian model DPVME CF B NJYUVSF PG EJČFSFOU OPSNBM EJTUSJCVUJPOT GPS FYBNQMF BOE J ZPV XPOU CF BCMF UP EFUFDU UIF VOEFSMZJOH OPSNBMJUZ KVTU CZ FZFCBMMJO TJUZ QMPU 4P UIF IFJHIUT BSF BQQSPYJNBUFMZ OPSNBMMZ EJTUSJCVUFE CVU XIJDI OPSNB VUJPO ćFSF BSF BO JOĕOJUF OVNCFS PG UIFN XJUI BO JOĕOJUF OVNCFS P OU NFBOT BOE WBSJBODFT 8FSF SFBEZ UP XSJUF EPXO UIF HFOFSBM NPEF UIF VOJRVF QBSBNFUFST UIBU NBYJNJ[F UIF MJLFMJIPPE 5P EFĕOF UIF I PSNBMMZ EJTUSJCVUFE XJUI B NFBO µ BOE TUBOEBSE EFWJBUJPO σ XF XSJUF IJ ∼ /PSNBM(µ, σ). NBOZ CPPLT ZPVMM TFF UIF TBNF NPEFM XSJUUFO BT IJ ∼ N(µ, σ) XIJDI N TBNF UIJOH ćF TZNCPM I SFGFST UP UIF MJTU PG IFJHIUT BOE UIF TVCTD OT FBDI JOEJWJEVBM FMFNFOU PG UIJT MJTU *U JT DPOWFOUJPOBM UP VTF J CFDB ET GPS JOEFY ćF JOEFY J UBLFT PO SPX OVNCFST BOE TP JO UIJT FYBNQ BOZ WBMVF GSPN  UP  UIF OVNCFS PG IFJHIUT JO ƽɠ# $"#/  "T TVD EFM BCPWF JT TBZJOH UIBU BMM UIF HPMFN LOPXT BCPVU FBDI IFJHIU NFBTVSF FĕOFE CZ UIF TBNF OPSNBM EJTUSJCVUJPO XJUI B DPOTUBOU NFBO µ BOE TUBO BUJPO σ #FGPSF MPOH UIPTF MJUUMF JT BSF HPJOH UP TIPX VQ PO UIF SJHIUIBO outcome “is distributed” likelihood Height hi of an individual i is distributed normally, with mean mu and standard deviation sigma. mean standard deviation
30. Gaussian model • Add priors: FNBJOT BMNPTU QFSGFDUMZ OPSNBM *O TVDI DBTFT JJE SFNBJOT QFSGFDUMZ PSJOH UIF DPSSFMBUJPOT UIF NPEFM XFSF HPJOH UP OFFE TPNF QSJPST ćF QBSBNFUFST UP CPUI µ BOE σ TP XF OFFE B QSJPS 1S(µ, σ) UIF KPJOU QSJPS QSPC SBNFUFST *O NPTU DBTFT QSJPST BSF TQFDJĕFE JOEFQFOEFOUMZ GPS XIJDI BNPVOUT UP BTTVNJOH 1S(µ, σ) = 1S(µ) 1S(σ) ćFO XF IJ ∼ /PSNBM(µ, σ) >OLNHOLKRRG@ µ ∼ /PSNBM(, ) >µ SULRU@ σ ∼ 6OJGPSN(, ) >σ SULRU@ SJHIU BSF OPU QBSU PG UIF NPEFM CVU JOTUFBE KVTU OPUFT UP IFMQ ZPV QVSQPTF PG FBDI MJOF ćF QSJPS GPS µ JT B CSPBE (BVTTJBO QSJPS DN XJUI  PG QSPCBCJMJUZ CFUXFFO  ±  PE JEFB UP QMPU ZPVS QSJPST TP ZPV IBWF B TFOTF PG UIF BTTVNQUJPO IF NPEFM *O UIJT DBTF 100 140 180 0.00 0.02 0.04 mu dnorm(x, 156, 10) 0 20 40 60 80 0.000 0.010 0.020 sigma dunif(x, 0, 50)
31. Gaussian model • What do these priors imply about height, before we see data? Simulate! 50 150 250 0.000 0.010 height (cm) Density XPVME JNQMZ UIBU  PG JOEJWJEVBM IFJHIUT MJF XJUIJO DN PG UIF BWFSBHF IFJHIU ćBUT B WFSZ MBSHF SBOHF "MM UIJT UBML JT OJDF CVU JUMM IFMQ UP SFBMMZ TFF XIBU UIFTF QSJPST JNQMZ BCPVU UIF EJTUSJCVUJPO PG JOEJWJEVBM IFJHIUT :PV EJEOU TQFDJGZ B QSJPS QSPCBCJMJUZ EFOTJUZ PG IFJHIUT EJSFDUMZ CVU PODF ZPVWF DIPTFO QSJPST GPS µ BOE σ UIFTF JNQMZ B QSJPS EFOTJUZ PG JOEJWJEVBM IFJHIUT :PV DBO RVJDLMZ TJNVMBUF IFJHIUT CZ TBNQMJOH GSPN UIF QSJPS MJLF ZPV TBNQMFE GSPN UIF QPTUFSJPS CBDL JO $IBQUFS  3FNFNCFS FWFSZ QPTUFSJPS JT B QSJPS GSPN TPNF PUIFS NPEFM TP ZPV DBO QSPDFTT QSJPST KVTU MJLF QPTUFSJPST 3 DPEF  .(+' Ǐ(0 ʄǤ -)*-(ǭ Ƽ ƿ ǐ Ƽǀǁ ǐ Ƽƻ Ǯ .(+' Ǐ.$"( ʄǤ -0)$!ǭ Ƽ ƿ ǐ ƻ ǐ ǀƻ Ǯ +-$*-Ǐ# ʄǤ -)*-(ǭ Ƽ ƿ ǐ .(+' Ǐ(0 ǐ .(+' Ǐ.$"( Ǯ  ).ǭ +-$*-Ǐ# Ǯ ćF EFOTJUZ QMPU ZPV HFU TIPXT B WBHVFMZ CFMMTIBQFE EFOTJUZ XJUI UIJDL UBJMT *U JT UIF FYQFDUFE EJTUSJCVUJPO PG IFJHIUT BWFSBHFE PWFS UIF QSJPS 1MBZ BSPVOE XJUI UIF OVNCFST JO UIF QSJPST BCPWF UP FYQMPSF UIFJS FČFDUT PO UIF QSJPS QSPCBCJMJUZ EFOTJUZ PG IFJHIUT  (SJE BQQSPYJNBUJPO PG UIF QPTUFSJPS EFOTJUZ 4JODF UIJT JT UIF ĕSTU (BVTT JBO NPEFM GBNJMZ JO UIF CPPL BOE JOEFFE UIF ĕSTU NPEFM TQBDF XJUI NPSF UIBO POF QBSBNFUFS JUT XPSUI RVJDLMZ NBQQJOH PVU UIF QPTUFSJPS EFOTJUZ UISPVHI CSVUF GPSDF DBMDVMBUJPOT ćJT JTOU UIF BQQSPBDI * FODPVSBHF JO BOZ PUIFS QMBDF CFDBVTF JU JT MBCPSJPVT BOE DPNQVUBUJPOBMMZ FYQFOTJWF *OEFFE JU JT VTVBMMZ TP JNQSBDUJDBM BT UP CF FTTFOUJBMMZ JNQPTTJCMF #VU BT BMXBZT JU JT XPSUI LOPXJOH XIBU UIF UBSHFU BDUVBMMZ MPPLT MJLF CFGPSF ZPV TUBSU BDDFQUJOH BQQSPYJNBUJPOT PG JU " MJUUMF MBUFS JO 100 cm = 3.3 feet 200 cm = 6.5 feet
32. Estimating mu and sigma • Aim for the posterior distribution, which is now 2-dimensional • Grid approximation: Compute posterior for many combinations of mu and sigma   -*/&"3 . 153.0 154.0 155.0 156.0 7.0 7.5 8.0 8.5 9.0 mu sigma ' S E U C 3 DPEF   ).ǭ .(+' Ǐ(0 Ǯ  ).ǭ .(+' Ǐ.$"( Ǯ
33. 153.0 154.0 155.0 156.0 0.0 0.2 0.4 0.6 0.8 1.0 mu Density 7.0 7.5 8.0 8.5 9.0 0.0 0.4 0.8 1.2 sigma Density mu sigma Figure 4.3
34. Quadratic approximation • How to do the same thing, using quadratic approximation (MAP and standard deviation)  ʄǤ *2 ''Ƽ ƽ ʄǤ ǯ ɠ" ʅʃ Ƽǃ ǐ ǰ /PX XFSF SFBEZ UP EFĕOF UIF NPEFM VTJOH 3T GPSNVMB TZOUBY ćF NPEFM EFĕOJUJPO JO UIJT DBTF JT KVTU BT CFGPSF CVU OPX XFMM SFQFBU JU XJUI FBDI DPSSFTQPOEJOH MJOF PG 3 DPEF TIPXO PO UIF SJHIUIBOE NBSHJO IJ ∼ /PSNBM(µ, σ) # $"#/ ʋ )*-(ǭ(0ǐ.$"(Ǯ µ ∼ /PSNBM(, ) (0 ʋ )*-(ǭƼǀǁǐƼƻǮ σ ∼ 6OJGPSN(, ) .$"( ʋ 0)$!ǭƻǐǀƻǮ /PX QMBDF UIF 3 DPEF FRVJWBMFOUT JOUP BO '$./ )FSFT BO '$./ PG UIF GPSNVMBT BCPWF 3 DPEF  !'$./ ʄǤ '$./ǭ # $"#/ ʋ )*-(ǭ (0 ǐ .$"( Ǯ ǐ (0 ʋ )*-(ǭ Ƽǀǁ ǐ Ƽƻ Ǯ ǐ .$"( ʋ 0)$!ǭ ƻ ǐ ǀƻ Ǯ   -*/&"3 .0%&-4 Ǯ /PUF UIF DPNNBT BU UIF FOE PG FBDI MJOF FYDFQU UIF MBTU ćFTF DPNNBT TFQBSBUF FBDI MJOF PG UIF NPEFM EFĕOJUJPO 'JU UIF NPEFM UP UIF EBUB JO UIF EBUB GSBNF ƽ XJUI 3 DPEF  (ƿǏƼ ʄǤ (+ǭ !'$./ ǐ /ʃƽ Ǯ "ęFS FYFDVUJOH UIJT DPEF ZPVMM IBWF B ĕU NPEFM TUPSFE JO UIF TZNCPM (ƿǏƼ /PX UBLF B MPPL BU UIF ĕU NBYJNVN B QPTUFSJPSJ NPEFM 3 DPEF  +- $.ǭ (ƿǏƼ Ǯ  ) / 1 ƽǏǀɳ DŽǂǏǀɳ   -*/&"3 .0%&-4 Ǯ /PUF UIF DPNNBT BU UIF FOE PG FBDI MJOF FYDFQU UIF MBTU ćFTF DPNNBT TFQBSBUF FBDI MJOF PG UIF NPEFM EFĕOJUJPO 'JU UIF NPEFM UP UIF EBUB JO UIF EBUB GSBNF ƽ XJUI 3 DPEF  (ƿǏƼ ʄǤ (+ǭ !'$./ ǐ /ʃƽ Ǯ "ęFS FYFDVUJOH UIJT DPEF ZPVMM IBWF B ĕU NPEFM TUPSFE JO UIF TZNCPM (ƿǏƼ /PX UBLF B MPPL BU UIF ĕU NBYJNVN B QPTUFSJPSJ NPEFM 3 DPEF  +- $.ǭ (ƿǏƼ Ǯ  ) / 1 ƽǏǀɳ DŽǂǏǀɳ (0 ƼǀƿǏǁƼ ƻǏƿƼ ƼǀƾǏǃƼ ƼǀǀǏƿƽ .$"( ǂǏǂƾ ƻǏƽDŽ ǂǏƼǁ ǃǏƾƻ ćFTF OVNCFST QSPWJEF (BVTTJBO BQQSPYJNBUJPOT GPS FBDI QBSBNFUFST NBSHJOBM EJTUSJCVUJPO
35. Using map PO UIF SJHIUIBOE NBSHJO IJ ∼ /PSNBM(µ, σ) # $"#/ ʋ )*-(ǭ(0ǐ µ ∼ /PSNBM(, ) (0 ʋ )*-(ǭƼǀǁ σ ∼ 6OJGPSN(, ) .$"( ʋ 0)$!ǭƻǐǀ /PX QMBDF UIF 3 DPEF FRVJWBMFOUT JOUP BO '$./ )FSFT BO '$./ PG UIF GPSNVMBT BC !'$./ ʄǤ '$./ǭ # $"#/ ʋ )*-(ǭ (0 ǐ .$"( Ǯ ǐ (0 ʋ )*-(ǭ Ƽǀǁ ǐ Ƽƻ Ǯ ǐ .$"( ʋ 0)$!ǭ ƻ ǐ ǀƻ Ǯ VTFGVM EFTQJUF JHOPSJOH UIF DPSSFMBUJPOT 5P DPNQMFUF UIF NPEFM XFSF HPJOH UP OFFE TPNF QSJPST ć CF FTUJNBUFE BSF CPUI µ BOE σ TP XF OFFE B QSJPS 1S(µ, σ) UIF BCJMJUZ GPS BMM QBSBNFUFST *O NPTU DBTFT QSJPST BSF TQFDJĕFE J FBDI QBSBNFUFS XIJDI BNPVOUT UP BTTVNJOH 1S(µ, σ) = 1S(µ DBO XSJUF IJ ∼ /PSNBM(µ, σ) µ ∼ /PSNBM(, ) σ ∼ 6OJGPSN(, ) ćF MBCFMT PO UIF SJHIU BSF OPU QBSU PG UIF NPEFM CVU JOTUFBE KVTU LFFQ USBDL PG UIF QVSQPTF PG FBDI MJOF ćF QSJPS GPS µ JT B CSPB DFOUFSFE PO DN XJUI  PG QSPCBCJMJUZ CFUXFFO  ±  *UT B WFSZ HPPE JEFB UP QMPU ZPVS QSJPST TP ZPV IBWF B TFOTF UIFZ CVJME JOUP UIF NPEFM *O UIJT DBTF 0-1 ǭ )*-(ǭ 3 ǐ Ƽǀǁ ǐ Ƽƻ Ǯ ǐ !-*(ʃƼƻƻ ǐ /*ʃƽƻƻ Ǯ &YFDVUF UIBU DPEF ZPVSTFMG UP TFF UIBU UIF HPMFN JT BTTVNJOH
36. 153.0 154.0 155.0 156.0 0.0 0.2 0.4 0.6 0.8 1.0 mu Density 7.0 7.5 8.0 8.5 9.0 0.0 0.4 0.8 1.2 sigma Density Samples Approximation "ęFS FYFDVUJOH UIJT DPEF ZPVMM IBWF B ĕU NPEFM TUPSFE JO UIF TZNCPM (ƿǏƼ /PX UBLF BU UIF ĕU NBYJNVN B QPTUFSJPSJ NPEFM 3 DPEF  +- $.ǭ (ƿǏƼ Ǯ  ) / 1 ƽǏǀɳ DŽǂǏǀɳ (0 ƼǀƿǏǁƼ ƻǏƿƼ ƼǀƾǏǃƼ ƼǀǀǏƿƽ .$"( ǂǏǂƾ ƻǏƽDŽ ǂǏƼǁ ǃǏƾƻ ćFTF OVNCFST QSPWJEF (BVTTJBO BQQSPYJNBUJPOT GPS FBDI QBSBNFUFST NBSHJOBM EJTUSJC ćJT NFBOT UIF QMBVTJCJMJUZ PG FBDI WBMVF PG µ BęFS BWFSBHJOH PWFS UIF QMBVTJCJMJUJFT P WBMVF PG σ JT HJWFO CZ B (BVTTJBO EJTUSJCVUJPO XJUI NFBO  BOE TUBOEBSE EFWJBUJPO ćF  BOE  RVBOUJMFT BSF QFSDFOUJMF JOUFSWBM CPVOEBSJFT * FODPVSBHF Z DPNQBSF UIFTF UP UIF )1%*T GSPN UIF HSJE BQQSPYJNBUJPO FBSMJFS UP UIF RVBESBUJD JOU EJTQMBZFE BCPWF :PVMM ĕOE UIBU UIFZ BSF BMNPTU JEFOUJDBM 8IFO UIF QPTUFSJPS JT BQ NBUFMZ (BVTTJBO UIFO UIJT JT XIBU ZPV TIPVME FYQFDU 0WFSUIJOLJOH 4UBSU WBMVFT GPS (+ (+ FTUJNBUFT UIF QPTUFSJPS CZ DMJNCJOH JU MJLF B IJMM UIJT JU IBT UP TUBSU DMJNCJOH TPNFQMBDF BU TPNF DPNCJOBUJPO PG QBSBNFUFS WBMVFT 6OMFTT Z JU PUIFSXJTF (+ TUBSUT BU SBOEPN WBMVFT TBNQMFE GSPN UIF QSJPS #VU JUT BMTP QPTTJCMF UP TQ TUBSUJOH WBMVF GPS BOZ QBSBNFUFS JO UIF NPEFM *O UIF FYBNQMF JO UIF QSFWJPVT TFDUJPO UIBU NF QBSBNFUFST µ BOE σ )FSFT B HPPE MJTU PG TUBSUJOH WBMVFT JO UIJT DBTF 3 DPEF  ./-/ ʄǤ '$./ǭ (0ʃ( )ǭƽɠ# $"#/Ǯǐ .$"(ʃ.ǭƽɠ# $"#/Ǯ Ǯ
37. Sample size matters • A subset of only 20 heights (via grid approximation):
38. Scaffolds • map is a scaffold • Forces full specification of model, so you learn it • Works with a very wide class of models • Hardly ever the easiest way to do anything
39. Adding a predictor variable • How does weight describe height? 30 35 40 45 50 55 60 140 150 160 170 180 d2$weight d2$height
40. Adding a predictor variable • Use a linear model of the mean, mu:  -*/&"3 .0%&-4 JO Y ćJT JT XIBU JU MPPLT MJLF XJUI FYQMBOBUJPO UP GPMMPX IJ ∼ /PSNBM(µJ, σ) >OLNHOLKRRG@ µJ = α + βYJ >OLQHDUPRGHO@ α ∼ /PSNBM(, ) >α SULRU@ β ∼ /PSNBM(, ) >β SULRU@ σ ∼ 6OJGPSN(, ) >σ SULRU@ CFMFE FBDI MJOF PO UIF SJHIUIBOE TJEF CZ UIF UZQF PG EFĕOJUJPO JU MM EJTDVTT UIFN JO UVSOT -JLFMJIPPE 5P EFDPEF BMM PG UIJT MFUT CFHJO XJUI KVTU UIF MJLFMJIPPE PG UIF NPEFM ćJT JT OFBSMZ JEFOUJDBM UP CFGPSF FYDFQU OPX UIFSF JT
41. Adding a predictor variable   -*/&"3 .0%&-4 PG UIF WBMVFT JO Y ćJT JT XIBU JU MPPLT MJLF XJUI FYQMBOBUJPO UP GPMMPX IJ ∼ /PSNBM(µJ, σ) >OLNHOLKRRG@ µJ = α + βYJ >OLQHDUPRGHO@ α ∼ /PSNBM(, ) >α SULRU@ β ∼ /PSNBM(, ) >β SULRU@ σ ∼ 6OJGPSN(, ) >σ SULRU@ "HBJO *WF MBCFMFE FBDI MJOF PO UIF SJHIUIBOE TJEF CZ UIF UZQF PG EFĕOJUJPO JU FODPEFT 8FMM EJTDVTT UIFN JO UVSOT  -JLFMJIPPE 5P EFDPEF BMM PG UIJT MFUT CFHJO XJUI KVTU UIF MJLFMJIPPE UIF ĕSTU MJOF PG UIF NPEFM ćJT JT OFBSMZ JEFOUJDBM UP CFGPSF FYDFQU OPX UIFSF JT B MJUUMF JOEFY J PO UIF µ BT XFMM BT UIF I ćJT JT OFDFTTBSZ OPX CFDBVTF UIF NFBO mean when xi = 0 “intercept” change in mean, per unit change xi “slope” weight on row i mean on row i
42. TQPOEJOH 3 DPEF PO UIF SJHIUIBOE TJEF IJ ∼ /PSNBM(µJ, σ) # $"#/ ʋ )*-(ǭ(0ǐ.$"(Ǯ µJ = α + βYJ (0 ʄǤ  ɾ Ƿ2 $"#/ α ∼ /PSNBM(, )  ʋ )*-(ǭƼǀǁǐƼƻƻǮ β ∼ /PSNBM(, )  ʋ )*-(ǭƻǐƼƻǮ σ ∼ 6OJGPSN(, ) .$"( ʋ 0)$!ǭƻǐǀƻǮ F UIBU UIF MJOFBS NPEFM JO UIF 3 DPEF PO UIF SJHIUIBOE TJEF VTFT UIF 3 BTTJHONFOU UPS ʄǤ FWFO UIPVHI UIF NBUIFNBUJDBM EFĕOJUJPO VTFT UIF TZNCPM  ćJT JT B DPEF FOUJPO TIBSFE CZ TFWFSBM #BZFTJBO NPEFM ĕUUJOH FOHJOFT TP JUT XPSUI HFUUJOH VTFE UP XJUDI :PV KVTU IBWF UP SFNFNCFS UP VTF ʄǤ JOTUFBE PG ʃ XIFO EFĕOJOH B MJOFBS NPEFM JU "OE UIF BCPWF BMMPXT VT UP CVJME UIF ."1 NPEFM ĕU  / "$)ǐ .$) $/ǘ.  '*)" 24 & -4ǭ- /#$)&$)"Ǯ ǭ *2 ''ƼǮ *2 ''Ƽ Ǥ ǯ ɠ" ʅʃ Ƽǃ ǐ ǰ / (* ' ʄǤ (+ǭ '$./ǭ # $"#/ ʋ )*-(ǭ (0 ǐ .$"( Ǯ ǐ /PUJDF UIBU UIF MJOFBS NPEFM JO UIF 3 DPEF PO UIF SJHIUIBOE TJEF VTFT UIF 3 BTTJHONFOU PQFSBUPS ʄǤ FWFO UIPVHI UIF NBUIFNBUJDBM EFĕOJUJPO VTFT UIF TZNCPM  ćJT JT B DPEF DPOWFOUJPO TIBSFE CZ TFWFSBM #BZFTJBO NPEFM ĕUUJOH FOHJOFT TP JUT XPSUI HFUUJOH VTFE UP UIF TXJUDI :PV KVTU IBWF UP SFNFNCFS UP VTF ʄǤ JOTUFBE PG ʃ XIFO EFĕOJOH B MJOFBS NPEFM ćBUT JU "OE UIF BCPWF BMMPXT VT UP CVJME UIF ."1 NPEFM ĕU 3 DPEF  ȃ '* / "$)ǐ .$) $/ǘ.  '*)" 24 & '$--4ǭ- /#$)&$)"Ǯ /ǭ *2 ''ƼǮ  ʄǤ *2 ''Ƽ ƽ ʄǤ ǯ ɠ" ʅʃ Ƽǃ ǐ ǰ ȃ !$/ (* ' (ƿǏƾ ʄǤ (+ǭ '$./ǭ # $"#/ ʋ )*-(ǭ (0 ǐ .$"( Ǯ ǐ (0 ʄǤ  ɾ Ƿ2 $"#/ ǐ  ʋ )*-(ǭ Ƽǀǁ ǐ Ƽƻƻ Ǯ ǐ  ʋ )*-(ǭ ƻ ǐ Ƽƻ Ǯ ǐ .$"( ʋ 0)$!ǭ ƻ ǐ ǀƻ Ǯ Ǯ ǐ /ʃƽ Ǯ ćF QBSBNFUFS (0 JT OP MPOHFS SFBMMZ B QBSBNFUFS IFSF CFDBVTF JU IBT CFFO SFQMBDFE CZ UIF MJOFBS NPEFM ɾǷ2 $"#/ XIFSF  JT α BOE  JT β BOE 2 $"#/ JT PG DPVSTF PVS Y JO UIJT JOTUBODF 4P UIFSF JT B QSJPS GPS UIF QBSBNFUFS  OPX CVU OPU POF GPS (0 TJODF (0 JT EFĕOFE CZ UIF MJOFBS NPEFM JOTUFBE *O UIF ./-/ MJTU (0 JT SFQMBDFE CZ  XIJDI TUBSUT BU UIF PWFSBMM NFBO KVTU MJLF (0 VTFE