L08 Statistical Rethinking Winter 2019

Lecture 08 of the Dec 2018 through March 2019 edition of Statistical Rethinking. Covers overfitting/cross-validation/information criteria.

Transcript

1. Ulysses’ Compass Statistical Rethinking Winter 2019 Lecture 08 / Week 4
2. Information theory • Machine prediction obeys information theory • Information: Reduction in uncertainty caused by learning an outcome. Today Tomorrow ?
3. Today Tomorrow Los Angeles Glasgow ? ? ? New York
4. Information entropy • 1948, Claude Shannon derived information entropy: Shannon (1916–2001) Uncertainty in a probability distribution is average (minus) log-probability of an event. IFO UIF VODFSUBJOUZ BCPVU IPU PS DPME  EJČFSFOU QPTTJCMF FWFOUT ODFSUBJOUZ PWFS UIF GPVS DPNCJOBUJPOT PG UIFTF FWFOUT‰SBJOIPU PME TIJOFIPU TIJOFDPME‰TIPVME CF UIF TVN PG UIF TFQBSBUF VO OUJFT OF GVODUJPO UIBU TBUJTĕFT UIFTF EFTJEFSBUB ćJT GVODUJPO JT VTVBMMZ ĿĺĮŁĶļĻ IJĻŁĿļĽņ BOE IBT B TVSQSJTJOHMZ TJNQMF EFĕOJUJPO *G FSFOU QPTTJCMF FWFOUT BOE FBDI FWFOU J IBT QSPCBCJMJUZ QJ BOE XF SPCBCJMJUJFT Q UIFO UIF VOJRVF NFBTVSF PG VODFSUBJOUZ XF TFFL JT )(Q) = − & MPH(QJ) = − O J= QJ MPH(QJ).  ET DFSUBJOUZ DPOUBJOFE JO B QSPCBCJMJUZ EJTUSJCVUJPO JT UIF BWFS QSPCBCJMJUZ PG BO FWFOU JHIU SFGFS UP B UZQF PG XFBUIFS MJLF SBJO PS TIJOF PS B QBSUJDVMBS PS FWFO B QBSUJDVMBS OVDMFPUJEF JO B %/" TFRVFODF 8IJMF JUT OPU UP UIF EFUBJMT PG UIF EFSJWBUJPO PG ) JU JT XPSUI QPJOUJOH PVU UIBU
5. Entropy to accuracy • Two probability distributions: p, q • p is true, q is model • How accurate is q, for describing p? • Distance from q to p: Divergence  */'03."5*0/ 5)&03: "/% .0%&- 1&3'03."/$&  PS FYBNQMF UIBU UIF USVF EJTUSJCVUJPO PG FWFOUT JT Q = ., Q = . OTUFBE UIBU UIFTF FWFOUT IBQQFO XJUI QSPCBCJMJUJFT R = ., R = DI BEEJUJPOBM VODFSUBJOUZ IBWF XF JOUSPEVDFE BT B DPOTFRVFODF PG , R} UP BQQSPYJNBUF Q = {Q, Q} ćF GPSNBM BOTXFS UP UIJT RVFT VQPO ) BOE IBT B TJNJMBSMZ TJNQMF GPSNVMB %,-(Q, R) = J QJ MPH(QJ) − MPH(RJ) . HVBHF UIF EJWFSHFODF JT UIF BWFSBHF EJČFSFODF JO MPH QSPCBCJMJUZ CF FU Q BOE NPEFM R  ćJT EJWFSHFODF JT KVTU UIF EJČFSFODF CFUXFFO  ćF FOUSPQZ PG UIF UBSHFU EJTUSJCVUJPO Q BOE UIF FOUSPQZ BSJTJOH UP QSFEJDU Q 8IFO Q = R XF LOPX UIF BDUVBM QSPCBCJMJUJFT PG UIF U DBTF Distance from q to p is the average difference in log-probability.
6. Computing divergence   OTUFBE UIBU UIFTF FWFOUT IBQQFO XJUI QSPCBCJMJUJFT R = ., R = DI BEEJUJPOBM VODFSUBJOUZ IBWF XF JOUSPEVDFE BT B DPOTFRVFODF PG , R} UP BQQSPYJNBUF Q = {Q, Q} ćF GPSNBM BOTXFS UP UIJT RVFT VQPO ) BOE IBT B TJNJMBSMZ TJNQMF GPSNVMB %,-(Q, R) = J QJ MPH(QJ) − MPH(RJ) . HVBHF UIF EJWFSHFODF JT UIF BWFSBHF EJČFSFODF JO MPH QSPCBCJMJUZ CF FU Q BOE NPEFM R  ćJT EJWFSHFODF JT KVTU UIF EJČFSFODF CFUXFFO  ćF FOUSPQZ PG UIF UBSHFU EJTUSJCVUJPO Q BOE UIF FOUSPQZ BSJTJOH UP QSFEJDU Q 8IFO Q = R XF LOPX UIF BDUVBM QSPCBCJMJUJFT PG UIF U DBTF ,-(Q, R) = %,-(Q, Q) = J QJ MPH(QJ) − MPH(QJ) = . EJUJPOBM VODFSUBJOUZ JOEVDFE XIFO XF VTF B QSPCBCJMJUZ EJTUSJCVUJPO TFMG ćBUT TPNFIPX B DPNGPSUJOH UIPVHIU #VU NPSF JNQPSUBOUMZ PSF EJČFSFOU GSPN Q UIF EJWFSHFODF %,- BMTP HSPXT FSHFODF DBO EP GPS VT OPX JT IFMQ VT DPOUSBTU EJČFSFOU BQQSPYJNB  */'03."5*0/ 5)&0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 2.5 q[1] Divergence of q from p q = p ' J Q U Q Q J U p <- c(0.3,0.7) DKL <- function(p,q) sum(p*(log(p)-log(q))) q1seq <- seq(from=0.01,to=0.99,by=0.01) DKLseq <- sapply(q1seq, function(q1) DKL(p,c(q1,1-q1)) ) plot( q1seq , DKLseq )
7. Divergence is not symmetric!
8. Divergence is not symmetric!
9. Estimating Divergence • Use log-score: Sum of log probabilities of each observation • In practice, need to average over posterior: OU IBWF UP LOPX UIF Q JOTJEF UIF FYQFDUBUJPO P XF DBO DPNQBSF UIF BWFSBHF MPHQSPCBCJMJUZ GSPN FBDI NPEFM UP HFU BO FTUJNBUF P F EJTUBODF PG FBDI NPEFM GSPN UIF UBSHFU ćJT BMTP NFBOT UIBU UIF BCTPMVUF NBHO TF WBMVFT XJMM OPU CF JOUFSQSFUBCMF‰OFJUIFS & MPH(RJ) OPS & MPH(SJ) CZ JUTFMG TVHHF PS CBE NPEFM 0OMZ UIF EJČFSFODF & MPH(RJ)−& MPH(SJ) JOGPSNT VT BCPVU UIF EJWFSH I NPEFM GSPN UIF UBSHFU Q P QVU BMM UIJT JOUP QSBDUJDF JU JT DPOWFOUJPOBM UP TVN PWFS BMM UIF PCTFSWBUJPOT J ZJF TDPSF GPS B NPEFM R 4(R) = J MPH(RJ) JOE PG TDPSF JT B MPHQSPCBCJMJUZ TDPSF BOE JU JT UIF HPME TUBOEBSE XBZ UP DPNQBS UJWF BDDVSBDZ PG EJČFSFOU NPEFMT *U JT BO FTUJNBUF PG & MPH(RJ) KVTU XJUIPVU UIF G EJWJEJOH CZ UIF OVNCFS PG PCTFSWBUJPOT P DPNQVUF UIJT TDPSF GPS B #BZFTJBO NPEFM XF IBWF UP VTF UIF FOUJSF QPTUFSJPS EJTU 0UIFSXJTF WFOHFGVM BOHFMT XJMM EFTDFOE VQPO ZPV 8IZ XJMM UIFZ CF BOHSZ *G XF F FOUJSF QPTUFSJPS XF BSF UISPXJOH BXBZ JOGPSNBUJPO #FDBVTF UIF QBSBNFUFST IBW POT UIF QSFEJDUJPOT BMTP IBWF B EJTUSJCVUJPO )PX DBO XF VTF UIF FOUJSF EJTUSJCVUJ UJPOT 8F OFFE UP ĕOE UIF MPH PG UIF BWFSBHF QSPCBCJMJUZ GPS FBDI PCTFSWBUJPO J X FSBHF JT UBLFO PWFS UIF QPTUFSJPS EJTUSJCVUJPO %PJOH UIJT DBMDVMBUJPO DPSSFDUMZ SFR ͧͥͥͧ ͟΀ͥͣͧͣͧ͢͢ ͟΀ͤͣͨͥͨ͟͢ ͟΀ͥͣͨͣ͢͢͡ ͟΀ͣͥͣͧͣ͢͠ ͟΀ͣͣͦͥͤ͢͟ TF WBMVFT JT UIF MPHQSPCBCJMJUZ TDPSF GPS B TQFDJĕD PCTFSWBUJPO 3FDBMM PCTFSWBUJPOT JO UIPTF EBUB *G ZPV TVN UIFTF WBMVFT ZPVMM IBWF UIF TDPSF GPS UIF NPEFM BOE EBUB 8IBU EP UIFTF WBMVFT NFBO -BSHFS VTF UIBU JOEJDBUFT MBSHFS BWFSBHF BDDVSBDZ *U BMTP RVJUF DPNNPO UP TFF T IJŃĶĮĻİIJ XIJDI JT MJLF B MQQE TDPSF CVU NVMUJQMJFE CZ − TP UIBU TNB ćF  JT UIFSF GPS IJTUPSJDBM SFBTPOT H $PNQVUJOH UIF MQQE ćF #BZFTJBO WFSTJPO PG UIF MPHQSPCBCJMJUZ TDPSF J ńĶŀIJĽĿIJıĶİŁĶŃIJıIJĻŀĶŁņ 'PS TPNF EBUB Z BOE QPTUFSJPS EJTUSJCVUJPO Θ MQQE(Z, Θ) = J MPH  4 T Q(ZJ|ΘT) F OVNCFS PG TBNQMFT BOE ΘT JT UIF TUI TFU PG TBNQMFE QBSBNFUFS WBMVFT JO UI 8IJMF JO QSJODJQMF UIJT JT FBTZ‰ZPV KVTU OFFE UP DPNQVUF UIF QSPCBCJMJUZ UJPO J GPS FBDI TBNQMF T UBLF UIF BWFSBHF BOE UIFO UIF MPHBSJUIN‰JO QSBDU log-pointwise-predictive-density
10. Everybody overfits (sometimes) • Common to scale log-score by –2, “Deviance” • Smaller values are better • A meta-model of forecasting: • Two samples: training and testing, size N • Fit model to training sample, get Dtrain • Use posterior from training to compute Dtest • Difference Dtest – Dtrain is overfitting
11. NFBTVSFE JO BOE PVU PG TBNQMF VTJOH B TJNQMF QSFEJDUJPO TDFOBSJP 5P WJTVBMJ[F UIF SFTVMUT PG UIF UIPVHIU FYQFSJNFOU XIBU XFMM EP OPX JT DPOE UIPVHIU FYQFSJNFOU UIPVTBOE UJNFT GPS FBDI PG  EJČFSFOU MJOFBS SFHSFTTJPO NPEFM UIBU HFOFSBUFT UIF EBUB JT ZJ ∼ /PSNBM(µJ, ) µJ = (.)Y,J − (.)Y,J ćJT DPSSFTQPOET UP B (BVTTJBO PVUDPNF Z GPS XIJDI UIF JOUFSDFQU JT α =  B GPS FBDI PG UXP QSFEJDUPST BSF β = . BOE β = −. ćF NPEFMT GPS EBUB BSF MJOFBS SFHSFTTJPOT XJUI CFUXFFO  BOE  GSFF QBSBNFUFST ćF ĕSTU NPE QBSBNFUFS UP FTUJNBUF JT KVTU B MJOFBS SFHSFTTJPO XJUI BO VOLOPXO NFBO BOE &BDI QBSBNFUFS BEEFE UP UIF NPEFM BEET B QSFEJDUPS WBSJBCMF BOE JUT CFUBDPFď UIF iUSVFw NPEFM IBT OPO[FSP DPFďDJFOUT GPS POMZ UIF ĕSTU UXP QSFEJDUPST XF UIF USVF NPEFM IBT  QBSBNFUFST #Z ĕUUJOH BMM ĕWF NPEFMT XJUI CFUXFFO  BOE UP USBJOJOH TBNQMFT GSPN UIF TBNF QSPDFTTFT XF DBO HFU BO JNQSFTTJPO GPS I CFIBWFT 'ĶĴłĿIJ ƎƏ TIPXT UIF SFTVMUT PG UIPVTBOE TJNVMBUJPOT GPS FBDI NPEFM UZQ GFSFOU TBNQMF TJ[FT ćF GVODUJPO UIBU DPOEVDUT UIF TJNVMBUJPOT JT .$(Ǐ/-$) Data generating model: Models fit to data: µJ = α µJ = α + β Y,J µJ = α + β Y,J + β Y,J µJ = α + β Y,J + β Y,J + β Y,J µJ = α + β Y,J + β Y,J + β Y,J + β Y,J MQQE = MPH PG QSPEVDU PG BWFSBHF MJLFMJIPPET = TVN PG MPHT PG BWFSBHF MJLFMJIPPET / (flat priors) Everybody overfits
12.   07&3'*55*/( 3&(6-"3*;"5*0/ "/% */'03."5*0/ $3*5&3*" 1 2 3 4 5 45 50 55 60 65 number of parameters deviance N = 20 in out +1SD –1SD 1 2 3 4 5 250 260 270 280 290 300 number of parameters deviance N = 100 in out 'ĶĴłĿIJ ƎƏ %FWJBODF JO BOE PVU PG TBNQMF *O FBDI QMPU NPEFMT XJUI EJG GFSFOU OVNCFST PG QSFEJDUPS WBSJBCMFT BSF TIPXO PO UIF IPSJ[POUBM BYJT %F WJBODF BDSPTT UIPVTBOE TJNVMBUJPOT JT TIPXO PO UIF WFSUJDBM #MVF TIPXT Data generating model Everybody overfits Figure 7.7
13.   07&3'*55*/( 3&(6-"3*;"5*0/ "/% */'03."5*0/ $3*5&3*" 1 2 3 4 5 45 50 55 60 65 number of parameters deviance N = 20 in out +1SD –1SD 1 2 3 4 5 250 260 270 280 290 300 number of parameters deviance N = 100 in out 'ĶĴłĿIJ ƎƏ %FWJBODF JO BOE PVU PG TBNQMF *O FBDI QMPU NPEFMT XJUI EJG GFSFOU OVNCFST PG QSFEJDUPS WBSJBCMFT BSF TIPXO PO UIF IPSJ[POUBM BYJT %F WJBODF BDSPTT UIPVTBOE TJNVMBUJPOT JT TIPXO PO UIF WFSUJDBM #MVF TIPXT Everybody overfits Figure 7.7
14.   07&3'*55*/( 3&(6-"3*;"5*0/ "/% */'03."5*0/ $3*5&3*" 1 2 3 4 5 45 50 55 60 65 number of parameters deviance N = 20 in out +1SD –1SD 1 2 3 4 5 250 260 270 280 290 300 number of parameters deviance N = 100 in out 'ĶĴłĿIJ ƎƏ %FWJBODF JO BOE PVU PG TBNQMF *O FBDI QMPU NPEFMT XJUI EJG GFSFOU OVNCFST PG QSFEJDUPS WBSJBCMFT BSF TIPXO PO UIF IPSJ[POUBM BYJT %F WJBODF BDSPTT UIPVTBOE TJNVMBUJPOT JT TIPXO PO UIF WFSUJDBM #MVF TIPXT Everybody overfits Figure 7.7
15. Regularization • Must be skeptical of the sample! • Use informative, conservative priors to reduce overfitting => model learns less from sample • But if too skeptical, model learns too little • Such priors are regularizing
16. Regularization value POF PG UVOJOH #VU BT ZPVMM TFF FWFO NJME TLFQUJDJTN DBO IFMQ B JOH CFUUFS JT BMM XF DBO SFBMMZ IPQF GPS JO UIF MBSHF XPSME XIFSF OP NBM FS UIJT (BVTTJBO NPEFM ZJ ∼ /PSNBM(µJ, σ) µJ = α + βYJ α ∼ /PSNBM(, ) β ∼ /PSNBM(, ) σ ∼ 6OJGPSN(, ) JDF UIBU UIF QSFEJDUPS Y JT TUBOEBSEJ[FE TP UIBU JUT TUBOEBSE EFWJBUJPO ćFO UIF QSJPS PO α JT B OFBSMZĘBU QSJPS UIBU IBT OP QSBDUJDBM FČFDU FFO JO FBSMJFS DIBQUFST OBSSPXFS BOE JT NFBOU UP SFHVMBSJ[F ćF QSJPS β ∼ /PSNBM(, ) IF EBUB UIF NBDIJOF TIPVME CF WFSZ TLFQUJDBM PG WBMVFT BCPWF  BOE O QSJPS XJUI B TUBOEBSE EFWJBUJPO PG  BTTJHOT POMZ  QMBVTJCJMJUZ   07&3'*55*/( 3&(6-"3 -3 -2 -1 0 1 2 3 0.0 0.5 1.0 1.5 2.0 parameter value Density prior N(0,1) N(0,0.5) N(0,0.2) Figure 7.8
17. Regularization  3&(6-"3*;"5*0/  1 2 3 4 5 48 50 52 54 56 58 60 number of parameters deviance N = 20 N(0,1) N(0,0.5) N(0,0.2) 1 2 3 4 5 260 265 270 275 280 285 number of parameters deviance N = 100 'ĶĴłĿIJ ƎƑ 3FHVMBSJ[JOH QSJPST BOE PVUPGTBNQMF EFWJBODF ćF QPJOUT JO   07&3'*55*/( 3&(6-"3*;"5*0/ " -3 -2 -1 0 1 2 3 0.0 0.5 1.0 1.5 2.0 parameter value Density 'ĶĴłĿIJ TUSPOH TUBOEBSE ĕUUJOH /PSNBM TPMJE / 4P UIF QSPCMFN JT SFBMMZ POF PG UVOJOH #VU BT ZP NPEFM EP CFUUFS BOE EPJOH CFUUFS JT BMM XF DBO SF NPEFM OPS QSJPS JT PQUJNBM N(0,1) N(0,0.5) N(0,0.2) in sample Figure 7.9
18. Regularization  3&(6-"3*;"5*0/  1 2 3 4 5 48 50 52 54 56 58 60 number of parameters deviance N = 20 N(0,1) N(0,0.5) N(0,0.2) 1 2 3 4 5 260 265 270 275 280 285 number of parameters deviance N = 100 'ĶĴłĿIJ ƎƑ 3FHVMBSJ[JOH QSJPST BOE PVUPGTBNQMF EFWJBODF ćF QPJOUT JO   07&3'*55*/( 3&(6-"3*;"5*0/ " -3 -2 -1 0 1 2 3 0.0 0.5 1.0 1.5 2.0 parameter value Density 'ĶĴłĿIJ TUSPOH TUBOEBSE ĕUUJOH /PSNBM TPMJE / 4P UIF QSPCMFN JT SFBMMZ POF PG UVOJOH #VU BT ZP NPEFM EP CFUUFS BOE EPJOH CFUUFS JT BMM XF DBO SF NPEFM OPS QSJPS JT PQUJNBM N(0,1) N(0,0.5) N(0,0.2) in sample out of sample Figure 7.9
19. Regularization  3&(6-"3*;"5*0/  1 2 3 4 5 48 50 52 54 56 58 60 number of parameters deviance N = 20 N(0,1) N(0,0.5) N(0,0.2) 1 2 3 4 5 260 265 270 275 280 285 number of parameters deviance N = 100 'ĶĴłĿIJ ƎƑ 3FHVMBSJ[JOH QSJPST BOE PVUPGTBNQMF EFWJBODF ćF QPJOUT JO in sample out of sample in sample out of sample Figure 7.9
20. Regularization Rare in Science • Why don’t more scientists regularize? • Never taught it • Makes significant results rarer • Most scientists judged not on predictive accuracy
21. Cross-validation & Information criteria • Can we estimate out-of-sample deviance? • In theory: Cross-validation • Also in theory: Information criteria • Both tend to perform similarly
22. Figure 7.5   6-:44&4 $0.1"44 body mass (kg) brain volume (cc) 35 47 60 450 900 1300 m7.1 body mass (kg) brain volume (cc) 35 47 60 0 900 2000 m7.4 Cross-validation • Leave out some observations • Train on remaining; score on those left out • Average over many leave-out sets is estimate of out- of-sample accuracy
23. Smooth Cross-validation • Most common: Leave-one-out • Very expensive! • Useful approximation: Importance sampling (IS) • More useful: Pareto-smoothed importance sampling (PSIS) • PSIS-LOO accurate, lots of useful diagnostics • LOO function in rethinking • See also loo package Prof Aki Vehtari (Helsinki), smooth estimator
24. Akaike information criterion • Estimate K-L Distance in theory • Most famous is the first, AIC • Under some strict conditions: Hirotugu Akaike (1927–2009) NQVUF UIF EFWJBODF PO UIF UFTU TBNQMF ćJT NFBOT VTJOH UIF ."1 NBUFT GSPN TUFQ  UP DPNQVUF UIF EFWJBODF GPS UIF EBUB JO UIF UFTU QMF $BMM UIJT EFWJBODF %UFTU  NQVUF UIF EJČFSFODF %UFTU − %USBJO  ćJT EJČFSFODF XJMM VTVBMMZ CF UJWF CFDBVTF UIF NPEFM XJMM UFOE UP QFSGPSN XPSTF IBWF B IJHIFS BODF JO UFTUJOH UIBO JO USBJOJOH MMZ JNBHJOF SFQFBUJOH UIJT QSPDFEVSF NBOZ UJNFT ćF BWFSBHF EJG ODF UIFO UFMMT VT UIF FYQFDUFE PWFSĕUUJOH IPX NVDI UIF USBJOJOH EF DF VOEFSFTUJNBUFT UIF EJWFSHFODF PG UIF NPEFM WF MPHJD B HBNCJU CFDBVTF JU DBOOPU QSPWJEF HVBSBOUFFT #VU JU DBO CMF BEWJDF *U UVSOT PVU UIBU UIJT HBNCJU MFBET UP BO BTUPOJTIJOHMZ B GPS UIF FYQFDUFE UFTUTBNQMF EFWJBODF "*$ = %USBJO + L ≈ & %UFTU OVNCFS PG QBSBNFUFST JO UIF NPEFM ćF UFSN L JT PęFO DBMMFE UIF *U JT B NFBTVSF PG FYQFDUFE PWFSĕUUJOH U EFQFOET VQPO XFBL QSJPST B (BVTTJBO QPTUFSJPS EJTUSJCVUJPO BOE SBNFUFST L NVDI MFTT UIBO UIF OVNCFS PG DBTFT / 4P JUT BQQSP k is parameter count [ah–ka–ee–kay]
25. Widely Applicable IC • AIC of historical interest now • Widely Applicable Information Criterion (WAIC) • Sumio Watanabe 2010 • Does not assume Gaussian posterior • WAIC function in rethinking OVNCFS PG QBSBNFUFST QSPWJEFE UIBU UIF NPEFM VOEFS DPOTJEFSBUJPO JTOU UPP BUB HFOFSBUJOH NPEFM .PSF JOUVJUJWFMZ UIF "LBJLF SFTVMU USBEFT PO B TZNNFUSZ OE PVUPGTBNQMF ćF UXP TBNQMFT BSF FYDIBOHFBCMF ćFZ DPNF GSPN UIF TBNF PO <OPU ĕOJTIFE  OFFE UP SFMBUF %JO BOE %PVU UP %CBS UP EFMJWFS TPNF JOUVJUJPO TOU JOUVJUJWF> NQVUF 8"*$ 6OGPSUVOBUFMZ JUT HFOFSBMJUZ DPNFT BU UIF FYQFOTF PG B NPSF VMB #VU SFBMMZ JU KVTU IBT UXP QJFDFT BOE ZPV DBO DPNQVUF CPUI EJSFDUMZ N UIF QPTUFSJPS EJTUSJCVUJPO 8"*$ JT KVTU UIF MPHQPTUFSJPSQSFEJDUJWF  UIBU XF DBMDVMBUFE FBSMJFS QMVT B QFOBMUZ QSPQPSUJPOBM UP UIF WBSJBODF FEJDUJPOT 8"*$(Z, Θ) = − MQQE − J WBS Θ MPH Q(ZJ|Θ) QFOBMUZ UFSN FSWBUJPOT BOE Θ JT UIF QPTUFSJPS EJTUSJCVUJPO ćF QFOBMUZ UFSN NFBOT BODF JO MPHQSPCBCJMJUJFT GPS FBDI PCTFSWBUJPO J BOE UIFO TVN VQ UIFTF F UPUBM QFOBMUZw 4P ZPV DBO UIJOL PG FBDI PCTFSWBUJPO BT IBWJOH JUT PXO DPSF "OE TJODF UIFTF TDPSFT NFBTVSF PWFSĕUUJOH SJTL ZPV DBO BMTP BTTFTT
26. 1 2 3 4 5 56.0 57.0 58.0 59.0 number of parameters average deviance N = 20 (1000 simulations) WAIC LOOIC LOOCV test 1 2 3 4 5 6.0 6.5 7.0 number of parameters average error (test deviance) N = 20 (1000 simulations) flat prior sigma = 0.5 1 2 3 4 5 270 275 280 285 number of parameters average deviance N = 100 (1000 simulations) 1 2 3 4 5 13.0 14.0 15.0 number of parameters average error (test deviance) N = 100 (1000 simulations) flat prior sigma = 0.5 Figure 7.10 Flat priors Regularizing priors Compare out-of-sample only: LOOCV: Actual leave-one-out CV LOOIC: PSIS-LOO WAIC: Widely Applicable IC Points are actual scores
27. 1 2 3 4 5 56.0 57.0 58.0 59.0 number of parameters average deviance N = 20 (1000 simulations) WAIC LOOIC LOOCV test 1 2 3 4 5 6.0 6.5 7.0 number of parameters average error (test deviance) N = 20 (1000 simulations) flat prior sigma = 0.5 1 2 3 4 5 270 275 280 285 number of parameters average deviance N = 100 (1000 simulations) 1 2 3 4 5 13.0 14.0 15.0 number of parameters average error (test deviance) N = 100 (1000 simulations) flat prior sigma = 0.5 Figure 7.10
28. 1 2 3 4 5 56.0 57.0 58.0 59.0 number of parameters average deviance N = 20 (1000 simulations) WAIC LOOIC LOOCV test 1 2 3 4 5 6.0 6.5 7.0 number of parameters average error (test deviance) N = 20 (1000 simulations) flat prior sigma = 0.5 1 2 3 4 5 270 275 280 285 number of parameters average deviance N = 100 (1000 simulations) 1 2 3 4 5 13.0 14.0 15.0 number of parameters average error (test deviance) N = 100 (1000 simulations) flat prior sigma = 0.5 Figure 7.10
29. Using CV & WAIC • Avoid model selection • Practice model comparison • Multiple models for causal inference • Multiple models competing to explain
30. Example: Model Mis-selection • Model comparison is not causal inference • Recall plant/fungus example from last week • Three models • m6.6: intercept only • m6.7: treatment + fungus • m6.8: treatment only 5P CFHJO MFUT VTF UIF   DPOWFOJFODF GVODUJPO UP DBMDVMBUF 8"*$ GPS ( 3 DPEF  . /ǡ. ǿǎǎȀ  ǿ (Ǔǡǔ Ȁ ȁǎȂ ǐǓǎǡǑǒǎǎ //-ǿǢǫ'++ǫȀ ȁǎȂ ǶǎǔǔǡǎǔǏǑ //-ǿǢǫ+ ǫȀ ȁǎȂ ǐǡǒǒǐǏ //-ǿǢǫ. ǫȀ ȁǎȂ ǎǑǡǎǔǍǐǒ ćF ĕSTU WBMVF JT UIF HVFTT GPS UIF PVUPGTBNQMF EFWJBODF ćF PUIFS WBMVFT B OFOUT BOE UIF TUBOEBSE FSSPS ćF 0WFSUIJOLJOH CPY JO UIF QSFWJPVT TFDUJPO DBMDVMBUF UIFTF OVNCFST GSPN TDSBUDI 5P NBLF JU FBTJFS UP DPNQBSF NVMUJQM - /#$)&$)" QBDLBHF QSPWJEFT B DPOWFOJFODF GVODUJPO *(+-  3 DPEF  . /ǡ. ǿǔǔȀ *(+- ǿ (ǓǡǓ Ǣ (Ǔǡǔ Ǣ (ǓǡǕ Ȁ   +    2 $"#/   (Ǔǡǔ ǐǓǎǡǖ ǐǡǕ ǍǡǍ ǎ ǎǑǡǏǓ  (ǓǡǕ ǑǍǏǡǕ ǏǡǓ ǑǍǡǖ Ǎ ǎǎǡǏǕ ǎǍǡǑǕ (ǓǡǓ ǑǍǒǡǖ ǎǡǓ ǑǑǡǍ Ǎ ǎǎǡǓǓ ǎǏǡǏǐ -00*4 XJMM HJWF ZPV UIF TBNF WBMVFT :PV DBO BEE !0)ʙ  UP UIF *(+- 3 DPEF  '$--4ǿ"$//4Ȁ +')/Ǿ" ʚǶ "$//4ǿ ǫ" ȃ Ǎ Ƕʛ ǎ  Ƕʛ ǎ  Ƕʛ  ȄǫȀ **-$)/ .ǿ +')/Ǿ" Ȁ ʚǶ '$./ǿ 3ʙǿ ǍʙǍǢʙǏǢʙǎǡǒǢ ǎʙǎȀ Ǣ 4ʙǿ ǍʙǍǢʙǍǢʙǎǢ ǎʙǏȀ Ȁ +'*/ǿ +')/Ǿ" Ȁ F H0 H1 T 4P UIF USFBUNFOU 5 JOĘVFODFT UIF QSFTFODF PG GVOHVT ' XIJDI JOĘVFODFT Q  ) 1MBOU IFJHIU BU UJNF  JT BMTP JOĘVFODFE CZ QMBOU IFJHIU BU UJNF %"( 8IFO XF JODMVEF ' UIF QPTUUSFBUNFOU FČFDU JO UIF NPEFM XF FO QBUI GSPN UIF USFBUNFOU UP UIF PVUDPNF ćJT JT UIF %"( XBZ PG TBZJOH USFBUNFOU UFMMT VT OPUIJOH BCPVU UIF PVUDPNF PODF XF LOPX UIF GVOHVT T "O FWFO NPSF %"( XBZ UP TBZ UIJT JT UIBU DPOEJUJPOJOH PO ' JOEVD ćF iEw TUBOET GPS EFQFOEFODF ETFQBSBUJPO NFBOT UIBU TPNF WBSJBCMFT B PUIFST HJWFO UIBU XF DPOEJUJPO PO TPNF PUIFS TFU PG WBSJBCMFT *O UIJT DBTF GSPN 5 XIFO XF DPOEJUJPO PO ' *G XF EP OPU DPOEJUJPO PO ' UIFO UIFZ B
31. • WAIC: estimated out-of-sample log-score • pWAIC: penalty, “effective number of parameters” • dWAIC: difference from top model • weight: Akaike weight — see text for details • SE: Standard error of WAIC • dSE: Standard error of dWAIC - /#$)&$)" QBDLBHF QSPWJEFT B DPOWFOJFODF GVODUJPO  3 DPEF  . /ǡ. ǿǔǔȀ *(+- ǿ (ǓǡǓ Ǣ (Ǔǡǔ Ǣ (ǓǡǕ Ȁ   +    2 $"#/   (Ǔǡǔ ǐǓǎǡǖ ǐǡǕ ǍǡǍ ǎ ǎǑǡǏǓ  (ǓǡǕ ǑǍǏǡǕ ǏǡǓ ǑǍǡǖ Ǎ ǎǎǡǏǕ ǎǍǡǑǕ (ǓǡǓ ǑǍǒǡǖ ǎǡǓ ǑǑǡǍ Ǎ ǎǎǡǓǓ ǎǏǡǏǐ -00*4 XJMM HJWF ZPV UIF TBNF WBMVFT :PV DBO BEE !0)ʙ 8IBU EP BMM PG UIFTF OVNCFST NFBO ćF ĕSTU DPMVNO D WBMVFT BSF CFUUFS BOE UIF NPEFMT BSF PSEFSFE CZ 8"*$ GSP DMVEFT UIF GVOHVT WBSJBCMF IBT UIF TNBMMFTU 8"*$ BT QSPN JT UIF QFOBMUZ UFSN PG 8"*$ ćFTF WBMVFT BSF DMPTF UP EJNFOTJPOT JO UIF QPTUFSJPS PG FBDI NPEFM XIJDI JT UP CF SFHVMBSJ[JOH QSJPST treat + fungus fungus intercept
32. - /#$)&$)" QBDLBHF QSPWJEFT B DPOWFOJFODF GVODUJPO  3 DPEF  . /ǡ. ǿǔǔȀ *(+- ǿ (ǓǡǓ Ǣ (Ǔǡǔ Ǣ (ǓǡǕ Ȁ   +    2 $"#/   (Ǔǡǔ ǐǓǎǡǖ ǐǡǕ ǍǡǍ ǎ ǎǑǡǏǓ  (ǓǡǕ ǑǍǏǡǕ ǏǡǓ ǑǍǡǖ Ǎ ǎǎǡǏǕ ǎǍǡǑǕ (ǓǡǓ ǑǍǒǡǖ ǎǡǓ ǑǑǡǍ Ǎ ǎǎǡǓǓ ǎǏǡǏǐ -00*4 XJMM HJWF ZPV UIF TBNF WBMVFT :PV DBO BEE !0)ʙ 8IBU EP BMM PG UIFTF OVNCFST NFBO ćF ĕSTU DPMVNO D WBMVFT BSF CFUUFS BOE UIF NPEFMT BSF PSEFSFE CZ 8"*$ GSP DMVEFT UIF GVOHVT WBSJBCMF IBT UIF TNBMMFTU 8"*$ BT QSPN JT UIF QFOBMUZ UFSN PG 8"*$ ćFTF WBMVFT BSF DMPTF UP EJNFOTJPOT JO UIF QPTUFSJPS PG FBDI NPEFM XIJDI JT UP CF SFHVMBSJ[JOH QSJPST treat + fungus fungus intercept  6-:44&4 $0.1"44 m6.6 m6.8 m6.7 360 370 380 390 400 410 420 deviance WAIC F ĕMMFE QPJOUT BSF UIF JOTBNQMF EFWJBODF WBMVFT ćF PQFO QPJOUT BSF UIF 8"*$ WBMVF UJDF UIBU OBUVSBMMZ FBDI NPEFM EPFT CFUUFS JOTBNQMF UIBO JU JT FYQFDUFE UP EP PVUP in-sample out SE dWAIC
33. Something About Cebus • Why do primates live a long time? • Consider: ǐǍǎȀ ǐǍǎ VNFOUBUJPO Ǩ-$(/ .ǐǍǎ GPS EFUBJMT 8FMM XPSL XJUI KVTU UISFF WBSJBCMFT F MPOHFWJUZ NBYJNVN MJGFTQBO JO NPOUIT CPEZ NBTT HSBNT BOE CSBJO DFOUJNFUFST  0VS RVFTUJPO DPODFSOT UIF JOĘVFODF PG CPEZ NBTT BOE CSBJO Z *U NBLFT TFOTF UIBU MBSHFS TQFDJFT MJWF MPOHFS‰GFXFS UIJOHT DBO LJMM UIFN TP MJLFMZ UP CF DPNNPO VOPCTFSWFE WBSJBCMFT UIBU JOĘVFODF CPUI CPEZ TJ[F #SBJO WPMVNF JT LOPXO UP CF IJHIMZ DPSSFMBUFE XJUI CPEZ TJ[F #VU XF XBOU UP BOZ EJSFDU JOĘVFODF PG CSBJO TJ[F PO MPOHFWJUZ QFSIBQT CFDBVTF CFJOH TNBSU MT UP TVSWJWF MPOHFS 1VUUJOH UIJT WFSCBM NPEFM JOUP %"( GPSN B L M U # JT CSBJO WPMVNF - JT MPOHFWJUZ BOE 6 JT TPNF TFU PG VOPCTFSWFE WBSJBCMFT → - 5P JOGFS UIF EJSFDU JOĘVFODF PG CSBJO WPMVNF PO MPOHFWJUZ XF KVTU OFFE PEZ NBTT‰UIBU XJMM DMPTF UIF CBDLEPPS # ← . → - 0G DPVSTF UIF EJSFDU brain size lifespan unobserved confounds body mass + + +
34. Something About Cebus • 112 primate species • Three models: • m7.8: log L ~ log M + log B • m7.9: log L ~ log B • m7.10: log L ~ log M • Funny stuff happens WPMVNF DVCJD DFOUJNFUFST  0VS RVFTUJPO DPODFSOT UIF JOĘVFO TJ[F PO MPOHFWJUZ *U NBLFT TFOTF UIBU MBSHFS TQFDJFT MJWF MPOHFS‰ #VU UIFSF BSF BMTP MJLFMZ UP CF DPNNPO VOPCTFSWFE WBSJBCMFT UI BOE MPOHFWJUZ #SBJO WPMVNF JT LOPXO UP CF IJHIMZ DPSSFMBUFE XJU LOPX JG UIFSF JT BOZ EJSFDU JOĘVFODF PG CSBJO TJ[F PO MPOHFWJUZ QF IFMQT JOEJWJEVBMT UP TVSWJWF MPOHFS 1VUUJOH UIJT WFSCBM NPEFM JOU B L M U . JT CPEZ NBTT # JT CSBJO WPMVNF - JT MPOHFWJUZ BOE 6 JT TPNF DPOGPVOEJOH . → - 5P JOGFS UIF EJSFDU JOĘVFODF PG CSBJO WPMVN UP DPOUSPM GPS CPEZ NBTT‰UIBU XJMM DMPTF UIF CBDLEPPS # ← . JOĘVFODF PG CPEZ NBTT JT TUJMM DPOGPVOEFE CZ 6‰XF DBOOPU DM NFBTVSJOH 6‰TP JU NJHIU MPPL MBSHFS UIBO JU SFBMMZ JT #VU XF T HPPE JOGFSFODF PG # → - "MM PG UIJT JT BTTVNJOH UIF %"( JT DPSS 4P XF XBOU B NPEFM UIBU SFHSFTTFT - PO # BOE . TJNVMUBOFP UIF MPH PG FBDI WBSJBCMF 8IZ #FDBVTF XF EPOU FYQFDU MJOFBS SFM + + + /ǡ. ǿǐǍǎȀ *(+- ǿ (ǔǡǕ Ǣ (ǔǡǖ Ǣ (ǔǡǎǍ Ȁ   +    2 $"#/   ǔǡǕ ǏǎǓǡǏ ǐǡǒ ǍǡǍ Ǎǡǒǐ ǎǑǡǔǏ  ǔǡǖ ǏǎǓǡǒ ǏǡǓ Ǎǡǐ ǍǡǑǔ ǎǑǡǕǑ ǎǡǒǎ ǔǡǎǍ ǏǏǖǡǑ Ǐǡǒ ǎǐǡǏ ǍǡǍǍ ǎǓǡǐǍ ǔǡǍǎ ćFTF WBMVFT BSF FBTJFS UP BQQSFDJBUF BT B QMPU '*/ǿ *(+- ǿ (ǔǡǕ Ǣ (ǔǡǖ Ǣ (ǔǡǎǍ Ȁ Ȁ m7.10 m7.9 m7.8 210 220 230 240 deviance WAIC
35. Something About Cebus • Inspect the posterior distributions for answers • Why does body mass go negative in joint model?  64*/( $30447"-*%"5*0/ "/% */'03."5*0/ $3*5&3*"  m7.8 m7.9 m7.10 m7.8 m7.9 m7.10 bM bB -0.5 0.0 0.5 1.0 Value UIBU UIF QPTUFSJPS EJTUSJCVUJPOT PG  BOE  BSF NVDI XJEFS JO UIF NPEFM UIBU DPO PUI ćJT NJHIU SFNJOE ZPV PG UIF NVMUJDPMMJOFBSJUZ FYBNQMFT GSPN $IBQUFS  *OEFFE
36. Pointwise perspective  64*/( $30447"-*%"5*0/ "/% */'03."5*0/ $3*5&3*" -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 -2 -1 0 1 2 pointwise difference in WAIC log longevity (std) ← m7.8 better m7.9 better → Cacajao melanocephalus Cebus apella Cebus capucinus Cebus olivaceus Cercopithecus lhoesti Eulemur fulvus Gorilla gorilla Lepilemur leucopus 'ĶĴłĿIJ ƏƉƊ %JČFSFOUJBM QPJOUXJTF QSFEJDUJWF BDDVSBDZ PG UIF QSJNBUF MPOHFWJUZ NPEFMT ćF IPSJ[POUBM BYJT JT UIF QPJOUXJTF EJČFSFODF JO 8"*$ GPS FBDI TQFDJFT TVCUSBDUJOH 8"*$ GSPN (ǔǡǕ UIF NPEFM XJUI CPUI QSFEJD Figure 7.12 Point size proportional to abs diff brain z-score – body z-score
37. Pointwise perspective  64*/( $30447"-*%"5*0/ "/% */'03."5*0/ $3*5&3*" -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 -2 -1 0 1 2 pointwise difference in WAIC log longevity (std) ← m7.8 better m7.9 better → Cacajao melanocephalus Cebus apella Cebus capucinus Cebus olivaceus Cercopithecus lhoesti Eulemur fulvus Gorilla gorilla Lepilemur leucopus 'ĶĴłĿIJ ƏƉƊ %JČFSFOUJBM QPJOUXJTF QSFEJDUJWF BDDVSBDZ PG UIF QSJNBUF MPOHFWJUZ NPEFMT ćF IPSJ[POUBM BYJT JT UIF QPJOUXJTF EJČFSFODF JO 8"*$ GPS FBDI TQFDJFT TVCUSBDUJOH 8"*$ GSPN (ǔǡǕ UIF NPEFM XJUI CPUI QSFEJD Figure 7.12 Model with brain + mass better Cebus: Large brain for body Lepilemur: Small brain for body Point size proportional to abs diff brain z-score – body z-score
38. Pointwise perspective  64*/( $30447"-*%"5*0/ "/% */'03."5*0/ $3*5&3*" -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 -2 -1 0 1 2 pointwise difference in WAIC log longevity (std) ← m7.8 better m7.9 better → Cacajao melanocephalus Cebus apella Cebus capucinus Cebus olivaceus Cercopithecus lhoesti Eulemur fulvus Gorilla gorilla Lepilemur leucopus 'ĶĴłĿIJ ƏƉƊ %JČFSFOUJBM QPJOUXJTF QSFEJDUJWF BDDVSBDZ PG UIF QSJNBUF MPOHFWJUZ NPEFMT ćF IPSJ[POUBM BYJT JT UIF QPJOUXJTF EJČFSFODF JO 8"*$ GPS FBDI TQFDJFT TVCUSBDUJOH 8"*$ GSPN (ǔǡǕ UIF NPEFM XJUI CPUI QSFEJD Figure 7.12 Model with brain better Point size proportional to abs diff brain z-score – body z-score Gorilla: Big brain & body Cacajao: Large brain for body, but average lifespan
39. Cebus Collider • Another idea: Reversal of body size coefficient consistent with collider bias (*ominous music*) • Conditioning on brain opens backdoor path M –> B <– L TPNFUIJOH JNQPSUBOU BU MFBTU GPS $FCVT ćFSF JT TPNFUIJOH BCPVU $FCVT 8F TUJMM I NZTUFSZ PG XIZ UIF QPTUFSJPS GPS  TIPVME CF MBSHFMZ OFHBUJWF XIFO UIFSF BSF TUS PSFUJDBM BOE FNQJSJDBM SFBTPOT UP CFMJFWF UIBU MBSHFS BOJNBMT MJGF MPOHFS MJWFT *T UIF UP FYQMBJO CPUI QJFDFT UIBU B NPEFM UIBU JHOPSFT CPEZ TJ[F DBOU FYQMBJO FYUSFNFMZ MP E TQFDJFT MJLF $FCVT BOE UIBU UIF TMPQF PO CPEZ TJ[F USFOET OFHBUJWF /P POF LOPXT XIBU JT HPJOH PO JO QSJNBUF MJGF IJTUPSZ FWPMVUJPO #VU * TVTQFDU XIB QFOJOH JO UIF NPEFMT DPOTJEFSFE IFSF JT BOPUIFS DBTF PG‰DVF ESBNBUJD NVTJD‰DPMMJ  ćF %"( UIBU * ESFX BU UIF TUBSU BTTVNFE UIBU CSBJO TJ[F JOĘVFODFE MPOHFWJUZ BU JG JU JT UIF SFWFSTF 8IBU JG JOTUFBE MPOH MJGFTQBOT NBLF JOWFTUNFOU JO MBSHFS CSB SF XPSUIXIJMF 8IZ XPVME UIFZ EP UIJT 'PS UIF TBNF SFBTPO UIBU BOZ JOWFTUNFOU Q SF JG ZPV DBO VTF BNPSUJ[F JU GPS MPOHFS )FSF JT PVU QSFWJPVT %"( PO UIF MFę XJ TFE %"( UIBU SFWFSTFT UIF BSSPX DPOOFDUJOH # BOE - B L M U B L M U PN DPMMJEFS $POEJUJPOJOH PO # PQFOT BO BEEJUJPOBM CBDLEPPS QBUI GSPN . UP - + + +
40. Cebus Collider JWFE TQFDJFT MJLF $FCVT BOE UIBU UIF TMPQF PO CPEZ TJ[F USFOET OFHBUJWF /P POF LOPXT XIBU JT HPJOH PO JO QSJNBUF MJGF IJTUPSZ FWPMVUJPO #VU * TVTQFDU IBQQFOJOH JO UIF NPEFMT DPOTJEFSFE IFSF JT BOPUIFS DBTF PG‰DVF ESBNBUJD NVTJD‰D CJBT ćF %"( UIBU * ESFX BU UIF TUBSU BTTVNFE UIBU CSBJO TJ[F JOĘVFODFE MPOHFWJ XIBU JG JU JT UIF SFWFSTF 8IBU JG JOTUFBE MPOH MJGFTQBOT NBLF JOWFTUNFOU JO MBSHFS NPSF XPSUIXIJMF 8IZ XPVME UIFZ EP UIJT 'PS UIF TBNF SFBTPO UIBU BOZ JOWFTUNF NPSF JG ZPV DBO VTF BNPSUJ[F JU GPS MPOHFS )FSF JT PVU QSFWJPVT %"( PO UIF MFę SFWJTFE %"( UIBU SFWFSTFT UIF BSSPX DPOOFDUJOH # BOE - B L M U B L M U #PPN DPMMJEFS $POEJUJPOJOH PO # PQFOT BO BEEJUJPOBM CBDLEPPS QBUI GSPN . UP DPVSTF UIF DBVTBM SFMBUJPOTIJQ DPVME SVO CPUI XBZT CVU JO UIBU DBTF UIFSF JT TUJMM B D BU # 0OF SFBTPO UP TVTQFDU UIBU UIJT JT UIF DBTF PUIFS UIBO UIFPSZ JT UIF QBUUFSO JO UI 0OF TZNQUPN PG B DPMMJEFS MJLF XF TBX JO UIF BHF BOE IBQQJOFTT FYBNQMF JO $IBQUFS  + + + 0OF TZNQUPN PG B DPMMJEFS MJLF XF TBX JO UIF BHF BOE IBQQJOFTT FYBNQMF JO $IBQUFS  JT UIBU B SFMBUJPOTIJQ MJLF . → - JT EJTUPSUFE EPXOXBSET CZ DPOEJUJPOJOH PO B DPMMJEFS MJLF # *G ZPV TJNVMBUF EBUB DPOTJTUFOU XJUI UIF %"( PO UIF SJHIU ZPV DBO SFDPWFS UIF TBNF RVBMJUBUJWFMZ QBUUFSO PG DPOGPVOEJOH 8IBU JG XF DPOTJEFS B NPEFM UIBU USFBUT CSBJO TJ[F BT UIF PVUDPNF BOE DPOEJUJPOT JO TUFBE PO CPEZ TJ[F BOE MPOHFWJUZ "DDPSEJOH UP UIF TFDPOE %"( BCPWF TVDI B NPEFM JT OPU DPOGPVOEFE 3 DPEF  (ǔǡǎǎ ʚǶ ,0+ǿ '$./ǿ '*"Ǿ ʡ )*-(ǿ (0 Ǣ .$"( ȀǢ (0 ʚǶ  ʔ ȉ'*"Ǿ ʔ  ȉ'*"Ǿ Ǣ  ʡ )*-(ǿǍǢǍǡǎȀǢ  ʡ )*-(ǿǍǢǍǡǒȀǢ  ʡ )*-(ǿǍǢǍǡǒȀǢ .$"( ʡ  3+ǿǎȀ Ȁ Ǣ /ʙǏ Ȁ +- $.ǿ (ǔǡǎǎ Ȁ ( ) . ǒǡǒʉ ǖǑǡǒʉ  ǶǍǡǍǒ ǍǡǍǏ ǶǍǡǍǔ ǶǍǡǍǏ  ǍǡǖǑ ǍǡǍǐ ǍǡǖǍ ǍǡǖǕ  64*/( $30447"-*%"5*0/ "/% */'03."5*0/ $3*5&3*"   ǍǡǎǏ ǍǡǍǐ ǍǡǍǔ ǍǡǎǓ .$"( Ǎǡǎǖ ǍǡǍǎ Ǎǡǎǔ ǍǡǏǎ ćJT NPEFM UIJOLT CPUI CPEZ TJ[F BOE MPOHFWJUZ BSF QPTJUJWFMZ BTTPDJBUFE XJUI CSBJO TJ[F *U XPVME CF VTFGVM OPX UP JOWFTUJHBUF BOE DPNQBSF UIF UXP JNQMJFE TJNQMFS NPEFMT‰POMZ '*"Ǿ BOE POMZ '*"Ǿ ‰BT XFMM *MM BTL ZPV EP UIBU JO UIF QSPCMFNT BU UIF FOE PG UIJT DIBQUFS ćF TDJFOUJĕD MJUFSBUVSF BCPVU QSJNBUF CSBJO FWPMVUJPO JT B CJU PG B NFTT ćF UZQF PG
41. Curse of Tippecanoe • 1840–1960: Every US president elected in year ending in digit “0” died in office • W. H. Harrison first, “Old Tippecanoe” • Lincoln, Garfield, McKinley, Harding, FD Roosevelt • J. F. Kennedy last, assassinated in 1963 • Reagan broke the curse! • Trying all possible models: A formula for overfitting • Be thoughtful • Be honest: Admit data exploration
42. Onwards! • Be patient: We’ll keep using these tools in future • Homework 4: Entropy, happiness, and foxes • Next week: Interactions, MCMC • Coming up: Maximum entropy, generalized linear models, multilevel models