Statistical Rethinking Fall 2017 Lecture 08

Week 4, Lecture 8, Statistical Rethinking: A Bayesian Course with Examples in R and Stan. This lecture covers Chapter 6 of the book.

Transcript

1. Week 4: Ockham, Ulysses, and the Model Richard McElreath Statistical Rethinking
2.   07&3'*55*/( 3&(6-"3*;"5*0/ "/% */'03."5*0/ $3*5&3*" 1 2 3 4 5 45 50 55 60 65 number of parameters deviance N = 20 in out +1SD –1SD 1 2 3 4 5 250 260 270 280 290 300 number of parameters deviance N = 100 in out 'ĶĴłĿIJ ƎƏ %FWJBODF JO BOE PVU PG TBNQMF *O FBDI QMPU NPEFMT XJUI EJG GFSFOU OVNCFST PG QSFEJDUPS WBSJBCMFT BSF TIPXO PO UIF IPSJ[POUBM BYJT %F WJBODF BDSPTT UIPVTBOE TJNVMBUJPOT JT TIPXO PO UIF WFSUJDBM #MVF TIPXT Everybody overfits
3. Regularization • Use informative, conservative priors to reduce overfitting => model learns less from sample • But if too informative, model learns too little • Such priors are regularizing 1 0 1 2 3 rameter value /PSNBM(, ) ćJO TPMJE /PSNBM(, .) ćJDL TPMJE /PSNBM(, .) T SFBMMZ POF PG UVOJOH #VU BT ZPVMM TFF FWFO NJME TLFQUJDJTN DBO IFMQ B BOE EPJOH CFUUFS JT BMM XF DBO SFBMMZ IPQF GPS JO UIF MBSHF XPSME XIFSF OP JT PQUJNBM DPOTJEFS UIJT (BVTTJBO NPEFM ZJ ∼ /PSNBM(µJ, σ) µJ = α + βYJ α ∼ /PSNBM(, ) β ∼ /PSNBM(, ) σ ∼ 6OJGPSN(, ) E QSBDUJDF UIBU UIF QSFEJDUPS Y JT TUBOEBSEJ[FE TP UIBU JUT TUBOEBSE EFWJBUJPO JT [FSP ćFO UIF QSJPS PO α JT B OFBSMZĘBU QSJPS UIBU IBT OP QSBDUJDBM FČFDU   07&3'*55*/( 3&(6-"3*;"5*0/ -3 -2 -1 0 1 2 3 0.0 0.5 1.0 1.5 2.0 parameter value Density 'ĶĴłĿIJ TUSPOH TUBOEBS ĕUUJOH /PSNB TPMJE / regularizing prior N(0,1) N(0,0.5) N(0,0.2)
4. Regularization  3&(6-"3*;"5*0/  1 2 3 4 5 48 50 52 54 56 58 60 number of parameters deviance N = 20 N(0,1) N(0,0.5) N(0,0.2) 1 2 3 4 5 260 265 270 275 280 285 number of parameters deviance N = 100 'ĶĴłĿIJ ƎƑ 3FHVMBSJ[JOH QSJPST BOE PVUPGTBNQMF EFWJBODF ćF QPJOUT JO   07&3'*55*/( 3&(6-"3*;"5*0/ " -3 -2 -1 0 1 2 3 0.0 0.5 1.0 1.5 2.0 parameter value Density 'ĶĴłĿIJ TUSPOH TUBOEBSE ĕUUJOH /PSNBM TPMJE / 4P UIF QSPCMFN JT SFBMMZ POF PG UVOJOH #VU BT ZP NPEFM EP CFUUFS BOE EPJOH CFUUFS JT BMM XF DBO SF NPEFM OPS QSJPS JT PQUJNBM N(0,1) N(0,0.5) N(0,0.2) in sample
5. Regularization  3&(6-"3*;"5*0/  1 2 3 4 5 48 50 52 54 56 58 60 number of parameters deviance N = 20 N(0,1) N(0,0.5) N(0,0.2) 1 2 3 4 5 260 265 270 275 280 285 number of parameters deviance N = 100 'ĶĴłĿIJ ƎƑ 3FHVMBSJ[JOH QSJPST BOE PVUPGTBNQMF EFWJBODF ćF QPJOUT JO   07&3'*55*/( 3&(6-"3*;"5*0/ " -3 -2 -1 0 1 2 3 0.0 0.5 1.0 1.5 2.0 parameter value Density 'ĶĴłĿIJ TUSPOH TUBOEBSE ĕUUJOH /PSNBM TPMJE / 4P UIF QSPCMFN JT SFBMMZ POF PG UVOJOH #VU BT ZP NPEFM EP CFUUFS BOE EPJOH CFUUFS JT BMM XF DBO SF NPEFM OPS QSJPS JT PQUJNBM N(0,1) N(0,0.5) N(0,0.2) in sample out of sample
6. Regularization  3&(6-"3*;"5*0/  1 2 3 4 5 48 50 52 54 56 58 60 number of parameters deviance N = 20 N(0,1) N(0,0.5) N(0,0.2) 1 2 3 4 5 260 265 270 275 280 285 number of parameters deviance N = 100 'ĶĴłĿIJ ƎƑ 3FHVMBSJ[JOH QSJPST BOE PVUPGTBNQMF EFWJBODF ćF QPJOUT JO in sample out of sample in sample out of sample
7. Information criteria • Can we estimate out-of-sample deviance? • In theory: Cross-validation • Also in theory: Information criteria • Information, because use of deviance based on information theoretic analysis • Criteria, because used to compare models • Information criteria estimate relative out of sample error • AIC, DIC, WAIC, many others
8. Akaike information criterion • A meta-model of forecasting: • Two samples: training and testing, size N • Fit model to training sample, get Dtrain • Use fit to training to compute Dtest • Difference Dtest – Dtrain is overfitting • Under some strict conditions: Hirotugu Akaike (1927–2009) NQVUF UIF EFWJBODF PO UIF UFTU TBNQMF ćJT NFBOT VTJOH UIF ."1 NBUFT GSPN TUFQ  UP DPNQVUF UIF EFWJBODF GPS UIF EBUB JO UIF UFTU QMF $BMM UIJT EFWJBODF %UFTU  NQVUF UIF EJČFSFODF %UFTU − %USBJO  ćJT EJČFSFODF XJMM VTVBMMZ CF UJWF CFDBVTF UIF NPEFM XJMM UFOE UP QFSGPSN XPSTF IBWF B IJHIFS BODF JO UFTUJOH UIBO JO USBJOJOH MMZ JNBHJOF SFQFBUJOH UIJT QSPDFEVSF NBOZ UJNFT ćF BWFSBHF EJG ODF UIFO UFMMT VT UIF FYQFDUFE PWFSĕUUJOH IPX NVDI UIF USBJOJOH EF DF VOEFSFTUJNBUFT UIF EJWFSHFODF PG UIF NPEFM WF MPHJD B HBNCJU CFDBVTF JU DBOOPU QSPWJEF HVBSBOUFFT #VU JU DBO CMF BEWJDF *U UVSOT PVU UIBU UIJT HBNCJU MFBET UP BO BTUPOJTIJOHMZ B GPS UIF FYQFDUFE UFTUTBNQMF EFWJBODF "*$ = %USBJO + L ≈ & %UFTU OVNCFS PG QBSBNFUFST JO UIF NPEFM ćF UFSN L JT PęFO DBMMFE UIF *U JT B NFBTVSF PG FYQFDUFE PWFSĕUUJOH U EFQFOET VQPO XFBL QSJPST B (BVTTJBO QPTUFSJPS EJTUSJCVUJPO BOE SBNFUFST L NVDI MFTT UIBO UIF OVNCFS PG DBTFT / 4P JUT BQQSP k is parameter count [ah–ka–ee–kay]
9. Akaike information criterion • Conditions: • You like the AIC forecasting model • Flat priors • No varying/mixed/random effects • Gaussian posterior distribution • k << N; as k approaches N: WJBODF VOEFSFTUJNBUFT UIF EJWFSHFODF PG UIF NPEFM BMM UIF BCPWF MPHJD B HBNCJU CFDBVTF JU DBOOPU QSPWJEF HVBSBOUFFT #VU JU D PWJEF WBMVBCMF BEWJDF *U UVSOT PVU UIBU UIJT HBNCJU MFBET UP BO BTUPOJTIJO NQMF GPSNVMB GPS UIF FYQFDUFE UFTUTBNQMF EFWJBODF "*$ = %USBJO + L ≈ & %UFTU IFSF L JT UIF OVNCFS PG QBSBNFUFST JO UIF NPEFM ćF UFSN L JT PęFO DBMMFE U OBMUZ UFSN *U JT B NFBTVSF PG FYQFDUFE PWFSĕUUJOH ćJT SFTVMU EFQFOET VQPO XFBL QSJPST B (BVTTJBO QPTUFSJPS EJTUSJCVUJPO B NCFS PG QBSBNFUFST L NVDI MFTT UIBO UIF OVNCFS PG DBTFT / 4P JUT BQQ BUF GPS PSEJOBSZ MJOFBS SFHSFTTJPO BOE JU FWFO XPSLT RVJUF XFMM GPS NBOZ OP VTTJBO SFHSFTTJPOT HFOFSBMJ[FE MJOFBS NPEFMT (-.T UIBU XFMM FYBNJOF MB UIJT CPPL UIJOLJOH "*$ BOE iUSVFw NPEFMT *U JT QPTTJCMF UP SFBE CPUI UIBU  "*$ BTTVN EBUB HFOFSBUJOH NPEFM JT POF PG UIF DBOEJEBUF NPEFMT BOE  "*$ EPFT OPU BTTV EBUB HFOFSBUJOH NPEFM JT B DBOEJEBUF ćJT DPOGVTJPO BSJTFT CFDBVTF UIFSF BSF NVMUJ ZT UP EFSJWF "*$ ćF HBNCJU EFTDSJCFE BCPWF EPFT OPU FNQMPZ B iUSVFw NPEFM FYDFQ BU MFBTU JO UIF MBSHF XPSME 4UJMM DBVUJPO SFRVJSFT UBLJOH OPUF PG WJPMBUFE BTTVNQUJPOT BOE IPQFGVMMZ FWBMVBUJOH UIF DPOTFRVFODFT PG UIFTF WJPMBUJPOT  -JNJUT UP "*$T HFOFSBMJUZ #VU NPSF HFOFSBMMZ "*$ JT OPU HFOFSBM *U JT B TQFDJBM DBTF PG NVDI MBSHFS QIFOPNFOPO UIF TFWFSJUZ PG PWFSĕUUJOH JT BO JO DSFBTJOH GVODUJPO PG UIF OVNCFS PG QBSBNFUFST #VU UIJT GVODUJPO JT OPU BMXBZT BT TJNQMF BT L BT JU JT JO "*$ " GFX DPNNPO DPOEJUJPOT CFOFĕU GSPN B NPSF HFOFSBM TPMVUJPO  1BSBNFUFS DPVOU DMPTF UP TBNQMF TJ[F 4VQQPTF B NPEFM IBT L QBSBNF UFST BOE JT ĕU UP / PCTFSWBUJPOT 8IFO L JT DMPTF UP / PWFSĕUUJOH SJTFT WFSZ SBQJEMZ ćJT IBQQFOT CFDBVTF UIF NPEFM TUBSUT QFSGFDUMZ FODPEJOH UIF USBJOJOH TBNQMF 4P XIFO UIF NPEFM TFFT UIF UFTU TBNQMF JUT BMXBZT WFSZ TVSQSJTFE " DPOTFSWBUJWF BQQSPYJNBUJPO GPS UIJT SJTF JO PWFSĕUUJOH JT HJWFO CZ B DPNNPO HFOFSBMJ[BUJPO PG "*$ "*$D = %USBJO + L  − (L + )// 8IFO / JT WFSZ NVDI MBSHFS UIBO L UIF BCPWF TJNQMJĕFT UP QMBJO "*$ #VU BT L BQQSPBDIFT / −  UIF QFOBMUZ PO UIF SJHIU BQQSPBDIFT JOĕOJUZ 4P BOZUJNF "*$ JT BQQSPQSJBUF "*$D NBZ CF B CFUUFS DIPJDF
10. Akaike information criterion • Prediction/forecasting task matters • Suppose we care about accumulated error over learning, aka prequential error • Consider the humble wurst • Grill-only or boil-then-grill? • Want to consume each wurst • How to learn and eat well at same time? • AIC not the right scenario
12. Figure 6.10 training testing AIC  */'03."5*0/ $3*5&3*" 1 2 3 4 5 48 50 52 54 56 58 60 number of parameters deviance N = 20 2 4.1 5.3 7.6 9.7 1 2 3 4 5 260 265 270 275 280 285 number of parameters deviance N = 100 1.9 4.1 4.9 7.1 8.5 'ĶĴłĿIJ ƎƉƈ %FWJBODF JO CMVF BOE PVU CMBDL PG TBNQMF VTJOH ĘBU QSJ PST ćF WFSUJDBM TFHNFOUT NFBTVSF UIF EJTUBODF CFUXFFO FBDI QBJS PG EF WJBODFT 'PS CPUI / =  BOE / =  UIJT EJTUBODF JT BQQSPYJNBUFMZ UXJDF UIF OVNCFS PG QBSBNFUFST ćF EBTIFE MJOFT TIPX FYBDUMZ UIF EFWJBODF JO “true” model
13. Figure 6.10 training testing AIC “true” model  */'03."5*0/ $3*5&3*" 1 2 3 4 5 48 50 52 54 56 58 60 number of parameters deviance N = 20 2 4.1 5.3 7.6 9.7 1 2 3 4 5 260 265 270 275 280 285 number of parameters deviance N = 100 1.9 4.1 4.9 7.1 8.5 'ĶĴłĿIJ ƎƉƈ %FWJBODF JO CMVF BOE PVU CMBDL PG TBNQMF VTJOH ĘBU QSJ PST ćF WFSUJDBM TFHNFOUT NFBTVSF UIF EJTUBODF CFUXFFO FBDI QBJS PG EF WJBODFT 'PS CPUI / =  BOE / =  UIJT EJTUBODF JT BQQSPYJNBUFMZ UXJDF UIF OVNCFS PG QBSBNFUFST ćF EBTIFE MJOFT TIPX FYBDUMZ UIF EFWJBODF JO
14. AIC DIC WAIC
15. Widely Applicable IC • Widely Applicable Information Criterion (WAIC) • Sumio Watanabe 2010 • Sometimes called “Watanabe-Akaike Information Criterion” • Does not assume Gaussian posterior • WAIC function in rethinking J= = / J= MPH & θ 1S(ZJ|θ) MQQE = / J= MPH  4 4 T= 1S(ZJ|θT) Q8"*$ = / J= WBS θ MPH 1S(ZJ|θ) 8"*$ = −MQQE + Q8"*$ "JK ∼ #JOPNJBM(OJ, QJK) MPHJU QJK = α + αK + (βN + βNK)NJK α ∼ /PSNBM(, ) βN ∼ /PSNBM(, )
16. At the beach, finally • Underfitting possible; overfitting inevitable • Regularizing priors reduce it • Information criteria measure it • Taste great together  64*/( */'03."5*0/ $3*5&3*" 1 2 3 4 5 55 56 57 58 59 number of parameters deviance N = 20 DIC N(0,100) 1 2 3 4 5 55 56 57 58 59 number of parameters deviance N = 20 WAIC N(0,0.5) 'ĶĴłĿIJ ƎƉƉ 0VUPGTBNQMF EFWJBODF BT NBUFE CZ %*$ BOE 8"*$ 1PJOUT BSF BWFSBH PGTBNQMF EFWJBODF PWFS UIPVTBOE TJNVMB ćF MJOFT BSF BWFSBHF %*$ UPQ BOE 8"*$ UPN DPNQVUFE GSPN UIF TBNF TJNVMBUJPOT CMBDL QPJOUT BOE MJOFT DPNF GSPN TJNVMBUJPO B OFBSMZĘBU /PSNBM(, ) QSJPS ćF CMVF BOE MJOFT VTFE B SFHVMBSJ[JOH /PSNBM(, .)
17. Using AIC/DIC/WAIC • Avoid model selection • Model comparison: quantify uncertainty about models, in addition to uncertainty about parameters • Model averaging: Simulate predictions, averaging over uncertainty about models • don’t average parameters, but only predictions
18. Primate milk again kcal.per.g -2 0 2 4 0.5 0.7 0.9 -2 0 2 4 log(mass) 0.5 0.7 0.9 55 65 75 55 65 75 neocortex.perc BOE QPTUFSJPS QSFEJDUJWF DIFDLT GSPN FBDI NPEFM *U JT KVTU BT JNQPSUBOU UP VOEFS TUBOE XIZ B NPEFM PVUQFSGPSNT BOPUIFS BT JU JT UP NFBTVSF UIF QFSGPSNBODF EJČFS FODF %*$8"*$ BMPOF TBZT WFSZ MJUUMF BCPVU TVDI EFUBJMT #VU JO DPNCJOBUJPO XJUI PUIFS JOGPSNBUJPO %*$8"*$ JT B CJH IFMQ • .ļıIJĹ ĮŃIJĿĮĴĶĻĴ NFBOT VTJOH %*$8"*$ UP DPOTUSVDU B QPTUFSJPS QSFEJDUJWF EJTUSJCVUJPO UIBU FYQMPJUT XIBU XF LOPX BCPVU SFMBUJWF BDDVSBDZ PG UIF NPEFMT ćJT IFMQT HVBSE BHBJOTU PWFSDPOĕEFODF JO NPEFM TUSVDUVSF JO UIF TBNF XBZ UIBU VTJOH UIF FOUJSF QPTUFSJPS EJTUSJCVUJPO IFMQT HVBSE BHBJOTU PWFSDPOĕEFODF JO QBSBNFUFS WBMVFT 8IBU NPEFM BWFSBHJOH EPFT OPU NFBO JT BWFSBHJOH QBSBNFUFS FTUJNBUFT CFDBVTF QBSBNFUFST JO EJČFSFOU NPEFMT IBWF EJČFSFOU NFBOJOHT BOE TIPVME OPU CF BWFSBHFE VOMFTT ZPV BSF TVSF ZPV BSF JO B TQFDJBM DBTF JO XIJDI JU JT TBGF UP EP TP 4P JU JT CFUUFS UP UIJOL PG NPEFM BWFSBHJOH BT QSFEJDUJPO BWFSBHJOH CFDBVTF UIBUT XIBU JT BDUVBMMZ CFJOH EPOF ćF TFDUJPO EFNPOTUSBUFT IPX UP DPOEVDU DPNQBSJTPO BOE BWFSBHJOH VTJOH B TJNQMF FYBNQMF XJUI B GFX QSFEJDUPS WBSJBCMFT -BUFS DIBQUFST DPOUJOVF VTJOH UIFTF UPPMT BOE UIF EFUBJMT PG FYBNQMFT EP WBSZ 4P CF XBSZ OPU UP PWFSHFOFSBMJ[F UIF FYBNQMF UIBU GPMMPXT  .PEFM DPNQBSJTPO 3FDBMM UIF QSJNBUF NJML EBUB GSPN UIF QSFWJPVT DIBQUFS -FUT MPBE JU JOUP 3 SFNPWF UIF T BOE SFTDBMF POF PG UIF FYQMBOBUPSZ WBSJBCMFT 3 DPEF  !1ǯ*&)(ǰ ! ʆǦ *&)(DZ ,*-)"1"Ǒ 0"0ǯ*&)(ǰ ǒ Dz !ɢ+", ,/1"5 ʆǦ !ɢ+", ,/1"5Ǒ-"/ ǵ ƾƽƽ !&*ǯ!ǰ DZƾDz ƾDŽ dž 4P ZPVS EBUB GSBNF TIPVME BMTP IBWF  SPXT DBTFT BOE  DPMVNOT WBSJBCMFT 
19. Primate milk again • Fit four different models: m6.11: kcal ~ 1 m6.12: kcal ~ 1 + neocortex m6.13: kcal ~ 1 + log(mass) m6.14: kcal ~ 1 + neocortex + log(mass)
20. Comparing • What is expected out-of-sample deviance for each model? ćFSFT OPUIJOH QSFWFOUJOH EFWJBODF GSPN CFJOH OFHBUJWF 4NBMMFS WBMVFT BSF T TFDPOE WBMVF SFQPSUFE JT UIF MQQE ćF UIJSE WBMVF JT Q8"*$  *G ZPV TVCUSBDU - BOE UIFO NVMUJQMZ UIBU EJČFSFODF CZ − ZPVMM HFU UIF 8"*$ WBMVF ćF ĕO UIF TUBOEBSE FSSPS PG UIF 8"*$ WBMVF ćJT TUBOEBSE FSSPS QSPWJEFT SPVHI H VODFSUBJOUZ JO 8"*$ UIBU BSJTFT GSPN TBNQMJOH *U DBO CF WFSZ SPVHI HVJEB TBNQMF TJ[F JT TNBMM 4UJMM BMXBZT SFNFNCFS UIBU 8"*$ JT BO FTUJNBUF 0ODF ZPV IBWF 8"*$ PS BOZ PUIFS JOGPSNBUJPO DSJUFSJPO DBMDVMBUFE GP ZPV DBO CFHJO CZ PSEFSJOH UIF NPEFMT CZ UIFJS 8"*$ WBMVFT ćF /"1%&+ BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBM ǘ ,*-/"  3 DPEF  ǯ *&)(Ǒ*,!")0 ʆǦ ,*-/"ǯ *ǃǑƾƾ ǒ *ǃǑƾƿ ǒ *ǃǑƾǀ ǒ *ǃǑƾǁ ǰ ǰ   -  !  4"&$%1  ! *ǃǑƾǁ ǦƾǂǑƽ ǁǑDž ƽǑƽ ƽǑdžǀ DŽǑǂǁ  *ǃǑƾƾ ǦDžǑǀ ƾǑDž ǃǑDŽ ƽǑƽǀ ǁǑǂƿ DŽǑƿǃ *ǃǑƾǀ ǦDŽǑdž ǀǑƽ DŽǑƾ ƽǑƽǀ ǂǑǃDŽ ǂǑǀǀ *ǃǑƾƿ ǦǃǑƿ ƿǑdž DžǑdž ƽǑƽƾ ǁǑǀǁ DŽǑǂDŽ ćF GVODUJPO ,*-/" UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPE GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CF PVUPGTBNQMF EFWJBODF TP NPEFM *ǃǑƾǁ JT SBOLFE ĕSTU
21. effective parameters WAIC negative okay! smaller still better “weight” difference from best WAIC standard error & std err of difference ǘ ,*-/"  3 DPEF  ǯ *&)(Ǒ*,!")0 ʆǦ ,*-/"ǯ *ǃǑƾƾ ǒ *ǃǑƾƿ ǒ *ǃǑƾǀ ǒ *ǃǑƾǁ ǰ ǰ   -  !  4"&$%1  ! *ǃǑƾǁ ǦƾǂǑƽ ǁǑDž ƽǑƽ ƽǑdžǀ DŽǑǂǁ  *ǃǑƾƾ ǦDžǑǀ ƾǑDž ǃǑDŽ ƽǑƽǀ ǁǑǂƿ DŽǑƿǃ *ǃǑƾǀ ǦDŽǑdž ǀǑƽ DŽǑƾ ƽǑƽǀ ǂǑǃDŽ ǂǑǀǀ *ǃǑƾƿ ǦǃǑƿ ƿǑdž DžǑdž ƽǑƽƾ ǁǑǀǁ DŽǑǂDŽ ćF GVODUJPO ,*-/" UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPE GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CF PVUPGTBNQMF EFWJBODF TP NPEFM *ǃǑƾǁ JT SBOLFE ĕSTU  -  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJE IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF  !  JT UIF EJČFSFODF CFUXFFO FBDI 8"*$ BOE UIF MPXFTU 8"*$ 4 UJWF EFWJBODF NBUUFST UIJT DPMVNO TIPXT UIF EJČFSFODFT JO SFMBUJWF  4"&$%1 JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF US GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX   JT UIF TUBOEBSE FSSPS PG UIF 8"*$ FTUJNBUF 8"*$ JT BO FTUJNBUF UIF TBNQMF TJ[F / JT MBSHF FOPVHI JUT VODFSUBJOUZ XJMM CF XFMM BQQSP
22. 0ODF ZPV IBWF 8"*$ PS BOZ PUIFS JOGPSNBUJPO DSJUFSJPO DBMDVMBUFE GP ZPV DBO CFHJO CZ PSEFSJOH UIF NPEFMT CZ UIFJS 8"*$ WBMVFT ćF /"1%&+( BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBMM ǘ ,*-/"  3 DPEF  ǯ *&)(Ǒ*,!")0 ʆǦ ,*-/"ǯ *ǃǑƾƾ ǒ *ǃǑƾƿ ǒ *ǃǑƾǀ ǒ *ǃǑƾǁ ǰ ǰ   -  !  4"&$%1  ! *ǃǑƾǁ ǦƾǂǑƽ ǁǑDž ƽǑƽ ƽǑdžǀ DŽǑǂǁ  *ǃǑƾƾ ǦDžǑǀ ƾǑDž ǃǑDŽ ƽǑƽǀ ǁǑǂƿ DŽǑƿǃ *ǃǑƾǀ ǦDŽǑdž ǀǑƽ DŽǑƾ ƽǑƽǀ ǂǑǃDŽ ǂǑǀǀ *ǃǑƾƿ ǦǃǑƿ ƿǑdž DžǑdž ƽǑƽƾ ǁǑǀǁ DŽǑǂDŽ ćF GVODUJPO ,*-/" UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPE GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CF PVUPGTBNQMF EFWJBODF TP NPEFM *ǃǑƾǁ JT SBOLFE ĕSTU  -  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJE IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF  !  JT UIF EJČFSFODF CFUXFFO FBDI 8"*$ BOE UIF MPXFTU 8"*$ 4 UJWF EFWJBODF NBUUFST UIJT DPMVNO TIPXT UIF EJČFSFODFT JO SFMBUJWF G  4"&$%1 JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF US GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX Weights • deviance estimate of relative divergence • convert to probability scale, standardize => “weight” • each weight is estimated probability model is best for prediction • BUT just a central estimate; need to look at std err...
23. 0ODF ZPV IBWF 8"*$ PS BOZ PUIFS JOGPSNBUJPO DSJUFSJPO DBMDVMBUFE GPS ZPV DBO CFHJO CZ PSEFSJOH UIF NPEFMT CZ UIFJS 8"*$ WBMVFT ćF /"1%&+( BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBMMZ ǘ ,*-/"  3 DPEF  ǯ *&)(Ǒ*,!")0 ʆǦ ,*-/"ǯ *ǃǑƾƾ ǒ *ǃǑƾƿ ǒ *ǃǑƾǀ ǒ *ǃǑƾǁ ǰ ǰ   -  !  4"&$%1  ! *ǃǑƾǁ ǦƾǂǑƽ ǁǑDž ƽǑƽ ƽǑdžǀ DŽǑǂǁ  *ǃǑƾƾ ǦDžǑǀ ƾǑDž ǃǑDŽ ƽǑƽǀ ǁǑǂƿ DŽǑƿǃ *ǃǑƾǀ ǦDŽǑdž ǀǑƽ DŽǑƾ ƽǑƽǀ ǂǑǃDŽ ǂǑǀǀ *ǃǑƾƿ ǦǃǑƿ ƿǑdž DžǑdž ƽǑƽƾ ǁǑǀǁ DŽǑǂDŽ ćF GVODUJPO ,*-/" UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPE GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CFU PVUPGTBNQMF EFWJBODF TP NPEFM *ǃǑƾǁ JT SBOLFE ĕSTU  -  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJEF IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF  !  JT UIF EJČFSFODF CFUXFFO FBDI 8"*$ BOE UIF MPXFTU 8"*$ 4J UJWF EFWJBODF NBUUFST UIJT DPMVNO TIPXT UIF EJČFSFODFT JO SFMBUJWF GB  4"&$%1 JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USB GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX Standard errors  2 $"#/ JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USBOTGPSNFE JO GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX   JT UIF TUBOEBSE FSSPS PG UIF 8"*$ FTUJNBUF 8"*$ JT BO FTUJNBUF BOE QSPWJEFE UIF TBNQMF TJ[F / JT MBSHF FOPVHI JUT VODFSUBJOUZ XJMM CF XFMMBQQSPYJNBUFE CZ JUT TUBOEBSE FSSPS 4P UIJT  WBMVF JTOU OFDFTTBSJMZ WFSZ QSFDJTF CVU JU EPFT QSPWJEF B DIFDL BHBJOTU PWFSDPOĕEFODF JO EJČFSFODFT CFUXFFO 8"*$ WBMVFT   JT UIF TUBOEBSE FSSPS PG UIF EJČFSFODF JO 8"*$ CFUXFFO FBDI NPEFM BOE UIF UPQSBOLFE NPEFM 4P JU JT NJTTJOH GPS UIF UPQ NPEFM "OE ZPV DBO QMPU UIFTF WBMVFT UP QSPWJEF B QPTTJCMZ NPSFJOUVJUJWF QSFTFOUBUJPO +'*/ǭ ($'&Ǐ(* '. ǐ ʃ ǐ ʃ Ǯ ćJT JT UIF SFTVMU m6.12 m6.13 m6.11 m6.14 -25 -20 -15 -10 -5 deviance WAIC &BDI SPX JT B NPEFM PSEFSFE CZ 8"*$ ćF ĕMMFE QPJOUT BSF UIF JOTBNQMF EFWJBODF PG FBDI in out
24. 0ODF ZPV IBWF 8"*$ PS BOZ PUIFS JOGPSNBUJPO DSJUFSJPO DBMDVMBUFE GPS ZPV DBO CFHJO CZ PSEFSJOH UIF NPEFMT CZ UIFJS 8"*$ WBMVFT ćF /"1%&+( BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBMMZ ǘ ,*-/"  3 DPEF  ǯ *&)(Ǒ*,!")0 ʆǦ ,*-/"ǯ *ǃǑƾƾ ǒ *ǃǑƾƿ ǒ *ǃǑƾǀ ǒ *ǃǑƾǁ ǰ ǰ   -  !  4"&$%1  ! *ǃǑƾǁ ǦƾǂǑƽ ǁǑDž ƽǑƽ ƽǑdžǀ DŽǑǂǁ  *ǃǑƾƾ ǦDžǑǀ ƾǑDž ǃǑDŽ ƽǑƽǀ ǁǑǂƿ DŽǑƿǃ *ǃǑƾǀ ǦDŽǑdž ǀǑƽ DŽǑƾ ƽǑƽǀ ǂǑǃDŽ ǂǑǀǀ *ǃǑƾƿ ǦǃǑƿ ƿǑdž DžǑdž ƽǑƽƾ ǁǑǀǁ DŽǑǂDŽ ćF GVODUJPO ,*-/" UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPE GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CFU PVUPGTBNQMF EFWJBODF TP NPEFM *ǃǑƾǁ JT SBOLFE ĕSTU  -  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJEF IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF  !  JT UIF EJČFSFODF CFUXFFO FBDI 8"*$ BOE UIF MPXFTU 8"*$ 4J UJWF EFWJBODF NBUUFST UIJT DPMVNO TIPXT UIF EJČFSFODFT JO SFMBUJWF GB  4"&$%1 JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USB GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX Standard errors  2 $"#/ JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USBOTGPSNFE JO GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX   JT UIF TUBOEBSE FSSPS PG UIF 8"*$ FTUJNBUF 8"*$ JT BO FTUJNBUF BOE QSPWJEFE UIF TBNQMF TJ[F / JT MBSHF FOPVHI JUT VODFSUBJOUZ XJMM CF XFMMBQQSPYJNBUFE CZ JUT TUBOEBSE FSSPS 4P UIJT  WBMVF JTOU OFDFTTBSJMZ WFSZ QSFDJTF CVU JU EPFT QSPWJEF B DIFDL BHBJOTU PWFSDPOĕEFODF JO EJČFSFODFT CFUXFFO 8"*$ WBMVFT   JT UIF TUBOEBSE FSSPS PG UIF EJČFSFODF JO 8"*$ CFUXFFO FBDI NPEFM BOE UIF UPQSBOLFE NPEFM 4P JU JT NJTTJOH GPS UIF UPQ NPEFM "OE ZPV DBO QMPU UIFTF WBMVFT UP QSPWJEF B QPTTJCMZ NPSFJOUVJUJWF QSFTFOUBUJPO +'*/ǭ ($'&Ǐ(* '. ǐ ʃ ǐ ʃ Ǯ ćJT JT UIF SFTVMU m6.12 m6.13 m6.11 m6.14 -25 -20 -15 -10 -5 deviance WAIC &BDI SPX JT B NPEFM PSEFSFE CZ 8"*$ ćF ĕMMFE QPJOUT BSF UIF JOTBNQMF EFWJBODF PG FBDI NPEFM XIJDI GPS 8"*$ JT DBMDVMBUFE BT −×MQQE XIJDI JT Q GSPN UIF DPSSFTQPOEJOH
25. 0ODF ZPV IBWF 8"*$ PS BOZ PUIFS JOGPSNBUJPO DSJUFSJPO DBMDVMBUFE GPS ZPV DBO CFHJO CZ PSEFSJOH UIF NPEFMT CZ UIFJS 8"*$ WBMVFT ćF /"1%&+( BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBMMZ ǘ ,*-/"  3 DPEF  ǯ *&)(Ǒ*,!")0 ʆǦ ,*-/"ǯ *ǃǑƾƾ ǒ *ǃǑƾƿ ǒ *ǃǑƾǀ ǒ *ǃǑƾǁ ǰ ǰ   -  !  4"&$%1  ! *ǃǑƾǁ ǦƾǂǑƽ ǁǑDž ƽǑƽ ƽǑdžǀ DŽǑǂǁ  *ǃǑƾƾ ǦDžǑǀ ƾǑDž ǃǑDŽ ƽǑƽǀ ǁǑǂƿ DŽǑƿǃ *ǃǑƾǀ ǦDŽǑdž ǀǑƽ DŽǑƾ ƽǑƽǀ ǂǑǃDŽ ǂǑǀǀ *ǃǑƾƿ ǦǃǑƿ ƿǑdž DžǑdž ƽǑƽƾ ǁǑǀǁ DŽǑǂDŽ ćF GVODUJPO ,*-/" UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPE GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CFU PVUPGTBNQMF EFWJBODF TP NPEFM *ǃǑƾǁ JT SBOLFE ĕSTU  -  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJEF IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF  !  JT UIF EJČFSFODF CFUXFFO FBDI 8"*$ BOE UIF MPXFTU 8"*$ 4J UJWF EFWJBODF NBUUFST UIJT DPMVNO TIPXT UIF EJČFSFODFT JO SFMBUJWF GB  4"&$%1 JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USB GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX Standard errors  2 $"#/ JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USBOTGPSNFE JO GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX   JT UIF TUBOEBSE FSSPS PG UIF 8"*$ FTUJNBUF 8"*$ JT BO FTUJNBUF BOE QSPWJEFE UIF TBNQMF TJ[F / JT MBSHF FOPVHI JUT VODFSUBJOUZ XJMM CF XFMMBQQSPYJNBUFE CZ JUT TUBOEBSE FSSPS 4P UIJT  WBMVF JTOU OFDFTTBSJMZ WFSZ QSFDJTF CVU JU EPFT QSPWJEF B DIFDL BHBJOTU PWFSDPOĕEFODF JO EJČFSFODFT CFUXFFO 8"*$ WBMVFT   JT UIF TUBOEBSE FSSPS PG UIF EJČFSFODF JO 8"*$ CFUXFFO FBDI NPEFM BOE UIF UPQSBOLFE NPEFM 4P JU JT NJTTJOH GPS UIF UPQ NPEFM "OE ZPV DBO QMPU UIFTF WBMVFT UP QSPWJEF B QPTTJCMZ NPSFJOUVJUJWF QSFTFOUBUJPO +'*/ǭ ($'&Ǐ(* '. ǐ ʃ ǐ ʃ Ǯ ćJT JT UIF SFTVMU m6.12 m6.13 m6.11 m6.14 -25 -20 -15 -10 -5 deviance WAIC &BDI SPX JT B NPEFM PSEFSFE CZ 8"*$ ćF ĕMMFE QPJOUT BSF UIF JOTBNQMF EFWJBODF PG FBDI NPEFM XIJDI GPS 8"*$ JT DBMDVMBUFE BT −×MQQE XIJDI JT Q GSPN UIF DPSSFTQPOEJOH
26. WAICA WAICB WAICC WAICD WAICE WAICF
27. Comparing estimates • Always learn more from set of models than any one model • Compare estimates to help understand differences in model performance
28. m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 a log.sigma bn bm -2 -1 0 1 2 3 4 Estimate ' E N & F E C B Figure 6.12 m6.11 m6.12 m6.13 -2 0 2 4 Estimate 3 DPEF  ,"#1ǯ*ǃǑƾƾǒ*ǃǑƾƿǒ*ǃǑƾǀǒ*ǃǑƾǁǰ *ǃǑƾƾ *ǃǑƾƿ *ǃǑƾǀ *ǃǑƾǁ  ƽǑǃǃ ƽǑǀǂ ƽǑDŽƾ ǦƾǑƽdž ),$Ǒ0&$* ǦƾǑDŽdž ǦƾǑDžƽ ǦƾǑDžǂ ǦƿǑƾǃ +  ƽǑǁǂ  ƿǑDŽdž *   ǦƽǑƽǀ ǦƽǑƾƽ +,0 ƾDŽ ƾDŽ ƾDŽ ƾDŽ ćF +,0 BU UIF CPUUPN BSF UIF OVNCFS PG PCTFSWBUJPOT KVTU UIFSF UP IFMQ ZPV NBLF TVSF ZPV ĕU FBDI NPEFM UP UIF TBNF PCTFSWBUJPOT 'SPN TDBOOJOH UIF UBCMF ZPV DBO TFF UIBU UIF FTUJNBUFT GPS CPUI + BOE * HFU GBSUIFS GSPN [FSP XIFO UIFZ BSF CPUI QSFTFOU JO UIF NPEFM #VU TUBOEBSE FSSPST BSFOU SFQSFTFOUFE IFSF BOE TFFJOH IPX UIF VODFSUBJOUZ DIBOHFT JT KVTU BT JNQPSUBOU BT TFFJOH IPX UIF MPDBUJPO DIBOHFT :PV DBO HFU ,"#1 UP BEE TUBOEBSE FSSPST UP UIF UBCMF TFF ǘ ,"#1 CVU UIBU TUJMM EPFTOU NBLF JU FBTZ UP BQQSFDJBUF DIBOHFT JO UIF XJEUI PG QPTUFSJPS EFOTJUJFT #FUUFS UP QMPU UIFTF FTUJNBUFT 3 DPEF  -),1ǯ ,"#1ǯ*ǃǑƾƾǒ*ǃǑƾƿǒ*ǃǑƾǀǒ*ǃǑƾǁǰ ǰ ćF SFTVMU JT TIPXO JO 'ĶĴłĿIJ ƎƉƊ &BDI QPJOU JT B ."1 FTUJNBUF BOE FBDI CMBDL MJOF TFH NFOU JT BO  QFSDFOUJMF JOUFSWBM &BDI HSPVQ PG FTUJNBUFT DPSSFTQPOET UP UIF TBNF OBNFE
29. m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 a log.sigma bn bm -2 -1 0 1 2 3 4 Estimate ' E N & F E C B Figure 6.12 m6.11 m6.12 m6.13 -2 0 2 4 Estimate 3 DPEF  ,"#1ǯ*ǃǑƾƾǒ*ǃǑƾƿǒ*ǃǑƾǀǒ*ǃǑƾǁǰ *ǃǑƾƾ *ǃǑƾƿ *ǃǑƾǀ *ǃǑƾǁ  ƽǑǃǃ ƽǑǀǂ ƽǑDŽƾ ǦƾǑƽdž ),$Ǒ0&$* ǦƾǑDŽdž ǦƾǑDžƽ ǦƾǑDžǂ ǦƿǑƾǃ +  ƽǑǁǂ  ƿǑDŽdž *   ǦƽǑƽǀ ǦƽǑƾƽ +,0 ƾDŽ ƾDŽ ƾDŽ ƾDŽ ćF +,0 BU UIF CPUUPN BSF UIF OVNCFS PG PCTFSWBUJPOT KVTU UIFSF UP IFMQ ZPV NBLF TVSF ZPV ĕU FBDI NPEFM UP UIF TBNF PCTFSWBUJPOT 'SPN TDBOOJOH UIF UBCMF ZPV DBO TFF UIBU UIF FTUJNBUFT GPS CPUI + BOE * HFU GBSUIFS GSPN [FSP XIFO UIFZ BSF CPUI QSFTFOU JO UIF NPEFM #VU TUBOEBSE FSSPST BSFOU SFQSFTFOUFE IFSF BOE TFFJOH IPX UIF VODFSUBJOUZ DIBOHFT JT KVTU BT JNQPSUBOU BT TFFJOH IPX UIF MPDBUJPO DIBOHFT :PV DBO HFU ,"#1 UP BEE TUBOEBSE FSSPST UP UIF UBCMF TFF ǘ ,"#1 CVU UIBU TUJMM EPFTOU NBLF JU FBTZ UP BQQSFDJBUF DIBOHFT JO UIF XJEUI PG QPTUFSJPS EFOTJUJFT #FUUFS UP QMPU UIFTF FTUJNBUFT 3 DPEF  -),1ǯ ,"#1ǯ*ǃǑƾƾǒ*ǃǑƾƿǒ*ǃǑƾǀǒ*ǃǑƾǁǰ ǰ ćF SFTVMU JT TIPXO JO 'ĶĴłĿIJ ƎƉƊ &BDI QPJOU JT B ."1 FTUJNBUF BOE FBDI CMBDL MJOF TFH NFOU JT BO  QFSDFOUJMF JOUFSWBM &BDI HSPVQ PG FTUJNBUFT DPSSFTQPOET UP UIF TBNF OBNFE
30. Standardized predictors help m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 bn bm -0.2 0.0 0.2 Estimate plot( coeftab(m6.11,m6.12,m6.13,m6.14),pars=c("bn","bm") ) Still better to contrast predictions, not estimates
31. Model averaging • When computing predictions, average over posterior • For more than one model, can average the averages • Do not average parameter estimates, just predictions • Because parameters in different models live in different small worlds => don’t mean same thing, even if named same thing • But predictions reference common large world
32. Model averaging • Model averaging procedure • Compute information weight for each model • Compute distribution of predictions for each model • Mix predictions using model weights • Result is one kind of prediction ensemble • Such ensembles can outperform single-model predictions
33. 0.55 0.60 0.65 0.70 0.75 0.5 0.6 0.7 0.8 neocortex kcal.per.g 'ĶĴłĿIJ ƎƉƋ .PEFM BWFSBHFE QPTUFSJPS QSF EJDUJWF EJTUSJCVUJPO GPS UIF QSJNBUF NJML BOBM ZTJT ćF EBTIFE SFHSFTTJPO MJOF BOE EBTIFE  QFSDFOUJMF JOUFSWBM DPSSFTQPOE UP UIF NJOJNVN8"*$ NPEFM *ǃǑƾǁ ćF TPMJE MJOF BOE TIBEFE  QFSDFOUJMF SFHJPO DPSSF TQPOE UP UIF NPEFM BWFSBHFE QSFEJDUJPOT "OE UIJT JT XIBU UIF GVODUJPO "+0"*)" DBO EP ćF "+0"*)" GVODUJPO XPSLT B MPU MJLF )&+( BOE 0&* *O GBDU JU KVTU DBMMT UIPTF GVODUJPOT GPS FBDI NPEFM ZPV HJWF JU BOE UIFO DPNCJOFT UIF SFTVMUT BDDPSEJOH UP "LBJLF XFJHIUT 4P UP CVJME BO FOTFNCMF BDDPSEJOH UP 8"*$ XFJHIU UIF EFGBVMU CFIBWJPS  3 DPEF  *&)(Ǒ"+0"*)" ʆǦ "+0"*)"ǯ *ǃǑƾƾ ǒ *ǃǑƾƿ ǒ *ǃǑƾǀ ǒ *ǃǑƾǁ ǒ !1ʅ!Ǒ-/"!& 1 ǰ *2 ʆǦ --)6ǯ *&)(Ǒ"+0"*)"ɢ)&+( ǒ ƿ ǒ *"+ ǰ *2Ǒ ʆǦ --)6ǯ *&)(Ǒ"+0"*)"ɢ)&+( ǒ ƿ ǒ  ǰ )&+"0ǯ + Ǒ0". ǒ *2 ǰ 0%!"ǯ *2Ǒ ǒ + Ǒ0". ǰ ćF TPMJE SFHSFTTJPO MJOF BOE TIBEFE SFHJPO JO 'ĶĴłĿIJ ƎƉƋ EJTQMBZ UIFTF DBMDVMBUJPOT ćF SFHSFTTJPO MJOF XIJDI TIPXT UIF BWFSBHF µ BU FBDI WBMVF IPSJ[POUBM BYJT IBT IBSEMZ NPWFE 0.55 0.60 0.65 0.70 0.75 0.5 0.6 0.7 0.8 0.9 neocortex kcal.per.g ' U ć D 8  F ćF SFTVMUJOH QMPU JT EJTQMBZFE JO 'ĶĴłĿIJ BOE GPDVT PO UIF EBTIFE SFHSFTTJPO MJOF BO ćPTF BSF UIF MJOFT UIF DPEF BCPWF QSPEVD /PX MFUT DPNQVUF BOE BEE NPEFM BWF DPNQVUF JT BO IJĻŀIJĺįĹIJ PG QPTUFSJPS QS UIFO *MM TIPX ZPV UIF DPEF UIBU BVUPNBUF FBDI TBNQMF JO UIF QPTUFSJPS Figure 6.13 GPDVTJOH PO DPVOUFSGBDUVBM QSFEJDUJPOT BDSPTT UIF SBOHF PG ) **-/ 3 3 DPEF  ȃ *(+0/ *0)/ -!/0' +- $/$*). ȃ ) **-/ 3 !-*( ƻǏǀ /* ƻǏǃ )Ǐ. , ʄǤ . ,ǭ!-*(ʃƻǏǀǐ/*ʃƻǏǃǐ' )"/#Ǐ*0/ʃƾƻǮ Ǐ+- $/ ʄǤ '$./ǭ &'Ǐ+ -Ǐ" ʃ - +ǭƻǐƾƻǮǐ ȃ (+/4 *0/*( ) **-/ 3 ʃ )Ǐ. ,ǐ ȃ . ,0 ) *! ) **-/ 3 (.. ʃ - +ǭƿǏǀǐƾƻǮ ȃ 1 -" (.. Ǯ +- Ǐ(ǁǏƼƿ ʄǤ '$)&ǭ (ǁǏƼƿ ǐ /ʃǏ+- $/ Ǯ (0 ʄǤ ++'4ǭ +- Ǐ(ǁǏƼƿ ǐ ƽ ǐ ( ) Ǯ (0Ǐ ʄǤ ++'4ǭ +- Ǐ(ǁǏƼƿ ǐ ƽ ǐ  Ǯ ȃ +'*/ $/ '' +'*/ǭ &'Ǐ+ -Ǐ" ʋ ) **-/ 3 ǐ  ǐ *'ʃ-)"$ƽ Ǯ '$) .ǭ )Ǐ. , ǐ (0 ǐ '/4ʃƽ Ǯ '$) .ǭ )Ǐ. , ǐ (0Ǐ ǯƼǐǰ ǐ '/4ʃƽ Ǯ '$) .ǭ )Ǐ. , ǐ (0Ǐ ǯƽǐǰ ǐ '/4ʃƽ Ǯ top model only weighted ensemble
35. Curse of Tippecanoe • 1840–1960: Every US president elected in year ending in digit “0” died in office • W. H. Harrison first, “Old Tippecanoe” • Lincoln, Garfield, McKinley, Harding, FD Roosevelt • J. F. Kennedy last, assassinated in 1963 • Reagan broke the curse! • Trying all possible models: A formula for overfitting • Be thoughtful • Model averaging mitigates the curse • Admit data exploration
36. Complexity can be good • Good reasons to use more complex models than AIC/DIC/WAIC recommend • Theory says predictor important, so estimate it • Lots of sources of variation, but *IC not focused right • Simpler model better may mean only that estimate should be smaller => average • Consistency critique has blunt teeth • Sometimes noted: As N –> infinity, *IC favors most complex model • But as N –> infinity, estimates infinitely precise • In hierarchical models, no coherent way N –> infinity?
37. On the horizon • Homework: 6H1, 6H2, 6H3 • Next week: Interactions, practicing model comparison • Week 6: Markov chain Monte Carlo, Maximum entropy, and generalized linear models