Statistical Rethinking - Lecture 08

Lecture 08 - Model comparison (2) - Statistical Rethinking: A Bayesian Course with R Examples

Transcript

1. Week 4: Model Comparison Richard McElreath Statistical Rethinking
2. Regularization • Use informative, conservative priors to reduce overfitting => model learns less from sample • But if too informative, model learns too little • Such priors are regularizing 1 0 1 2 3 rameter value /PSNBM(, ) ćJO TPMJE /PSNBM(, .) ćJDL TPMJE /PSNBM(, .) T SFBMMZ POF PG UVOJOH #VU BT ZPVMM TFF FWFO NJME TLFQUJDJTN DBO IFMQ B BOE EPJOH CFUUFS JT BMM XF DBO SFBMMZ IPQF GPS JO UIF MBSHF XPSME XIFSF OP JT PQUJNBM DPOTJEFS UIJT (BVTTJBO NPEFM ZJ ∼ /PSNBM(µJ, σ) µJ = α + βYJ α ∼ /PSNBM(, ) β ∼ /PSNBM(, ) σ ∼ 6OJGPSN(, ) E QSBDUJDF UIBU UIF QSFEJDUPS Y JT TUBOEBSEJ[FE TP UIBU JUT TUBOEBSE EFWJBUJPO JT [FSP ćFO UIF QSJPS PO α JT B OFBSMZĘBU QSJPS UIBU IBT OP QSBDUJDBM FČFDU   07&3'*55*/( 3&(6-"3*;"5*0/ -3 -2 -1 0 1 2 3 0.0 0.5 1.0 1.5 2.0 parameter value Density 'ĶĴłĿIJ TUSPOH TUBOEBS ĕUUJOH /PSNB TPMJE / regularizing prior N(0,1) N(0,0.5) N(0,0.2)
3. Regularization  3&(6-"3*;"5*0/  1 2 3 4 5 48 50 52 54 56 58 60 number of parameters deviance N = 20 N(0,1) N(0,0.5) N(0,0.2) 1 2 3 4 5 260 265 270 275 280 285 number of parameters deviance N = 100 'ĶĴłĿIJ ƎƑ 3FHVMBSJ[JOH QSJPST BOE PVUPGTBNQMF EFWJBODF ćF QPJOUT JO in sample out of sample in sample out of sample
4. Information criteria • Can we estimate out-of-sample deviance? • Well, in theory: “Information criteria” • Information, because use of deviance based on information theoretic analysis • Criteria, because used to compare models • Information criteria estimate deviance out of sample • AIC, DIC, WAIC, many others
5. Akaike information criterion • A meta-model of forecasting: • Two samples: training and testing, size N • Fit model to training sample, get Dtrain • Use fit to training to compute Dtest • Difference Dtest – Dtrain is overfitting • Under some strict conditions: Hirotugu Akaike (1927–2009) NBUFT GSPN TUFQ  UP DPNQVUF UIF EFWJBODF GPS UIF EBUB JO UIF UFTU QMF $BMM UIJT EFWJBODF %UFTU  NQVUF UIF EJČFSFODF %UFTU − %USBJO  ćJT EJČFSFODF XJMM VTVBMMZ CF UJWF CFDBVTF UIF NPEFM XJMM UFOE UP QFSGPSN XPSTF IBWF B IJHIFS BODF JO UFTUJOH UIBO JO USBJOJOH MMZ JNBHJOF SFQFBUJOH UIJT QSPDFEVSF NBOZ UJNFT ćF BWFSBHF EJG ODF UIFO UFMMT VT UIF FYQFDUFE PWFSĕUUJOH IPX NVDI UIF USBJOJOH EF DF VOEFSFTUJNBUFT UIF EJWFSHFODF PG UIF NPEFM WF MPHJD B HBNCJU CFDBVTF JU DBOOPU QSPWJEF HVBSBOUFFT #VU JU DBO CMF BEWJDF *U UVSOT PVU UIBU UIJT HBNCJU MFBET UP BO BTUPOJTIJOHMZ B GPS UIF FYQFDUFE UFTUTBNQMF EFWJBODF "*$ = %USBJO + L ≈ & %UFTU OVNCFS PG QBSBNFUFST JO UIF NPEFM ćF UFSN L JT PęFO DBMMFE UIF *U JT B NFBTVSF PG FYQFDUFE PWFSĕUUJOH U EFQFOET VQPO XFBL QSJPST B (BVTTJBO QPTUFSJPS EJTUSJCVUJPO BOE SBNFUFST L NVDI MFTT UIBO UIF OVNCFS PG DBTFT / 4P JUT BQQSP OBSZ MJOFBS SFHSFTTJPO BOE JU FWFO XPSLT RVJUF XFMM GPS NBOZ OPO k is parameter count [ah–ka–ee–kay]
6. Akaike information criterion • Conditions: • You like the AIC forecasting model • Flat priors • No varying/mixed/random effects • Gaussian posterior distribution • k << N; as k approaches N: WJBODF VOEFSFTUJNBUFT UIF EJWFSHFODF PG UIF NPEFM BMM UIF BCPWF MPHJD B HBNCJU CFDBVTF JU DBOOPU QSPWJEF HVBSBOUFFT #VU JU D PWJEF WBMVBCMF BEWJDF *U UVSOT PVU UIBU UIJT HBNCJU MFBET UP BO BTUPOJTIJO NQMF GPSNVMB GPS UIF FYQFDUFE UFTUTBNQMF EFWJBODF "*$ = %USBJO + L ≈ & %UFTU IFSF L JT UIF OVNCFS PG QBSBNFUFST JO UIF NPEFM ćF UFSN L JT PęFO DBMMFE U OBMUZ UFSN *U JT B NFBTVSF PG FYQFDUFE PWFSĕUUJOH ćJT SFTVMU EFQFOET VQPO XFBL QSJPST B (BVTTJBO QPTUFSJPS EJTUSJCVUJPO B NCFS PG QBSBNFUFST L NVDI MFTT UIBO UIF OVNCFS PG DBTFT / 4P JUT BQQ BUF GPS PSEJOBSZ MJOFBS SFHSFTTJPO BOE JU FWFO XPSLT RVJUF XFMM GPS NBOZ OP VTTJBO SFHSFTTJPOT HFOFSBMJ[FE MJOFBS NPEFMT (-.T UIBU XFMM FYBNJOF MB UIJT CPPL UIJOLJOH "*$ BOE iUSVFw NPEFMT *U JT QPTTJCMF UP SFBE CPUI UIBU  "*$ BTTVN EBUB HFOFSBUJOH NPEFM JT POF PG UIF DBOEJEBUF NPEFMT BOE  "*$ EPFT OPU BTTV EBUB HFOFSBUJOH NPEFM JT B DBOEJEBUF ćJT DPOGVTJPO BSJTFT CFDBVTF UIFSF BSF NVMUJ ZT UP EFSJWF "*$ ćF HBNCJU EFTDSJCFE BCPWF EPFT OPU FNQMPZ B iUSVFw NPEFM FYDFQ BU MFBTU JO UIF MBSHF XPSME 4UJMM DBVUJPO SFRVJSFT UBLJOH OPUF PG WJPMBUFE BTTVNQUJPOT BOE IPQFGVMMZ FWBMVBUJOH UIF DPOTFRVFODFT PG UIFTF WJPMBUJPOT  -JNJUT UP "*$T HFOFSBMJUZ #VU NPSF HFOFSBMMZ "*$ JT OPU HFOFSBM *U JT B TQFDJBM DBTF PG NVDI MBSHFS QIFOPNFOPO UIF TFWFSJUZ PG PWFSĕUUJOH JT BO JO DSFBTJOH GVODUJPO PG UIF OVNCFS PG QBSBNFUFST #VU UIJT GVODUJPO JT OPU BMXBZT BT TJNQMF BT L BT JU JT JO "*$ " GFX DPNNPO DPOEJUJPOT CFOFĕU GSPN B NPSF HFOFSBM TPMVUJPO  1BSBNFUFS DPVOU DMPTF UP TBNQMF TJ[F 4VQQPTF B NPEFM IBT L QBSBNF UFST BOE JT ĕU UP / PCTFSWBUJPOT 8IFO L JT DMPTF UP / PWFSĕUUJOH SJTFT WFSZ SBQJEMZ ćJT IBQQFOT CFDBVTF UIF NPEFM TUBSUT QFSGFDUMZ FODPEJOH UIF USBJOJOH TBNQMF 4P XIFO UIF NPEFM TFFT UIF UFTU TBNQMF JUT BMXBZT WFSZ TVSQSJTFE " DPOTFSWBUJWF BQQSPYJNBUJPO GPS UIJT SJTF JO PWFSĕUUJOH JT HJWFO CZ B DPNNPO HFOFSBMJ[BUJPO PG "*$ "*$D = %USBJO + L  − (L + )// 8IFO / JT WFSZ NVDI MBSHFS UIBO L UIF BCPWF TJNQMJĕFT UP QMBJO "*$ #VU BT L BQQSPBDIFT / −  UIF QFOBMUZ PO UIF SJHIU BQQSPBDIFT JOĕOJUZ 4P BOZUJNF "*$ JT BQQSPQSJBUF "*$D NBZ CF B CFUUFS DIPJDF
7. Akaike information criterion • Prediction/forecasting task matters • Suppose we care about accumulated error over learning, aka prequential error • Consider the humble wurst • Grill-only or boil-then-grill? • Want to consume each wurst • How to learn and eat well at same time? • AIC not the right scenario
9. Figure 6.10 training testing AIC “true” model  */'03."5*0/ $3*5&3*" 1 2 3 4 5 48 50 52 54 56 58 60 number of parameters deviance N = 20 2 4.1 5.3 7.6 9.7 1 2 3 4 5 260 265 270 275 280 285 number of parameters deviance N = 100 1.9 4.1 4.9 7.1 8.5 'ĶĴłĿIJ ƎƉƈ %FWJBODF JO CMVF BOE PVU CMBDL PG TBNQMF VTJOH ĘBU QSJ PST ćF WFSUJDBM TFHNFOUT NFBTVSF UIF EJTUBODF CFUXFFO FBDI QBJS PG EF WJBODFT 'PS CPUI / =  BOE / =  UIJT EJTUBODF JT BQQSPYJNBUFMZ UXJDF UIF OVNCFS PG QBSBNFUFST ćF EBTIFE MJOFT TIPX FYBDUMZ UIF EFWJBODF JO
10. Deviance information criterion • DIC, a generalized AIC: • Doesn’t require flat priors • Does require reasonably Gaussian posterior • Does require effective parameter count << N • Computed from posterior samples • DIC function in rethinking David J. Spiegelhalter (1953–)
11. Deviance information criterion • D-bar: average deviance, averaged over posterior • D-hat: deviance of average, computed at average of posterior • pD : “effective” number of parameters, measure of flexibility of model  ćF EFWJBODF PG UIF BWFSBHF PG UIF QBSBNFUFST JNQMJFT ĕOEJOH UIF N PG UIF QBSBNFUFST BOE UIFO DPNQVUJOH B TJOHMF EFWJBODF WBMVF GPS QPTUFSJPS NFBO ćBUT UIF EFWJBODF PG UIF BWFSBHF PG UIF QBSBNFUFS O %*$ JT EFĕOFE BT %*$ = ˆ % + (¯ % − ˆ %) ≈ & %UFTU OHMJTI UIF PWFSĕUUJOH QFOBMUZ JT UXJDF UIF EJČFSFODF CFUXFFO UIF BWFSBHF DF BOE UIF EFWJBODF PG UIF BWFSBHF PG UIF QBSBNFUFST ćF EJČFSFODF ¯ % MMFE UIF IJijijIJİŁĶŃIJ ĻłĺįIJĿ ļij ĽĮĿĮĺIJŁIJĿŀ PęFO MBCFMFE Q% *U UBLFT F PG UIF QBSBNFUFS DPVOU L JO "*$T GPSNVMB UIJOLJOH $PNQVUJOH %*$ VTJOH QPTUFSJPS WBSJBODF "O BMUFSOBUJWF GPSNVM JT %*$ = ˆ % + WBS(%) F WBS(%) JT UIF WBSJBODF PG UIF QPTUFSJPS EJTUSJCVUJPO PG UIF EFWJBODF ćJT JN WBS(%)/ JT BO FTUJNBUF PG UIF FČFDUJWF OVNCFS PG QBSBNFUFST Q%  ćJT GPSNV UJNFT NPSF BDDVSBUF PS NPSF DPOWFOJFOU UIBO UIF PSJHJOBM EJČFSFODF GPSNVMB . 4P JG XF ESBX UIPVTBOE TBNQMFT GSPN UIF QPTUFSJPS XF DPNQVUF  WBMVFT ćF BWFSBHF PS FYQFDUBUJPO PG % JT ¯ % = & % "MTP EFĕOF ˆ % BT UI BU UIF QPTUFSJPS NFBO ćJT NFBOT XF DPNQVUF UIF BWFSBHF PG FBDI QBS JPS EJTUSJCVUJPO ćFO XF QMVH UIPTF BWFSBHFT JOUP UIF EFWJBODF GPSNV PTU DBTFT 3 JT HPJOH UP EP BMM PG UIJT GPS ZPV ZPV IBWF ¯ % BOE ˆ % %*$ JT DBMDVMBUFE BT %*$ = ¯ % + (¯ % − ˆ %) = ¯ % + Q% FODF ¯ % − ˆ % = Q% JT BOBMPHPVT UP UIF OVNCFS PG QBSBNFUFST VTFE JO D BO iFČFDUJWFw OVNCFS PG QBSBNFUFST UIBU NFBTVSFT IPX ĘFYJCMF UIF N USBJOJOH TBNQMF .PSF ĘFYJCMF NPEFMT FOUBJM HSFBUFS SJTL PG PWFSĕUUJOH NFUJNFT DBMMFE B QFOBMUZ UFSN *U JT KVTU UIF FYQFDUFE EJTUBODF CFUXFFO UI BOE UIF EFWJBODF PVUPGTBNQMF *O UIF DBTF PG ĘBU QSJPST %*$ SFEVDFT VTF UIF FYQFDUFE EJTUBODF JT KVTU UIF OVNCFS PG QBSBNFUFST #VU NPSF HF NF GSBDUJPO PG UIF OVNCFS PG QBSBNFUFST CFDBVTF SFHVMBSJ[JOH QSJPST D FYJCJMJUZ VODUJPO   JO UIF - /#$)&$)" QBDLBHF XJMM DPNQVUF %*$ GPS B NPEFM ĕ
12. Effective parameters 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 prior standard deviation pD (effective parameters) m <- map( alist( y ~ dnorm(mu,1), mu ~ dnorm(0,s) ) , data=list(y=y,s=1) ) DIC(m)
13. Widely Applicable IC • Widely Applicable Information Criterion (WAIC) • Sumio Watanabe 2010 • Sometimes called “Watanabe-Akaike Information Criterion” • Does not assume Gaussian posterior • WAIC function in rethinking J= = / J= MPH & θ 1S(ZJ|θ) MQQE = / J= MPH  4 4 T= 1S(ZJ|θT) Q8"*$ = / J= WBS θ MPH 1S(ZJ|θ) 8"*$ = −MQQE + Q8"*$ "JK ∼ #JOPNJBM(OJ, QJK) MPHJU QJK = α + αK + (βN + βNK)NJK α ∼ /PSNBM(, ) βN ∼ /PSNBM(, )
14. WAIC • Based on Bayesian deviance: –2(lppd) • log pointwise predictive density is log probability of data, averaged over posterior distribution: • Does all calculations pointwise • For each separable piece of data yi : • (1) compute Pr(yi |theta) for each sample of theta • (2) average the likelihoods • Sum all the log average likelihoods • 1000 observations and 5000 samples => 5 million likelihoods = TVN PG MPHT PG BWFSBHF MJLFMJIPPET MQQE = / J= MPH 1S(ZJ|θ) 1S(θ)Eθ = / J= MPH & θ 1S(ZJ|θ) MQQE = / J= MPH 1S(ZJ|θ) 1S(θ)Eθ = / J= MPH & θ 1S(ZJ|θ) MQQE = / J= MPH  4 4 T= 1S(ZJ|θT) Q8"*$ = / J= WBS θ MPH 1S(ZJ|θ) 8"*$ = −MQQE + Q8"*$
15. WAIC • Just like deviance, lppd biased by overfitting to sample • Bias correction again effective number of parameters: • (1) Compute log Pr(yi |theta) for each sample of theta • (2) Compute variance of these log-likelihoods • (3) Do this for every i and sum the variances = J= MPH & θ 1S(ZJ|θ) MQQE = / J= MPH 1S(ZJ|θ) 1S(θ)Eθ = / J= MPH & θ 1S(ZJ|θ) MQQE = / J= MPH  4 4 T= 1S(ZJ|θT) Q8"*$ = / J= WBS θ MPH 1S(ZJ|θ) 8"*$ = −MQQE + Q8"*$ "JK ∼ #JOPNJBM(OJ, QJK) MPHJU QJK = α + αK + (βN + βNK)NJK
16. WAIC better than DIC • Very often DIC and WAIC agree • When mean isn’t good summary of posterior, DIC can go squirrelly • pD < 0 • Mixture models routinely frustrate DIC • In any event, don’t mix criteria; use WAIC for all models or rather DIC for all models • Drawback: WAIC requires separating data • Time series: Is entire trend for each unit an observation? • Spatial/network models: All outcomes joint?
17. 1 2 3 4 5 55 56 57 58 59 number of parameters deviance N = 20 DIC N(0,100) 1 2 3 4 5 55 56 57 58 59 number of parameters deviance N = 20 WAIC N(0,0.5) 'ĶĴłĿIJ ƎƉƉ 0VUPGTBNQMF EFWJB NBUFE CZ %*$ BOE 8"*$ 1PJOUT BSF PGTBNQMF EFWJBODF PWFS UIPVTBOE ćF MJOFT BSF BWFSBHF %*$ UPQ BOE UPN DPNQVUFE GSPN UIF TBNF TJN CMBDL QPJOUT BOE MJOFT DPNF GSPN TJN B OFBSMZĘBU /PSNBM(, ) QSJPS ć BOE MJOFT VTFE B SFHVMBSJ[JOH /PSNBM Figure 6.11 Only out-of-sample deviance WAIC slightly more accurate Both DIC & WAIC good, conditional on forecasting model being correct
18. At the beach, finally • Underfitting possible; overfitting inevitable • Regularizing priors reduce it • Information criteria measure it • Taste great together  64*/( */'03."5*0/ $3*5&3*" 1 2 3 4 5 55 56 57 58 59 number of parameters deviance N = 20 DIC N(0,100) 1 2 3 4 5 55 56 57 58 59 number of parameters deviance N = 20 WAIC N(0,0.5) 'ĶĴłĿIJ ƎƉƉ 0VUPGTBNQMF EFWJBODF BT NBUFE CZ %*$ BOE 8"*$ 1PJOUT BSF BWFSBH PGTBNQMF EFWJBODF PWFS UIPVTBOE TJNVMB ćF MJOFT BSF BWFSBHF %*$ UPQ BOE 8"*$ UPN DPNQVUFE GSPN UIF TBNF TJNVMBUJPOT CMBDL QPJOUT BOE MJOFT DPNF GSPN TJNVMBUJPO B OFBSMZĘBU /PSNBM(, ) QSJPS ćF CMVF BOE MJOFT VTFE B SFHVMBSJ[JOH /PSNBM(, .)
19. Using AIC/DIC/WAIC • Avoid model selection • Model comparison: quantify uncertainty about models, in addition to uncertainty about parameters • Model averaging: Simulate predictions, averaging over uncertainty about models • don’t average parameters, but only predictions
20. Primate milk again kcal.per.g -2 0 2 4 0.5 0.7 0.9 -2 0 2 4 log(mass) 0.5 0.7 0.9 55 65 75 55 65 75 neocortex.perc • .ļıIJĹ İļĺĽĮĿĶŀļĻ NFBOT VTJOH %*$8"*$ JO DPNCJOBUJPO XJUI UIF FTUJN BOE QPTUFSJPS QSFEJDUJWF DIFDLT GSPN FBDI NPEFM *U JT KVTU BT JNQPSUBOU UP V TUBOE XIZ B NPEFM PVUQFSGPSNT BOPUIFS BT JU JT UP NFBTVSF UIF QFSGPSNBODF E FODF %*$8"*$ BMPOF TBZT WFSZ MJUUMF BCPVU TVDI EFUBJMT #VU JO DPNCJOBUJPO PUIFS JOGPSNBUJPO %*$8"*$ JT B CJH IFMQ • .ļıIJĹ ĮŃIJĿĮĴĶĻĴ NFBOT VTJOH %*$8"*$ UP DPOTUSVDU B QPTUFSJPS QSFE EJTUSJCVUJPO UIBU FYQMPJUT XIBU XF LOPX BCPVU SFMBUJWF BDDVSBDZ PG UIF NPEFMT IFMQT HVBSE BHBJOTU PWFSDPOĕEFODF JO NPEFM TUSVDUVSF JO UIF TBNF XBZ UIBU UIF FOUJSF QPTUFSJPS EJTUSJCVUJPO IFMQT HVBSE BHBJOTU PWFSDPOĕEFODF JO QBSBN WBMVFT 8IBU NPEFM BWFSBHJOH EPFT OPU NFBO JT BWFSBHJOH QBSBNFUFS FTUJN CFDBVTF QBSBNFUFST JO EJČFSFOU NPEFMT IBWF EJČFSFOU NFBOJOHT BOE TIPVME O BWFSBHFE VOMFTT ZPV BSF TVSF ZPV BSF JO B TQFDJBM DBTF JO XIJDI JU JT TBGF UP EP ćF TFDUJPO EFNPOTUSBUFT IPX UP DPOEVDU DPNQBSJTPO BOE BWFSBHJOH VTJOH B TJNQMF FYB XJUI B GFX QSFEJDUPS WBSJBCMFT -BUFS DIBQUFST DPOUJOVF VTJOH UIFTF UPPMT BOE UIF EFUB FYBNQMFT EP WBSZ 4P CF XBSZ OPU UP PWFSHFOFSBMJ[F UIF FYBNQMF UIBU GPMMPXT  .PEFM DPNQBSJTPO 3FDBMM UIF QSJNBUF NJML EBUB GSPN UIF QSFWJPVT DIBQUFS MPBE JU JOUP 3 SFNPWF UIF T BOE SFTDBMF POF PG UIF FYQMBOBUPSZ WBSJBCMFT 3 DPEF  /ǭ($'&Ǯ  ʄǤ ($'&ǯ *(+' / Ǐ. .ǭ($'&Ǯ ǐ ǰ ɠ) **-/ 3 ʄǤ ɠ) **-/ 3Ǐ+ - dz Ƽƻƻ $(ǭǮ ǯƼǰ Ƽǂ DŽ 4P ZPVS EBUB GSBNF TIPVME BMTP IBWF  SPXT DBTFT BOE  DPMVNOT WBSJBCMFT 
21. MPPLT MJLF JOTJEF UIF DPEF 3 DPEF  Ǐ./-/ ʄǤ ( )ǭɠ&'Ǐ+ -Ǐ"Ǯ .$"(Ǐ./-/ ʄǤ '*"ǭ.ǭɠ&'Ǐ+ -Ǐ"ǮǮ (ǁǏƼƼ ʄǤ (+ǭ '$./ǭ &'Ǐ+ -Ǐ" ʋ )*-(ǭ  ǐ 3+ǭ'*"Ǐ.$"(Ǯ Ǯ Ǯ ǐ /ʃ ǐ ./-/ʃ'$./ǭʃǏ./-/ǐ'*"Ǐ.$"(ʃ.$"(Ǐ./-/Ǯ Ǯ (ǁǏƼƽ ʄǤ (+ǭ '$./ǭ &'Ǐ+ -Ǐ" ʋ )*-(ǭ (0 ǐ 3+ǭ'*"Ǐ.$"(Ǯ Ǯ ǐ (0 ʄǤ  ɾ )Ƿ) **-/ 3 Ǯ ǐ /ʃ ǐ ./-/ʃ'$./ǭʃǏ./-/ǐ)ʃƻǐ'*"Ǐ.$"(ʃ.$"(Ǐ./-/Ǯ Ǯ (ǁǏƼƾ ʄǤ (+ǭ '$./ǭ &'Ǐ+ -Ǐ" ʋ )*-(ǭ (0 ǐ 3+ǭ'*"Ǐ.$"(Ǯ Ǯ ǐ (0 ʄǤ  ɾ (Ƿ'*"ǭ(..Ǯ Ǯ ǐ /ʃ ǐ ./-/ʃ'$./ǭʃǏ./-/ǐ(ʃƻǐ'*"Ǐ.$"(ʃ.$"(Ǐ./-/Ǯ Ǯ (ǁǏƼƿ ʄǤ (+ǭ '$./ǭ &'Ǐ+ -Ǐ" ʋ )*-(ǭ (0 ǐ 3+ǭ'*"Ǐ.$"(Ǯ Ǯ ǐ (0 ʄǤ  ɾ )Ƿ) **-/ 3 ɾ (Ƿ'*"ǭ(..Ǯ Ǯ ǐ /ʃ ǐ ./-/ʃ'$./ǭʃǏ./-/ǐ)ʃƻǐ(ʃƻǐ'*"Ǐ.$"(ʃ.$"(Ǐ./-/Ǯ Ǯ ćF QSJPST BSF BMM ĘBU BCPWF XIJDI JT DMFBSMZ OPU UIF CFTU JEFB #VU UIJT XJMM MFU ZPV HFU B TFOTF
22. Comparing • What is expected out-of-sample deviance for each model? • Can also compute DIC. See ?compare ćF ĕSTU WBMVF SFQPSUFE JT UIF 8"*$ WBMVF /PUF UIBU JU JT OFHBUJWF JO UIJT DBTF ćBUT ĕ ćFSFT OPUIJOH QSFWFOUJOH EFWJBODF GSPN CFJOH OFHBUJWF 4NBMMFS WBMVFT BSF TUJMM CFUUFS TFDPOE WBMVF SFQPSUFE JT UIF MQQE ćF UIJSE WBMVF JT Q8"*$  *G ZPV TVCUSBDU +  GSPN ' BOE UIFO NVMUJQMZ UIBU EJČFSFODF CZ − ZPVMM HFU UIF 8"*$ WBMVF ćF ĕOBM WBMVF . UIF TUBOEBSE FSSPS PG UIF 8"*$ WBMVF ćJT TUBOEBSE FSSPS QSPWJEFT SPVHI HVJEBODF UP VODFSUBJOUZ JO 8"*$ UIBU BSJTFT GSPN TBNQMJOH *U DBO CF WFSZ SPVHI HVJEBODF XIFO TBNQMF TJ[F JT TNBMM 4UJMM BMXBZT SFNFNCFS UIBU 8"*$ JT BO FTUJNBUF 0ODF ZPV IBWF 8"*$ PS BOZ PUIFS JOGPSNBUJPO DSJUFSJPO DBMDVMBUFE GPS FBDI NP ZPV DBO CFHJO CZ PSEFSJOH UIF NPEFMT CZ UIFJS 8"*$ WBMVFT ćF - /#$)&$)" QBDL BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBMMZ CZ %*$ ǖ*(+-  3 DPEF  ǭ ($'&Ǐ(* '. ʄǤ *(+- ǭ (ǁǏƼƼ ǐ (ǁǏƼƽ ǐ (ǁǏƼƾ ǐ (ǁǏƼƿ Ǯ Ǯ   +    2 $"#/   (ǁǏƼƿ ǤƼǀǏƻ ƿǏǃ ƻǏƻ ƻǏDŽƾ ǂǏǀƿ  (ǁǏƼƼ ǤǃǏƾ ƼǏǃ ǁǏǂ ƻǏƻƾ ƿǏǀƽ ǂǏƽǁ (ǁǏƼƾ ǤǂǏDŽ ƾǏƻ ǂǏƼ ƻǏƻƾ ǀǏǁǂ ǀǏƾƾ (ǁǏƼƽ ǤǁǏƽ ƽǏDŽ ǃǏDŽ ƻǏƻƼ ƿǏƾƿ ǂǏǀǂ ćF GVODUJPO *(+- UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPEFMT BSF SBO GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CFUUFS FTUJNB PVUPGTBNQMF EFWJBODF TP NPEFM (ǁǏƼƿ JT SBOLFE ĕSTU  +  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJEFT B DMVF B IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF
23. effective parameters WAIC negative okay! smaller still better “weight” difference from best WAIC BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBMMZ CZ %*$ ǖ*(+-  3 DPEF  ǭ ($'&Ǐ(* '. ʄǤ *(+- ǭ (ǁǏƼƼ ǐ (ǁǏƼƽ ǐ (ǁǏƼƾ ǐ (ǁǏƼƿ Ǯ Ǯ   +    2 $"#/   (ǁǏƼƿ ǤƼǀǏƻ ƿǏǃ ƻǏƻ ƻǏDŽƾ ǂǏǀƿ  (ǁǏƼƼ ǤǃǏƾ ƼǏǃ ǁǏǂ ƻǏƻƾ ƿǏǀƽ ǂǏƽǁ (ǁǏƼƾ ǤǂǏDŽ ƾǏƻ ǂǏƼ ƻǏƻƾ ǀǏǁǂ ǀǏƾƾ (ǁǏƼƽ ǤǁǏƽ ƽǏDŽ ǃǏDŽ ƻǏƻƼ ƿǏƾƿ ǂǏǀǂ ćF GVODUJPO *(+- UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPEFMT BSF SBO GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CFUUFS FTUJNB PVUPGTBNQMF EFWJBODF TP NPEFM (ǁǏƼƿ JT SBOLFE ĕSTU  +  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJEFT B DMVF B IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF    JT UIF EJČFSFODF CFUXFFO FBDI 8"*$ BOE UIF MPXFTU 8"*$ 4JODF POMZ BUJWF EFWJBODF NBUUFST UIJT DPMVNO TIPXT UIF EJČFSFODFT JO SFMBUJWF GBTIJPO standard error & std err of difference
24. Weights • deviance is estimate of relative divergence; on log scale • convert to probability scale, standardize => “weight” • each weight is estimated probability model is best for prediction • BUT just a central estimate; need to look at std err... TBNQMF TJ[F JT TNBMM 4UJMM BMXBZT SFNFNCFS UIBU 8"*$ JT BO FTUJNBUF 0ODF ZPV IBWF 8"*$ PS BOZ PUIFS JOGPSNBUJPO DSJUFSJPO DBMDVMBUFE GPS FBDI NP ZPV DBO CFHJO CZ PSEFSJOH UIF NPEFMT CZ UIFJS 8"*$ WBMVFT ćF - /#$)&$)" QBDL BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBMMZ CZ %*$ ǖ*(+-  3 DPEF  ǭ ($'&Ǐ(* '. ʄǤ *(+- ǭ (ǁǏƼƼ ǐ (ǁǏƼƽ ǐ (ǁǏƼƾ ǐ (ǁǏƼƿ Ǯ Ǯ   +    2 $"#/   (ǁǏƼƿ ǤƼǀǏƻ ƿǏǃ ƻǏƻ ƻǏDŽƾ ǂǏǀƿ  (ǁǏƼƼ ǤǃǏƾ ƼǏǃ ǁǏǂ ƻǏƻƾ ƿǏǀƽ ǂǏƽǁ (ǁǏƼƾ ǤǂǏDŽ ƾǏƻ ǂǏƼ ƻǏƻƾ ǀǏǁǂ ǀǏƾƾ (ǁǏƼƽ ǤǁǏƽ ƽǏDŽ ǃǏDŽ ƻǏƻƼ ƿǏƾƿ ǂǏǀǂ ćF GVODUJPO *(+- UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPEFMT BSF SBO GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CFUUFS FTUJNB PVUPGTBNQMF EFWJBODF TP NPEFM (ǁǏƼƿ JT SBOLFE ĕSTU  +  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJEFT B DMVF B IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF    JT UIF EJČFSFODF CFUXFFO FBDI 8"*$ BOE UIF MPXFTU 8"*$ 4JODF POMZ BUJWF EFWJBODF NBUUFST UIJT DPMVNO TIPXT UIF EJČFSFODFT JO SFMBUJWF GBTIJPO
25. Standard errors TBNQMF TJ[F JT TNBMM 4UJMM BMXBZT SFNFNCFS UIBU 8"*$ JT BO FTUJNBUF 0ODF ZPV IBWF 8"*$ PS BOZ PUIFS JOGPSNBUJPO DSJUFSJPO DBMDVMBUFE GPS FBDI NP ZPV DBO CFHJO CZ PSEFSJOH UIF NPEFMT CZ UIFJS 8"*$ WBMVFT ćF - /#$)&$)" QBDL BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBMMZ CZ %*$ ǖ*(+-  3 DPEF  ǭ ($'&Ǐ(* '. ʄǤ *(+- ǭ (ǁǏƼƼ ǐ (ǁǏƼƽ ǐ (ǁǏƼƾ ǐ (ǁǏƼƿ Ǯ Ǯ   +    2 $"#/   (ǁǏƼƿ ǤƼǀǏƻ ƿǏǃ ƻǏƻ ƻǏDŽƾ ǂǏǀƿ  (ǁǏƼƼ ǤǃǏƾ ƼǏǃ ǁǏǂ ƻǏƻƾ ƿǏǀƽ ǂǏƽǁ (ǁǏƼƾ ǤǂǏDŽ ƾǏƻ ǂǏƼ ƻǏƻƾ ǀǏǁǂ ǀǏƾƾ (ǁǏƼƽ ǤǁǏƽ ƽǏDŽ ǃǏDŽ ƻǏƻƼ ƿǏƾƿ ǂǏǀǂ ćF GVODUJPO *(+- UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPEFMT BSF SBO GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CFUUFS FTUJNB PVUPGTBNQMF EFWJBODF TP NPEFM (ǁǏƼƿ JT SBOLFE ĕSTU  +  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJEFT B DMVF B IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF    JT UIF EJČFSFODF CFUXFFO FBDI 8"*$ BOE UIF MPXFTU 8"*$ 4JODF POMZ BUJWF EFWJBODF NBUUFST UIJT DPMVNO TIPXT UIF EJČFSFODFT JO SFMBUJWF GBTIJPO  64*/( */'03."5*0/ $3*5&3*"   2 $"#/ JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USBOTGPSNFE JO GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX   JT UIF TUBOEBSE FSSPS PG UIF 8"*$ FTUJNBUF 8"*$ JT BO FTUJNBUF BOE QSPWJEFE UIF TBNQMF TJ[F / JT MBSHF FOPVHI JUT VODFSUBJOUZ XJMM CF XFMMBQQSPYJNBUFE CZ JUT TUBOEBSE FSSPS 4P UIJT  WBMVF JTOU OFDFTTBSJMZ WFSZ QSFDJTF CVU JU EPFT QSPWJEF B DIFDL BHBJOTU PWFSDPOĕEFODF JO EJČFSFODFT CFUXFFO 8"*$ WBMVFT   JT UIF TUBOEBSE FSSPS PG UIF EJČFSFODF JO 8"*$ CFUXFFO FBDI NPEFM BOE UIF UPQSBOLFE NPEFM 4P JU JT NJTTJOH GPS UIF UPQ NPEFM "OE ZPV DBO QMPU UIFTF WBMVFT UP QSPWJEF B QPTTJCMZ NPSFJOUVJUJWF QSFTFOUBUJPO 3 DPEF  +'*/ǭ ($'&Ǐ(* '. ǐ ʃ ǐ ʃ Ǯ ćJT JT UIF SFTVMU m6.12 m6.13 m6.11 m6.14 -25 -20 -15 -10 -5 deviance WAIC  2 $"#/ JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USBOTGPSNFE JO GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX   JT UIF TUBOEBSE FSSPS PG UIF 8"*$ FTUJNBUF 8"*$ JT BO FTUJNBUF BOE QSPWJEFE UIF TBNQMF TJ[F / JT MBSHF FOPVHI JUT VODFSUBJOUZ XJMM CF XFMMBQQSPYJNBUFE CZ JUT TUBOEBSE FSSPS 4P UIJT  WBMVF JTOU OFDFTTBSJMZ WFSZ QSFDJTF CVU JU EPFT QSPWJEF B DIFDL BHBJOTU PWFSDPOĕEFODF JO EJČFSFODFT CFUXFFO 8"*$ WBMVFT   JT UIF TUBOEBSE FSSPS PG UIF EJČFSFODF JO 8"*$ CFUXFFO FBDI NPEFM BOE UIF UPQSBOLFE NPEFM 4P JU JT NJTTJOH GPS UIF UPQ NPEFM "OE ZPV DBO QMPU UIFTF WBMVFT UP QSPWJEF B QPTTJCMZ NPSFJOUVJUJWF QSFTFOUBUJPO +'*/ǭ ($'&Ǐ(* '. ǐ ʃ ǐ ʃ Ǯ ćJT JT UIF SFTVMU m6.12 m6.13 m6.11 m6.14 -25 -20 -15 -10 -5 deviance WAIC in out
26. Standard errors TBNQMF TJ[F JT TNBMM 4UJMM BMXBZT SFNFNCFS UIBU 8"*$ JT BO FTUJNBUF 0ODF ZPV IBWF 8"*$ PS BOZ PUIFS JOGPSNBUJPO DSJUFSJPO DBMDVMBUFE GPS FBDI NP ZPV DBO CFHJO CZ PSEFSJOH UIF NPEFMT CZ UIFJS 8"*$ WBMVFT ćF - /#$)&$)" QBDL BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBMMZ CZ %*$ ǖ*(+-  3 DPEF  ǭ ($'&Ǐ(* '. ʄǤ *(+- ǭ (ǁǏƼƼ ǐ (ǁǏƼƽ ǐ (ǁǏƼƾ ǐ (ǁǏƼƿ Ǯ Ǯ   +    2 $"#/   (ǁǏƼƿ ǤƼǀǏƻ ƿǏǃ ƻǏƻ ƻǏDŽƾ ǂǏǀƿ  (ǁǏƼƼ ǤǃǏƾ ƼǏǃ ǁǏǂ ƻǏƻƾ ƿǏǀƽ ǂǏƽǁ (ǁǏƼƾ ǤǂǏDŽ ƾǏƻ ǂǏƼ ƻǏƻƾ ǀǏǁǂ ǀǏƾƾ (ǁǏƼƽ ǤǁǏƽ ƽǏDŽ ǃǏDŽ ƻǏƻƼ ƿǏƾƿ ǂǏǀǂ ćF GVODUJPO *(+- UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPEFMT BSF SBO GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CFUUFS FTUJNB PVUPGTBNQMF EFWJBODF TP NPEFM (ǁǏƼƿ JT SBOLFE ĕSTU  +  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJEFT B DMVF B IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF    JT UIF EJČFSFODF CFUXFFO FBDI 8"*$ BOE UIF MPXFTU 8"*$ 4JODF POMZ BUJWF EFWJBODF NBUUFST UIJT DPMVNO TIPXT UIF EJČFSFODFT JO SFMBUJWF GBTIJPO  64*/( */'03."5*0/ $3*5&3*"   2 $"#/ JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USBOTGPSNFE JO GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX   JT UIF TUBOEBSE FSSPS PG UIF 8"*$ FTUJNBUF 8"*$ JT BO FTUJNBUF BOE QSPWJEFE UIF TBNQMF TJ[F / JT MBSHF FOPVHI JUT VODFSUBJOUZ XJMM CF XFMMBQQSPYJNBUFE CZ JUT TUBOEBSE FSSPS 4P UIJT  WBMVF JTOU OFDFTTBSJMZ WFSZ QSFDJTF CVU JU EPFT QSPWJEF B DIFDL BHBJOTU PWFSDPOĕEFODF JO EJČFSFODFT CFUXFFO 8"*$ WBMVFT   JT UIF TUBOEBSE FSSPS PG UIF EJČFSFODF JO 8"*$ CFUXFFO FBDI NPEFM BOE UIF UPQSBOLFE NPEFM 4P JU JT NJTTJOH GPS UIF UPQ NPEFM "OE ZPV DBO QMPU UIFTF WBMVFT UP QSPWJEF B QPTTJCMZ NPSFJOUVJUJWF QSFTFOUBUJPO 3 DPEF  +'*/ǭ ($'&Ǐ(* '. ǐ ʃ ǐ ʃ Ǯ ćJT JT UIF SFTVMU m6.12 m6.13 m6.11 m6.14 -25 -20 -15 -10 -5 deviance WAIC  2 $"#/ JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USBOTGPSNFE JO GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX   JT UIF TUBOEBSE FSSPS PG UIF 8"*$ FTUJNBUF 8"*$ JT BO FTUJNBUF BOE QSPWJEFE UIF TBNQMF TJ[F / JT MBSHF FOPVHI JUT VODFSUBJOUZ XJMM CF XFMMBQQSPYJNBUFE CZ JUT TUBOEBSE FSSPS 4P UIJT  WBMVF JTOU OFDFTTBSJMZ WFSZ QSFDJTF CVU JU EPFT QSPWJEF B DIFDL BHBJOTU PWFSDPOĕEFODF JO EJČFSFODFT CFUXFFO 8"*$ WBMVFT   JT UIF TUBOEBSE FSSPS PG UIF EJČFSFODF JO 8"*$ CFUXFFO FBDI NPEFM BOE UIF UPQSBOLFE NPEFM 4P JU JT NJTTJOH GPS UIF UPQ NPEFM "OE ZPV DBO QMPU UIFTF WBMVFT UP QSPWJEF B QPTTJCMZ NPSFJOUVJUJWF QSFTFOUBUJPO +'*/ǭ ($'&Ǐ(* '. ǐ ʃ ǐ ʃ Ǯ ćJT JT UIF SFTVMU m6.12 m6.13 m6.11 m6.14 -25 -20 -15 -10 -5 deviance WAIC
27. Standard errors TBNQMF TJ[F JT TNBMM 4UJMM BMXBZT SFNFNCFS UIBU 8"*$ JT BO FTUJNBUF 0ODF ZPV IBWF 8"*$ PS BOZ PUIFS JOGPSNBUJPO DSJUFSJPO DBMDVMBUFE GPS FBDI NP ZPV DBO CFHJO CZ PSEFSJOH UIF NPEFMT CZ UIFJS 8"*$ WBMVFT ćF - /#$)&$)" QBDL BMTP QSPWJEFT B IBOEZ GVODUJPO GPS SBOLJOH NPEFMT CZ 8"*$ BOE PQUJPOBMMZ CZ %*$ ǖ*(+-  3 DPEF  ǭ ($'&Ǐ(* '. ʄǤ *(+- ǭ (ǁǏƼƼ ǐ (ǁǏƼƽ ǐ (ǁǏƼƾ ǐ (ǁǏƼƿ Ǯ Ǯ   +    2 $"#/   (ǁǏƼƿ ǤƼǀǏƻ ƿǏǃ ƻǏƻ ƻǏDŽƾ ǂǏǀƿ  (ǁǏƼƼ ǤǃǏƾ ƼǏǃ ǁǏǂ ƻǏƻƾ ƿǏǀƽ ǂǏƽǁ (ǁǏƼƾ ǤǂǏDŽ ƾǏƻ ǂǏƼ ƻǏƻƾ ǀǏǁǂ ǀǏƾƾ (ǁǏƼƽ ǤǁǏƽ ƽǏDŽ ǃǏDŽ ƻǏƻƼ ƿǏƾƿ ǂǏǀǂ ćF GVODUJPO *(+- UBLFT ĕU NPEFMT BT JOQVU *U SFUVSOT B UBCMF JO XIJDI NPEFMT BSF SBO GSPN CFTU UP XPSTU XJUI TJY DPMVNOT PG JOGPSNBUJPO    JT PCWJPVTMZ 8"*$ GPS FBDI NPEFM 4NBMMFS 8"*$ JOEJDBUFT CFUUFS FTUJNB PVUPGTBNQMF EFWJBODF TP NPEFM (ǁǏƼƿ JT SBOLFE ĕSTU  +  JT UIF FTUJNBUFE FČFDUJWF OVNCFS PG QBSBNFUFST ćJT QSPWJEFT B DMVF B IPX ĘFYJCMF FBDI NPEFM JT JO ĕUUJOH UIF TBNQMF    JT UIF EJČFSFODF CFUXFFO FBDI 8"*$ BOE UIF MPXFTU 8"*$ 4JODF POMZ BUJWF EFWJBODF NBUUFST UIJT DPMVNO TIPXT UIF EJČFSFODFT JO SFMBUJWF GBTIJPO  64*/( */'03."5*0/ $3*5&3*"   2 $"#/ JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USBOTGPSNFE JO GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX   JT UIF TUBOEBSE FSSPS PG UIF 8"*$ FTUJNBUF 8"*$ JT BO FTUJNBUF BOE QSPWJEFE UIF TBNQMF TJ[F / JT MBSHF FOPVHI JUT VODFSUBJOUZ XJMM CF XFMMBQQSPYJNBUFE CZ JUT TUBOEBSE FSSPS 4P UIJT  WBMVF JTOU OFDFTTBSJMZ WFSZ QSFDJTF CVU JU EPFT QSPWJEF B DIFDL BHBJOTU PWFSDPOĕEFODF JO EJČFSFODFT CFUXFFO 8"*$ WBMVFT   JT UIF TUBOEBSE FSSPS PG UIF EJČFSFODF JO 8"*$ CFUXFFO FBDI NPEFM BOE UIF UPQSBOLFE NPEFM 4P JU JT NJTTJOH GPS UIF UPQ NPEFM "OE ZPV DBO QMPU UIFTF WBMVFT UP QSPWJEF B QPTTJCMZ NPSFJOUVJUJWF QSFTFOUBUJPO 3 DPEF  +'*/ǭ ($'&Ǐ(* '. ǐ ʃ ǐ ʃ Ǯ ćJT JT UIF SFTVMU m6.12 m6.13 m6.11 m6.14 -25 -20 -15 -10 -5 deviance WAIC  2 $"#/ JT UIF "ĸĮĶĸIJ ńIJĶĴĵŁ GPS FBDI NPEFM ćFTF WBMVFT BSF USBOTGPSNFE JO GPSNBUJPO DSJUFSJPO WBMVFT *MM FYQMBJO UIFN CFMPX   JT UIF TUBOEBSE FSSPS PG UIF 8"*$ FTUJNBUF 8"*$ JT BO FTUJNBUF BOE QSPWJEFE UIF TBNQMF TJ[F / JT MBSHF FOPVHI JUT VODFSUBJOUZ XJMM CF XFMMBQQSPYJNBUFE CZ JUT TUBOEBSE FSSPS 4P UIJT  WBMVF JTOU OFDFTTBSJMZ WFSZ QSFDJTF CVU JU EPFT QSPWJEF B DIFDL BHBJOTU PWFSDPOĕEFODF JO EJČFSFODFT CFUXFFO 8"*$ WBMVFT   JT UIF TUBOEBSE FSSPS PG UIF EJČFSFODF JO 8"*$ CFUXFFO FBDI NPEFM BOE UIF UPQSBOLFE NPEFM 4P JU JT NJTTJOH GPS UIF UPQ NPEFM "OE ZPV DBO QMPU UIFTF WBMVFT UP QSPWJEF B QPTTJCMZ NPSFJOUVJUJWF QSFTFOUBUJPO +'*/ǭ ($'&Ǐ(* '. ǐ ʃ ǐ ʃ Ǯ ćJT JT UIF SFTVMU m6.12 m6.13 m6.11 m6.14 -25 -20 -15 -10 -5 deviance WAIC
28. WAICA WAICB WAICC WAICD WAICE WAICF
29. Comparing estimates • Always learn more from set of models than any one model • Compare estimates to help understand differences in model performance
30. m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 a log.sigma bn bm -2 -1 0 1 2 3 4 Estimate ' E N & F E C B Figure 6.12 *O UIF QSJNBUF NJML FYBNQMF DPNQBSJOH FTUJNBUFT DPOĕSNT XIBU ZPV BMSFBEZ MFBSOFE JO UIF QSFWJPVT DIBQUFS UIF NPEFM XJUI CPUI QSFEJDUPST EPFT NVDI CFUUFS CFDBVTF FBDI QSF EJDUPS NBTLT UIF PUIFS *O PSEFS UP EFNPOTUSBUF UIBU JO UIF QSFWJPVT DIBQUFS XF BDUVBMMZ EJE ĕU UISFF PG NPEFMT DVSSFOUMZ BU IBOE -PPLJOH BU B DPOTPMJEBUFE UBCMF PG UIF ."1 FTUJNBUFT NBLFT UIF DPNQBSJTPO B MPU FBTJFS ćF * !/ GVODUJPO UBLFT B TFSJFT PG ĕU NPEFMT BT JOQVU BOE CVJMET TVDI B UBCMF 3 DPEF  * !/ǭ(ǁǏƼƼǐ(ǁǏƼƽǐ(ǁǏƼƾǐ(ǁǏƼƿǮ (ǁǏƼƼ (ǁǏƼƽ (ǁǏƼƾ (ǁǏƼƿ  ƻǏǁǁ ƻǏƾǀ ƻǏǂƼ ǤƼǏƻDŽ '*"Ǐ.$"( ǤƼǏǂDŽ ǤƼǏǃƻ ǤƼǏǃǀ ǤƽǏƼǁ )  ƻǏƿǀ  ƽǏǂDŽ (   ǤƻǏƻƾ ǤƻǏƼƻ )*. Ƽǂ Ƽǂ Ƽǂ Ƽǂ ćF )*. BU UIF CPUUPN BSF UIF OVNCFS PG PCTFSWBUJPOT KVTU UIFSF UP IFMQ ZPV NBLF TVSF ZPV ĕU FBDI NPEFM UP UIF TBNF PCTFSWBUJPOT 'SPN TDBOOJOH UIF UBCMF ZPV DBO TFF UIBU UIF FTUJNBUFT GPS CPUI ) BOE ( HFU GBSUIFS GSPN [FSP XIFO UIFZ BSF CPUI QSFTFOU JO UIF NPEFM #VU TUBOEBSE FSSPST BSFOU SFQSFTFOUFE IFSF BOE TFFJOH IPX UIF VODFSUBJOUZ DIBOHFT JT KVTU BT JNQPSUBOU BT TFFJOH IPX UIF MPDBUJPO DIBOHFT :PV DBO HFU * !/ UP BEE TUBOEBSE FSSPST UP UIF UBCMF TFF ǖ* !/ CVU UIBU TUJMM EPFTOU NBLF JU FBTZ UP BQQSFDJBUF DIBOHFT JO UIF XJEUI PG QPTUFSJPS EFOTJUJFT #FUUFS UP QMPU UIFTF FTUJNBUFT 3 DPEF  +'*/ǭ * !/ǭ(ǁǏƼƼǐ(ǁǏƼƽǐ(ǁǏƼƾǐ(ǁǏƼƿǮ Ǯ ćF SFTVMU JT TIPXO JO 'ĶĴłĿIJ ƎƉƊ &BDI QPJOU JT B ."1 FTUJNBUF BOE FBDI CMBDL MJOF TFH NFOU JT B  QFSDFOUJMF JOUFSWBM &BDI HSPVQ PG FTUJNBUFT DPSSFTQPOE UP UIF TBNF OBNFE QBSBNFUFS BDSPTT NPEFMT &BDI SPX JO FBDI HSPVQ JT B NPEFM MBCFMFE PO UIF MFę /PX ZPV DBO RVJDLMZ TDBO FBDI HSPVQ PG FTUJNBUFT UP TFF IPX FTUJNBUFT DIBOHF PS OPU BDSPTT NPEFMT BOE CVJMET TVDI B UBCMF 3 DPEF  * !/ǭ(ǁǏƼƼǐ(ǁǏƼƽǐ(ǁǏƼƾǐ(ǁǏƼƿǮ (ǁǏƼƼ (ǁǏƼƽ (ǁǏƼƾ (ǁǏƼƿ  ƻǏǁǁ ƻǏƾǀ ƻǏǂƼ ǤƼǏƻDŽ '*"Ǐ.$"( ǤƼǏǂDŽ ǤƼǏǃƻ ǤƼǏǃǀ ǤƽǏƼǁ )  ƻǏƿǀ  ƽǏǂDŽ (   ǤƻǏƻƾ ǤƻǏƼƻ )*. Ƽǂ Ƽǂ Ƽǂ Ƽǂ ćF )*. BU UIF CPUUPN BSF UIF OVNCFS PG PCTFSWBUJPOT KVTU UIFSF UP IFMQ ZPV NBLF TVSF ZPV ĕU FBDI NPEFM UP UIF TBNF PCTFSWBUJPOT 'SPN TDBOOJOH UIF UBCMF ZPV DBO TFF UIBU UIF FTUJNBUFT GPS CPUI ) BOE ( HFU GBSUIFS GSPN [FSP XIFO UIFZ BSF CPUI QSFTFOU JO UIF NPEFM #VU TUBOEBSE FSSPST BSFOU SFQSFTFOUFE IFSF BOE TFFJOH IPX UIF VODFSUBJOUZ DIBOHFT JT KVTU BT JNQPSUBOU BT TFFJOH IPX UIF MPDBUJPO DIBOHFT :PV DBO HFU * !/ UP BEE TUBOEBSE FSSPST UP UIF UBCMF TFF ǖ* !/ CVU UIBU TUJMM EPFTOU NBLF JU FBTZ UP BQQSFDJBUF DIBOHFT JO UIF XJEUI PG QPTUFSJPS EFOTJUJFT #FUUFS UP QMPU UIFTF FTUJNBUFT 3 DPEF  +'*/ǭ * !/ǭ(ǁǏƼƼǐ(ǁǏƼƽǐ(ǁǏƼƾǐ(ǁǏƼƿǮ Ǯ ćF SFTVMU JT TIPXO JO 'ĶĴłĿIJ ƎƉƊ &BDI QPJOU JT B ."1 FTUJNBUF BOE FBDI CMBDL MJOF TFH NFOU JT B  QFSDFOUJMF JOUFSWBM &BDI HSPVQ PG FTUJNBUFT DPSSFTQPOE UP UIF TBNF OBNFE QBSBNFUFS BDSPTT NPEFMT &BDI SPX JO FBDI HSPVQ JT B NPEFM MBCFMFE PO UIF MFę /PX ZPV DBO RVJDLMZ TDBO FBDI HSPVQ PG FTUJNBUFT UP TFF IPX FTUJNBUFT DIBOHF PS OPU BDSPTT NPEFMT :PV DBO BEKVTU UIFTF QMPUT UP HSPVQ CZ NPEFM JOTUFBE PG CZ QBSBNFUFS BOE UP EJTQMBZ POMZ TPNF QBSBNFUFST 4FF ǖ* !/Ǐ+'*/ GPS EFUBJMT 3FUIJOLJOH #BSQMPUT TVDL ćF QMPU JO 'ĶĴłĿIJ ƎƉƊ JT DBMMFE B ıļŁİĵĮĿŁ *U JT B SFQMBDFNFOU GPS B UZQJDBM įĮĿĽĹļŁ ćF POMZ QSPCMFN XJUI CBSQMPUT JT UIBU UIFZ IBWF CBST ćF CBST DBSSZ POMZ B MJUUMF
31. Standardized predictors help m6.11 m6.12 m6.13 m6.14 m6.11 m6.12 m6.13 m6.14 bn bm -0.2 0.0 0.2 Estimate plot( coeftab(m6.11,m6.12,m6.13,m6.14),pars=c("bn","bm") ) Still better to contrast predictions, not estimates
32. Model averaging • When computing predictions, average over posterior • For more than one model, can average the averages • Do not average parameter estimates, just predictions • Because parameters in different models live in different small worlds => don’t mean same thing, even if named same thing • But predictions reference common large world
33. Model averaging • Model averaging procedure • Compute information weight for each model • Compute distribution of predictions for each model • Mix predictions using model weights • Result is one kind of prediction ensemble • Such ensembles can outperform single-model predictions
34. 0.55 0.60 0.65 0.70 0.75 0.5 0.6 0.7 0.8 0.9 neocortex kcal.per.g ' U ć D 8  F ćF SFTVMUJOH QMPU JT EJTQMBZFE JO 'ĶĴłĿIJ BOE GPDVT PO UIF EBTIFE SFHSFTTJPO MJOF BO ćPTF BSF UIF MJOFT UIF DPEF BCPWF QSPEVD /PX MFUT DPNQVUF BOE BEE NPEFM BWF DPNQVUF JT BO IJĻŀIJĺįĹIJ PG QPTUFSJPS QS UIFO *MM TIPX ZPV UIF DPEF UIBU BVUPNBUF FBDI TBNQMF JO UIF QPTUFSJPS Figure 6.13 GPDVTJOH PO DPVOUFSGBDUVBM QSFEJDUJPOT BDSPTT UIF SBOHF PG ) **-/ 3 3 DPEF  ȃ *(+0/ *0)/ -!/0' +- $/$*). ȃ ) **-/ 3 !-*( ƻǏǀ /* ƻǏǃ )Ǐ. , ʄǤ . ,ǭ!-*(ʃƻǏǀǐ/*ʃƻǏǃǐ' )"/#Ǐ*0/ʃƾƻǮ Ǐ+- $/ ʄǤ '$./ǭ &'Ǐ+ -Ǐ" ʃ - +ǭƻǐƾƻǮǐ ȃ (+/4 *0/*( ) **-/ 3 ʃ )Ǐ. ,ǐ ȃ . ,0 ) *! ) **-/ 3 (.. ʃ - +ǭƿǏǀǐƾƻǮ ȃ 1 -" (.. Ǯ +- Ǐ(ǁǏƼƿ ʄǤ '$)&ǭ (ǁǏƼƿ ǐ /ʃǏ+- $/ Ǯ (0 ʄǤ ++'4ǭ +- Ǐ(ǁǏƼƿ ǐ ƽ ǐ ( ) Ǯ (0Ǐ ʄǤ ++'4ǭ +- Ǐ(ǁǏƼƿ ǐ ƽ ǐ  Ǯ ȃ +'*/ $/ '' +'*/ǭ &'Ǐ+ -Ǐ" ʋ ) **-/ 3 ǐ  ǐ *'ʃ-)"$ƽ Ǯ '$) .ǭ )Ǐ. , ǐ (0 ǐ '/4ʃƽ Ǯ '$) .ǭ )Ǐ. , ǐ (0Ǐ ǯƼǐǰ ǐ '/4ʃƽ Ǯ '$) .ǭ )Ǐ. , ǐ (0Ǐ ǯƽǐǰ ǐ '/4ʃƽ Ǯ ćF SFTVMUJOH QMPU JT EJTQMBZFE JO 'ĶĴłĿIJ ƎƉƋ 'PS UIF NPNFOU JHOPSF UIF TIBEFE SFHJPO BOE GPDVT PO UIF EBTIFE SFHSFTTJPO MJOF BOE UIF EBTIFE  QFSDFOUJMF JOUFSWBM PG UIF NFBO ćPTF BSF UIF MJOFT UIF DPEF BCPWF QSPEVDFT :PVWF TFFO UIFN CFGPSF QBHF   /PX MFUT DPNQVUF BOE BEE NPEFM BWFSBHFE QPTUFSJPS QSFEJDUJPOT 8IBU XFSF HPJOH UP DPNQVUF JT BO IJĻŀIJĺįĹIJ PG QPTUFSJPS QSFEJDUJPOT )FSFT UIF DPODFQUVBM QSPDFEVSF BOE UIFO *MM TIPX ZPV UIF DPEF UIBU BVUPNBUFT JU NVDI MJLF '$)& BVUPNBUFT DPNQVUJOH µ GPS FBDI TBNQMF JO UIF QPTUFSJPS  $PNQVUF 8"*$ PS BOPUIFS JOGPSNBUJPO DSJUFSJPO GPS FBDI NPEFM  $PNQVUF UIF XFJHIU GPS FBDI NPEFM  $PNQVUF MJOFBS NPEFM BOE TJNVMBUFE PVUDPNFT GPS FBDI NPEFM  $PNCJOF UIFTF WBMVFT JOUP BO FOTFNCMF PG QSFEJDUJPOT VTJOH UIF NPEFM XFJHIUT BT QSPQPSUJPOT "OE UIJT JT XIBU UIF GVODUJPO ). (' DBO EP ćF ). (' GVODUJPO XPSLT B MPU MJLF '$)& BOE .$( *O GBDU JU KVTU DBMMT UIPTF GVODUJPOT GPS FBDI NPEFM ZPV HJWF JU BOE UIFO DPNCJOFT UIF SFTVMUT BDDPSEJOH UP "LBJLF XFJHIUT 4P UP CVJME BO FOTFNCMF BDDPSEJOH UP 8"*$ XFJHIU UIF EFGBVMU CFIBWJPS  3 DPEF  ($'&Ǐ ). (' ʄǤ ). (' ǭ (ǁǏƼƼ ǐ (ǁǏƼƽ ǐ (ǁǏƼƾ ǐ (ǁǏƼƿ ǐ /ʃǏ+- $/ Ǯ (0 ʄǤ ++'4ǭ ($'&Ǐ ). (' ɠ'$)& ǐ ƽ ǐ ( ) Ǯ (0Ǐ ʄǤ ++'4ǭ ($'&Ǐ ). (' ɠ'$)& ǐ ƽ ǐ  Ǯ '$) .ǭ )Ǐ. , ǐ (0 Ǯ .# ǭ (0Ǐ ǐ )Ǐ. , Ǯ ćF TPMJE SFHSFTTJPO MJOF BOE TIBEFE SFHJPO JO 'ĶĴłĿIJ ƎƉƋ EJTQMBZ UIFTF DBMDVMBUJPOT ćF SFHSFTTJPO MJOF XIJDI TIPXT UIF BWFSBHF µ BU FBDI WBMVF IPSJ[POUBM BYJT IBT IBSEMZ NPWFE BU BMM .PEFM (ǁǏƼƿ IBT BCPVU  PG UIF XFJHIU SFDBMM 4P UIJT NBLFT TFOTF top model only weighted ensemble
36. Curse of Tippecanoe • 1840–1960: Every US president elected in year ending in digit “0” died in office • W. H. Harrison first, “Old Tippecanoe” • Lincoln, Garfield, McKinley, Harding, FD Roosevelt • J. F. Kennedy last, assassinated in 1963 • Reagan broke the curse! • Trying all possible models: A formula for overfitting • Be thoughtful • Model averaging mitigates the curse • Admit data exploration
37. Complexity can be good • Good reasons to use more complex models than AIC/DIC/WAIC recommend • Theory says predictor important, so estimate it • If you have a theory-motivated model, you want to know what data says about it • Good reasons to use flat priors • If regularizing >> flat, then why ever use flat? • Lots of sensible answers, but my favorite: • Flat prior lets you study the likelihood, and often that’s the most important thing
38. On the horizon • Next week: Interactions, practicing model comparison • Week 6: MCMC • Week 7: Maximum entropy and the generalized linear model