L15 Statistical Rethinking Winter 2019

Lecture 15 of the Dec 2018 through March 2019 edition of Statistical Rethinking. Covers Chapter 13, intro to multilevel modeling.

Transcript

1. Models With Memory Statistical Rethinking Winter 2019 Lecture 15 / Week 8
2. Anterograde amnesia • Musicologist and conductor Clive Wearing • Lost parts of prefrontal and hippocampus • Can still play piano • Can’t remember what happened 1 min ago
3. Anterograde amnesia • “Fixed effects” models have anterograde amnesia • Every new cluster (individual, pond, road, classroom) is a new world • No information passed among clusters • Multilevel models remember and pool information • Properties of clusters come from a “population” • Inferred population defines pooling • If previous clusters improve your guess about a new cluster, you want to use pooling
4. Learning, forward and back
5. Depends upon variation
6. Cause & Reconciliation • Fighting a battle on two fronts: • (1) Causal inference — don’t make causal salad • (2) Functional inference — estimation not trivial • No unique solutions, but lots of good options
7. have shown that losses loom larger than the equivalent gains, a phenomenon known as loss aversion (19). Thus, changes in the de- fault may result in a change of choice. out default options for individuals’ deci- sions to become organ donors. Actual deci- sions about organ donation may be affected by governmental educational programs, the 4.25 27.5 17.17 12 99.98 98 99.91 99.97 99.5 99.64 85.9 0 10 20 30 40 50 60 70 80 90 100 Denmark Netherlands Effective consent percentage United Kingdom Germany Austria Belgium France Hungary Poland Portugal Sweden Effective consent rates, by country. Explicit consent (opt-in, gold) and presumed consent (opt- out, blue). EMBER 2003 VOL 302 SCIENCE www.sciencemag.org opt-in opt-out organ donation consent percentage
8. Multilevel should be default • Defaults are powerful things • Single-level regression is default • People justify multilevel models • This is backwards • Multilevel estimates usually better • Should have to justify not using multilevel model
9. Goals • Introduce multilevel models • How shrinkage and pooling work • Why they produce better estimates • How to program with ulam • Methods of plotting and comparing • Open up more options
10. Multilevel models • Usual use is to model clustering • Classrooms within schools • Students within classrooms • Grades within students • Questions within exams • Repeat measures of units • Imbalance in sampling • “pseudoreplication”
11. • Examples from earlier: • !Kung individuals in families • Species in clades • Nations in continents • Applicants in departments Multilevel models
12. Example: Tadpole predation • Numbers of surviving tadpoles • Different densities/sizes • With and without predators • We’ll focus on variation across tanks β BOE τ BT QBSBNFUFST 5P HFU UIF QPTUFSJPS GPS θ ZPVE KVTU FYQPOFOUJBUF U GPS τ *MM VTF UIJT LJOE PG MJOL JO POF PG UIF NPEFM ĕUT UP DPNF TP ZPVMM HFU U MPPLT MJLF JO DPEF GPSN  &YBNQMF #PMLFST 3FFEGSPHT 8FMM VTF BO FYBNQMF BMTP GSPN #FO # UBMJUZ EBUB PO SFFE GSPH UBEQPMFT WBSJBCMZ FYQPTFE UP BRVBUJD QSFEBUPST BU FYQ EFUFSNJOFE EFOTJUJFT :PV DBO MPBE UIF QBDLBHF BOE EBUB XJUI )&//6ǯ/"1%&+(&+$ǰ !1ǯ/""!#/,$0ǰ ! ʆǦ /""!#/,$0 ćF EBUB GSBNF IBT  SPXT BOE  DPMVNOT 8FSF HPJOH UP CF JOUFSFTUFE JO QSFE UIF OVNCFS PG UBEQPMFT UIBU TVSWJWFE UIF EVSBUJPO PG UIF FYQFSJNFOU PVS PG !" BU UIF TUBSU ćF QSFEJDUPS WBSJBCMFT PG JOUFSFTU XJMM CF !"+0&16 -/"! UIF BCTFODF PG QSFEBUPST BOE 0&7" UIF TJ[F PG UBEQPMFT *OUFSDFQUPOMZ NPEFMT *UMM CF VTFGVM UP CFHJO XJUI TJNQMF NPEFMT UIBU E QSFEJDUPST TP ZPV DBO HFU B TFOTF GPS IPX B CFUBCJOPNJBM NPEFM DPNQBSF UJPO UP B QMBJO CJOPNJBM NPEFM 4P MFUT ĕU CPUI UP UIFTF EBUB BO PMEGBTIJPO SFHSFTTJPO BOE B CFUBCJOPNJBM SFHSFTTJPO 'JSTU UIF SFHVMBS CJOPNJBM NPEFM ćJT DPEF JT KVTU MJLF ZPVE FYQFDU BęFS *ƾƿǑǁ ʆǦ *-ǯ )&01ǯ
13. Tadpole models • Structure: • Tadpoles in tanks, different densities • Outcome: number surviving • Can fit two basic models: 1. Dummy variable for each tank 2. Multilevel model with varying intercepts by tank
14. Regularized intercepts number surviving, tank i regularizing prior 7BSZJOH JOUFSDFQUT BSF UIF TJNQMFTU LJOE PG ŃĮĿņĶĻĴ IJijijIJİŁŀ 'PS FBDI DMVTUFS XF VTF B VOJRVF JOUFSDFQU QBSBNFUFS ćJT JT OP EJČFSFOU UIBO UIF DBUFHPSJDBM WBS QMFT GSPN QSFWJPVT DIBQUFST FYDFQU OPX XF BMTP BEBQUJWFMZ MFBSO UIF QSJPS UIBU UP BMM PG UIFTF JOUFSDFQUT ćJT BEBQUJWF MFBSOJOH JT UIF BCTFODF PG BNOFTJB EJTDV TUBSU PG UIF DIBQUFS 8IFO XIBU XF MFBSO BCPVU FBDI DMVTUFS JOGPSNT BMM UIF PUI XF MFBSO UIF QSJPS TJNVMUBOFPVT UP MFBSOJOH UIF JOUFSDFQUT )FSF JT B NPEFM GPS QSFEJDUJOH UBEQPMF NPSUBMJUZ JO FBDI UBOL VTJOH UIF S QSJPST PG FBSMJFS DIBQUFST 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] [unique log-odds f αK ∼ /PSNBM(, .) GPS K = .. "OE ZPV DBO BQQSPYJNBUF UIJT QPTUFSJPS VTJOH 0'( BT JO QSFWJPVT DIBQUFST ȕ (& /# /)& '0./ - 1-$' ɶ/)& ʚǶ ǎǣ)-*2ǿȀ / ʚǶ '$./ǿ  ʙ ɶ.0-1Ǣ
15. 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] [unique log-odd αK ∼ /PSNBM(, .) GPS K = .. "OE ZPV DBO BQQSPYJNBUF UIJT QPTUFSJPS VTJOH 0'( BT JO QSFWJPVT DIBQUFST 3 DPEF  ȕ (& /# /)& '0./ - 1-$' ɶ/)& ʚǶ ǎǣ)-*2ǿȀ / ʚǶ '$./ǿ  ʙ ɶ.0-1Ǣ  ʙ ɶ ).$/4Ǣ /)& ʙ ɶ/)& Ȁ ȕ ++-*3$(/ +*./ -$*- (ǎǐǡǎ ʚǶ 0'(ǿ '$./ǿ  ʡ $)*(ǿ  Ǣ + Ȁ Ǣ '*"$/ǿ+Ȁ ʚǶ ȁ/)&Ȃ Ǣ ȁ/)&Ȃ ʡ )*-(ǿ Ǎ Ǣ ǎǡǒ Ȁ ȀǢ /ʙ/ Ǣ #$).ʙǑ Ǣ '*"Ǿ'$&ʙ Ȁ *G ZPV JOTQFDU UIF QPTUFSJPS +- $.ǿ(ǎǐǡǎǢ +/#ʙǏȀ ZPVMM TFF  EJČFSFOU JO Regularized intercepts BOE UIF WBSJBUJPO BNPOH UBOLT JT XIBU XF XBOU ćJT XJMM 7BSZJOH JOUFSDFQUT BSF UIF TJNQMFTU LJOE PG ŃĮĿņĶĻĴ IJijijIJİ XF VTF B VOJRVF JOUFSDFQU QBSBNFUFS ćJT JT OP EJČFSFOU U QMFT GSPN QSFWJPVT DIBQUFST FYDFQU OPX XF BMTP BEBQUJWF UP BMM PG UIFTF JOUFSDFQUT ćJT BEBQUJWF MFBSOJOH JT UIF BC TUBSU PG UIF DIBQUFS 8IFO XIBU XF MFBSO BCPVU FBDI DMV XF MFBSO UIF QSJPS TJNVMUBOFPVT UP MFBSOJOH UIF JOUFSDFQUT )FSF JT B NPEFM GPS QSFEJDUJOH UBEQPMF NPSUBMJUZ JO QSJPST PG FBSMJFS DIBQUFST 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αK ∼ /PSNBM(, .) GPS "OE ZPV DBO BQQSPYJNBUF UIJT QPTUFSJPS VTJOH 0'( BT JO 3 DPEF  ȕ (& /# /)& '0./ - 1-$' ɶ/)& ʚǶ ǎǣ)-*2ǿȀ / ʚǶ '$./ǿ
16. Adaptive regularization  &9".1-& .6-5*-&7&- 5"%10-&4 DIBOHFT GSPN UIF QSFWJPVT NPEFM IJHIMJHIUFE JO CMVF 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αK ∼ /PSNBM(¯ α, σ) ¯ α ∼ /PSNBM(, .) [pr σ ∼ &YQPOFOUJBM() [prior for standar /PUJDF UIBU UIF QSJPS GPS UIF UBOL JOUFSDFQUT JT OPX B GVODUJPO PG UXP QB σ :PV DBO TBZ ¯ α MJLF iCBS BMQIBw ćF CBS NFBOT BWFSBHF ćFTF UXP QBSB QSJPS JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF (BVTTJBO EJTUSJCVUJPO TUBOEBSE EFWJBUJPO σ JT UIF QSJPS GPS FBDI UBOLT JOUFSDFQU #VU UIBU QSJPS JUTF BOE σ 4P UIFSF BSF UXP MFWFMT JO UIF NPEFM FBDI SFTFNCMJOH B TJNQMFS NPEF UIF PVUDPNF JT 4 UIF QBSBNFUFST BSF UIF WFDUPS α BOE UIF QSJPS JT αK ∼ /PS TFDPOE MFWFM UIF iPVUDPNFw WBSJBCMF JT UIF WFDUPS PG JOUFSDFQU QBSBNFUFST α
17. Adaptive regularization varying intercepts  &9".1-& .6-5*-&7&- 5"%10-&4 DIBOHFT GSPN UIF QSFWJPVT NPEFM IJHIMJHIUFE JO CMVF 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αK ∼ /PSNBM(¯ α, σ) ¯ α ∼ /PSNBM(, .) [pr σ ∼ &YQPOFOUJBM() [prior for standar /PUJDF UIBU UIF QSJPS GPS UIF UBOL JOUFSDFQUT JT OPX B GVODUJPO PG UXP QB σ :PV DBO TBZ ¯ α MJLF iCBS BMQIBw ćF CBS NFBOT BWFSBHF ćFTF UXP QBSB QSJPS JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF (BVTTJBO EJTUSJCVUJPO TUBOEBSE EFWJBUJPO σ JT UIF QSJPS GPS FBDI UBOLT JOUFSDFQU #VU UIBU QSJPS JUTF BOE σ 4P UIFSF BSF UXP MFWFMT JO UIF NPEFM FBDI SFTFNCMJOH B TJNQMFS NPEF UIF PVUDPNF JT 4 UIF QBSBNFUFST BSF UIF WFDUPS α BOE UIF QSJPS JT αK ∼ /PS TFDPOE MFWFM UIF iPVUDPNFw WBSJBCMF JT UIF WFDUPS PG JOUFSDFQU QBSBNFUFST α
18. Terminology • Varying intercepts also called random intercepts • Neither of these terms makes much sense • “random”? Sometimes associated with research design, but design irrelevant • Ordinary dummy variables also “vary” across clusters • Distinctive because individual intercepts learn from one another • mnestic: opposite of amnestic
19.  &9".1-& .6-5*-&7&- 5"%10-&4 DIBOHFT GSPN UIF QSFWJPVT NPEFM IJHIMJHIUFE JO CMVF 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αK ∼ /PSNBM(¯ α, σ) ¯ α ∼ /PSNBM(, .) [pr σ ∼ &YQPOFOUJBM() [prior for standar /PUJDF UIBU UIF QSJPS GPS UIF UBOL JOUFSDFQUT JT OPX B GVODUJPO PG UXP QB σ :PV DBO TBZ ¯ α MJLF iCBS BMQIBw ćF CBS NFBOT BWFSBHF ćFTF UXP QBSB QSJPS JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF (BVTTJBO EJTUSJCVUJPO TUBOEBSE EFWJBUJPO σ JT UIF QSJPS GPS FBDI UBOLT JOUFSDFQU #VU UIBU QSJPS JUTF BOE σ 4P UIFSF BSF UXP MFWFMT JO UIF NPEFM FBDI SFTFNCMJOH B TJNQMFS NPEF UIF PVUDPNF JT 4 UIF QBSBNFUFST BSF UIF WFDUPS α BOE UIF QSJPS JT αK ∼ /PS TFDPOE MFWFM UIF iPVUDPNFw WBSJBCMF JT UIF WFDUPS PG JOUFSDFQU QBSBNFUFST α Adaptive regularization mean standard deviation varying intercepts
20.  &9".1-& .6-5*-&7&- 5"%10-&4 DIBOHFT GSPN UIF QSFWJPVT NPEFM IJHIMJHIUFE JO CMVF 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αK ∼ /PSNBM(¯ α, σ) ¯ α ∼ /PSNBM(, .) [pr σ ∼ &YQPOFOUJBM() [prior for standar /PUJDF UIBU UIF QSJPS GPS UIF UBOL JOUFSDFQUT JT OPX B GVODUJPO PG UXP QB σ :PV DBO TBZ ¯ α MJLF iCBS BMQIBw ćF CBS NFBOT BWFSBHF ćFTF UXP QBSB QSJPS JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF (BVTTJBO EJTUSJCVUJPO TUBOEBSE EFWJBUJPO σ JT UIF QSJPS GPS FBDI UBOLT JOUFSDFQU #VU UIBU QSJPS JUTF BOE σ 4P UIFSF BSF UXP MFWFMT JO UIF NPEFM FBDI SFTFNCMJOH B TJNQMFS NPEF UIF PVUDPNF JT 4 UIF QBSBNFUFST BSF UIF WFDUPS α BOE UIF QSJPS JT αK ∼ /PS TFDPOE MFWFM UIF iPVUDPNFw WBSJBCMF JT UIF WFDUPS PG JOUFSDFQU QBSBNFUFST α Adaptive regularization varying intercepts Survival across tanks has a distribution. This distribution is the prior for each tank. Distribution needs its own prior.
21.   .0%&-4 8*5)065 "./&4*" .$"( ʡ  3+ǿ ǎ Ȁ ȀǢ /ʙ/ Ǣ #$).ʙǑ Ǣ '*"Ǿ'$&ʙ Ȁ ćJT NPEFM QSPWJEFT QPTUFSJPS EJTUSJCVUJPOT GPS  QBSBNFUFST POF PWFSBMM TBNQMF JOU ¯ α UIF TUBOEBSE EFWJBUJPO BNPOH UBOLT σ BOE UIFO  QFSUBOL JOUFSDFQUT -FUT DIFDL 8 UIPVHI UP TFF UIF FČFDUJWF OVNCFS PG QBSBNFUFST 8FMM DPNQBSF UIF FBSMJFS NPEFM ( XJUI UIF OFX NVMUJMFWFM NPEFM DIBOHFT GSPN UIF QSFWJPVT NPEFM IJHIMJHIUFE JO CMVF 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αK ∼ /PSNBM(¯ α, σ) ¯ α ∼ /PSNBM(, .) [pri σ ∼ &YQPOFOUJBM() [prior for standard /PUJDF UIBU UIF QSJPS GPS UIF UBOL JOUFSDFQUT JT OPX B GVODUJPO PG UXP QB σ :PV DBO TBZ ¯ α MJLF iCBS BMQIBw ćF CBS NFBOT BWFSBHF ćFTF UXP QBSBN QSJPS JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF (BVTTJBO EJTUSJCVUJPO X TUBOEBSE EFWJBUJPO σ JT UIF QSJPS GPS FBDI UBOLT JOUFSDFQU #VU UIBU QSJPS JUTFM BOE σ 4P UIFSF BSF UXP MFWFMT JO UIF NPEFM FBDI SFTFNCMJOH B TJNQMFS NPEFM UIF PVUDPNF JT 4 UIF QBSBNFUFST BSF UIF WFDUPS α BOE UIF QSJPS JT αK ∼ /PSN TFDPOE MFWFM UIF iPVUDPNFw WBSJBCMF JT UIF WFDUPS PG JOUFSDFQU QBSBNFUFST α BSF ¯ α BOE σ BOE UIFJS QSJPST BSF ¯ α ∼ /PSNBM(, .) BOE σ ∼ &YQPOFOUJBM ćFTF UXP QBSBNFUFST ¯ α BOE σ BSF PęFO SFGFSSFE UP BT ĵņĽIJĿĽĮĿĮĺ QBSBNFUFST GPS QBSBNFUFST "OE UIFJS QSJPST BSF PęFO DBMMFE ĵņĽIJĿĽĿĶļĿ UIFSF JT OP MJNJU UP IPX NBOZ iIZQFSw MFWFMT ZPV DBO JOTUBMM JO B NPEFM 'PS FY QPQVMBUJPOT PG UBOLT DPVME CF FNCFEEFE XJUIJO EJČFSFOU SFHJPOT PG IBCJUBU UIFSF BSF MJNJUT CPUI CFDBVTF PG DPNQVUBUJPO BOE PVS BCJMJUZ UP VOEFSTUBOE (BVTTJBO QSJPS PS MJLFMJIPPE JT ĕOJUF WBSJBODF ćF EJTUSJCVUJPO MPPLT TZNNFUSJD CFDBVTF EPOU TBZ IPX JU JT TLFXFE UIFO TZNNFUSJD JT UIF NBYJNVN FOUSPQZ TIBQF "CPWF BMM UIFSF JT O SFRVJSJOH UIF (BVTTJBO EJTUSJCVUJPO PG WBSZJOH FČFDUT 4P JG ZPV IBWF B HPPE SFBTPO UP VTF B EJTUSJCVUJPO UIFO EP TP ćF QSBDUJDF QSPCMFNT BU UIF FOE PG UIF DIBQUFS QSPWJEF BO FYBNQMF $PNQVUJOH UIF QPTUFSJPS DPNQVUFT CPUI MFWFMT TJNVMUBOFPVTMZ JO UIF TBNF XBZ UI SPCPU BU UIF TUBSU PG UIF DIBQUFS MFBSOFE CPUI BCPVU FBDI DBGÏ BOE UIF WBSJBUJPO BNPOH #VU ZPV DBOOPU ĕU UIJT NPEFM XJUI ,0+ 8IZ #FDBVTF UIF QSPCBCJMJUZ PG UIF EBUB NVT BWFSBHF PWFS UIF MFWFM  QBSBNFUFST ¯ α BOE σ #VU ,0+ KVTU IJMM DMJNCT VTJOH TUBUJD WBMV BMM PG UIF QBSBNFUFST *U DBOU TFF UIF MFWFMT 'PS NPSF FYQMBOBUJPO TFF UIF 0WFSUIJOLJO GVSUIFS EPXO :PV DBO IPXFWFS ĕU UIJT NPEFM XJUI 0'( (ǎǐǡǏ ʚǶ 0'(ǿ '$./ǿ  ʡ $)*(ǿ  Ǣ + Ȁ Ǣ '*"$/ǿ+Ȁ ʚǶ ȁ/)&Ȃ Ǣ ȁ/)&Ȃ ʡ )*-(ǿ Ǿ- Ǣ .$"( Ȁ Ǣ Ǿ- ʡ )*-(ǿ Ǎ Ǣ ǎǡǒ Ȁ Ǣ
22. UIPVHI UP TFF UIF FČFDUJWF OVNCFS PG QBSBNFUFST 8FMM DPNQBSF UIF XJUI UIF OFX NVMUJMFWFM NPEFM 3 DPEF  *(+- ǿ (ǎǐǡǎ Ǣ (ǎǐǡǏ Ȁ   +    2 $"#/   (ǎǐǡǏ ǏǍǏ Ǐǎǡǔ Ǎ ǎ ǔǡǐǒ  (ǎǐǡǎ Ǐǎǐ ǏǑǡǕ ǎǎ Ǎ Ǒǡǔǎ ǐǡǒǕ ćFSF BSF UXP GBDUT UP OPUF IFSF 'JSTU UIF NVMUJMFWFM NPEFM IBT POMZ  ćFSF BSF  GFXFS FČFDUJWF QBSBNFUFST UIBO BDUVBM QBSBNFUFST CFDBV UP FBDI JOUFSDFQU TISJOLT UIFN BMM UPXBSET UIF NFBO ¯ α *O UIJT DBTF BCMZ TUSPOH $IFDL UIF NFBO PG .$"( XJUI +- $. BOE ZPVMM TFF JUT ĿIJĴłĹĮĿĶŇĶĻĴ ĽĿĶļĿ MJLF ZPVWF VTFE JO QSFWJPVT DIBQUFST CVU OPX MBSJ[BUJPO IBT CFFO MFBSOFE GSPN UIF EBUB JUTFMG 4FDPOE OPUJDF UIBU (ǎǐǡǏ IBT GFXFS FČFDUJWF QBSBNFUFST UIBO UIF PSEJOBSZ ĕYFE NPEFM ( UIF GBDU UIBU UIF PSEJOBSZ NPEFM IBT GFXFS BDUVBM QBSBNFUFST POMZ  USB UXP QBSBNFUFST JO UIF NVMUJMFWFM NPEFM BMMPXFE JU UP MFBSO B NPSF B QSJPS UP BEBQUJWFMZ SFHVMBSJ[F ćJT SFTVMUFE JO B MFTT ĘFYJCMF QPTUFSJP FČFDUJWF QBSBNFUFST 0WFSUIJOLJOH 26"1 GBJMT .$.$ TVDDFFET 8IZ EPFTOU TJNQMF RVBESBUJ GPS FYBNQMF ,0+ XPSL XJUI NVMUJMFWFM NPEFMT 8IFO B QSJPS JT JUTFMG B GVODU • m13.1: 48 parameters vs 25 effective • m13.2: 50 parameters vs 22 effective • Model with more parameters has fewer effective parameters • Why? Ended up with stronger prior.
23. Don’t expect predictions to match observations exactly. Instead expect shrinkage. Fixed estimate Multilevel estimate 0.0 0.2 0.4 0.6 0.8 1.0 tank proportion survival 1 16 32 48 small tanks medium tanks large tanks 'ĶĴłĿIJ ƉƋƉ &NQJSJDBM QSPQPSUJPOT PG TVSWJWPST JO FBDI UBEQPMF UBOL TIPXO CZ UIF ĕMMFE CMVF QPJOUT QMPUUFE XJUI UIF  QFSUBOL QBSBNFUFST GSPN UIF NVMUJMFWFM NPEFM TIPXO CZ UIF CMBDL DJSDMFT ćF EBTIFE MJOF MP
24. 0.0 0.2 0.4 0.6 0.8 1.0 tank proportion survival 1 16 32 48 small tanks medium tanks large tanks 'ĶĴłĿIJ ƉƋƉ &NQJSJDBM QSPQPSUJPOT PG TVSWJWPST JO FBDI UBEQPMF UBOL TIPXO CZ UIF ĕMMFE CMVF QPJOUT QMPUUFE XJUI UIF  QFSUBOL QBSBNFUFST GSPN UIF NVMUJMFWFM NPEFM TIPXO CZ UIF CMBDL DJSDMFT ćF EBTIFE MJOF MP Population mean not equal to raw empirical mean. Why? Imbalance in amount of evidence across tanks. Fixed estimate Multilevel estimate raw mean pop mean
25. 0.0 0.2 0.4 0.6 0.8 1.0 tank proportion survival 1 16 32 48 small tanks medium tanks large tanks 'ĶĴłĿIJ ƉƋƉ &NQJSJDBM QSPQPSUJPOT PG TVSWJWPST JO FBDI UBEQPMF UBOL TIPXO CZ UIF ĕMMFE CMVF QPJOUT QMPUUFE XJUI UIF  QFSUBOL QBSBNFUFST GSPN UIF NVMUJMFWFM NPEFM TIPXO CZ UIF CMBDL DJSDMFT ćF EBTIFE MJOF MP Small tanks => high sampling variation. More shrinkage towards mean. Further from mean => more shrinkage. Fixed estimate Multilevel estimate
26. 0.0 0.2 0.4 0.6 0.8 1.0 tank proportion survival 1 16 32 48 small tanks medium tanks large tanks 'ĶĴłĿIJ ƉƋƉ &NQJSJDBM QSPQPSUJPOT PG TVSWJWPST JO FBDI UBEQPMF UBOL TIPXO CZ UIF ĕMMFE CMVF QPJOUT QMPUUFE XJUI UIF  QFSUBOL QBSBNFUFST GSPN UIF NVMUJMFWFM NPEFM TIPXO CZ UIF CMBDL DJSDMFT ćF EBTIFE MJOF MP Fixed estimate Multilevel estimate Large tanks => low sampling variation. Less shrinkage towards mean at all distances from mean.
27. Shrinkage • Varying effect estimates shrink towards mean (a_bar) • Further from mean, more shrinkage • Fewer data in cluster, more shrinkage • Same as regression to the mean, really • Shrinkage results from pooling of information 0.2 0.4 0.6 0.8 1.0 tank probability of survival in tank 1 16 32 10 25 25
28. Ulysses’ Compass again • Why are varying effects (partial pooling) more accurate than fixed effects (no pooling)? • Grand mean: maximum underfitting • Fixed effects: maximum overfitting • Varying effects: adaptive regularization
29. Ulysses’ Compass again TFUT POF GPS FBDI UBOL 5P HFU FBDI UBOLT FYQFDUFE TV Ǿ/)& WBMVFT BOE UIFO VTF UIF MPHJTUJD USBOTGPSN 4P /PX MFUT ĕU UIF NVMUJMFWFM NPEFM XIJDI BEBQUJWF UIBU JT SFRVJSFE UP FOBCMF BEBQUJWF QPPMJOH JT UP NBLF GVODUJPO PG JUT PXO QBSBNFUFST )FSF JT UIF NVMUJMFWF TJ ∼ #JOPNJBM(OJ, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αŁĮĻĸ ∼ /PSNBM(α, σ) α ∼ /PSNBM(, ) σ ∼ )BMG$BVDIZ(, ) /PUJDF UIBU UIF QSJPS GPS UIF αŁĮĻĸ JOUFSDFQUT JT OPX B ćJT JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF (  &9".1-& .6-5*-&7&- 5"%10-&4 DIBOHFT GSPN UIF QSFWJPVT NPEFM IJHIMJHIUFE JO CMVF 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αK ∼ /PSNBM(¯ α, σ) ¯ α ∼ /PSNBM(, .) [pri σ ∼ &YQPOFOUJBM() [prior for standard /PUJDF UIBU UIF QSJPS GPS UIF UBOL JOUFSDFQUT JT OPX B GVODUJPO PG UXP QB σ :PV DBO TBZ ¯ α MJLF iCBS BMQIBw ćF CBS NFBOT BWFSBHF ćFTF UXP QBSBN QSJPS JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF (BVTTJBO EJTUSJCVUJPO X TUBOEBSE EFWJBUJPO σ JT UIF QSJPS GPS FBDI UBOLT JOUFSDFQU #VU UIBU QSJPS JUTFM BOE σ 4P UIFSF BSF UXP MFWFMT JO UIF NPEFM FBDI SFTFNCMJOH B TJNQMFS NPEFM UIF PVUDPNF JT 4 UIF QBSBNFUFST BSF UIF WFDUPS α BOE UIF QSJPS JT αK ∼ /PSN TFDPOE MFWFM UIF iPVUDPNFw WBSJBCMF JT UIF WFDUPS PG JOUFSDFQU QBSBNFUFST α BSF ¯ α BOE σ BOE UIFJS QSJPST BSF ¯ α ∼ /PSNBM(, .) BOE σ ∼ &YQPOFOUJBM ćFTF UXP QBSBNFUFST ¯ α BOE σ BSF PęFO SFGFSSFE UP BT ĵņĽIJĿĽĮĿĮĺ
30. Ulysses’ Compass again TFUT POF GPS FBDI UBOL 5P HFU FBDI UBOLT FYQFDUFE TV Ǿ/)& WBMVFT BOE UIFO VTF UIF MPHJTUJD USBOTGPSN 4P /PX MFUT ĕU UIF NVMUJMFWFM NPEFM XIJDI BEBQUJWF UIBU JT SFRVJSFE UP FOBCMF BEBQUJWF QPPMJOH JT UP NBLF GVODUJPO PG JUT PXO QBSBNFUFST )FSF JT UIF NVMUJMFWF TJ ∼ #JOPNJBM(OJ, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αŁĮĻĸ ∼ /PSNBM(α, σ) α ∼ /PSNBM(, ) σ ∼ )BMG$BVDIZ(, ) /PUJDF UIBU UIF QSJPS GPS UIF αŁĮĻĸ JOUFSDFQUT JT OPX B ćJT JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF ( 0 Complete pooling All clusters same  &9".1-& .6-5*-&7&- 5"%10-&4 DIBOHFT GSPN UIF QSFWJPVT NPEFM IJHIMJHIUFE JO CMVF 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αK ∼ /PSNBM(¯ α, σ) ¯ α ∼ /PSNBM(, .) [pri σ ∼ &YQPOFOUJBM() [prior for standard /PUJDF UIBU UIF QSJPS GPS UIF UBOL JOUFSDFQUT JT OPX B GVODUJPO PG UXP QB σ :PV DBO TBZ ¯ α MJLF iCBS BMQIBw ćF CBS NFBOT BWFSBHF ćFTF UXP QBSBN QSJPS JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF (BVTTJBO EJTUSJCVUJPO X TUBOEBSE EFWJBUJPO σ JT UIF QSJPS GPS FBDI UBOLT JOUFSDFQU #VU UIBU QSJPS JUTFM BOE σ 4P UIFSF BSF UXP MFWFMT JO UIF NPEFM FBDI SFTFNCMJOH B TJNQMFS NPEFM UIF PVUDPNF JT 4 UIF QBSBNFUFST BSF UIF WFDUPS α BOE UIF QSJPS JT αK ∼ /PSN TFDPOE MFWFM UIF iPVUDPNFw WBSJBCMF JT UIF WFDUPS PG JOUFSDFQU QBSBNFUFST α BSF ¯ α BOE σ BOE UIFJS QSJPST BSF ¯ α ∼ /PSNBM(, .) BOE σ ∼ &YQPOFOUJBM ćFTF UXP QBSBNFUFST ¯ α BOE σ BSF PęFO SFGFSSFE UP BT ĵņĽIJĿĽĮĿĮĺ
31. Ulysses’ Compass again TFUT POF GPS FBDI UBOL 5P HFU FBDI UBOLT FYQFDUFE TV Ǿ/)& WBMVFT BOE UIFO VTF UIF MPHJTUJD USBOTGPSN 4P /PX MFUT ĕU UIF NVMUJMFWFM NPEFM XIJDI BEBQUJWF UIBU JT SFRVJSFE UP FOBCMF BEBQUJWF QPPMJOH JT UP NBLF GVODUJPO PG JUT PXO QBSBNFUFST )FSF JT UIF NVMUJMFWF TJ ∼ #JOPNJBM(OJ, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αŁĮĻĸ ∼ /PSNBM(α, σ) α ∼ /PSNBM(, ) σ ∼ )BMG$BVDIZ(, ) /PUJDF UIBU UIF QSJPS GPS UIF αŁĮĻĸ JOUFSDFQUT JT OPX B ćJT JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF ( 0 ∞ Complete pooling All clusters same No pooling All clusters unrelated  &9".1-& .6-5*-&7&- 5"%10-&4 DIBOHFT GSPN UIF QSFWJPVT NPEFM IJHIMJHIUFE JO CMVF 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αK ∼ /PSNBM(¯ α, σ) ¯ α ∼ /PSNBM(, .) [pri σ ∼ &YQPOFOUJBM() [prior for standard /PUJDF UIBU UIF QSJPS GPS UIF UBOL JOUFSDFQUT JT OPX B GVODUJPO PG UXP QB σ :PV DBO TBZ ¯ α MJLF iCBS BMQIBw ćF CBS NFBOT BWFSBHF ćFTF UXP QBSBN QSJPS JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF (BVTTJBO EJTUSJCVUJPO X TUBOEBSE EFWJBUJPO σ JT UIF QSJPS GPS FBDI UBOLT JOUFSDFQU #VU UIBU QSJPS JUTFM BOE σ 4P UIFSF BSF UXP MFWFMT JO UIF NPEFM FBDI SFTFNCMJOH B TJNQMFS NPEFM UIF PVUDPNF JT 4 UIF QBSBNFUFST BSF UIF WFDUPS α BOE UIF QSJPS JT αK ∼ /PSN TFDPOE MFWFM UIF iPVUDPNFw WBSJBCMF JT UIF WFDUPS PG JOUFSDFQU QBSBNFUFST α BSF ¯ α BOE σ BOE UIFJS QSJPST BSF ¯ α ∼ /PSNBM(, .) BOE σ ∼ &YQPOFOUJBM ćFTF UXP QBSBNFUFST ¯ α BOE σ BSF PęFO SFGFSSFE UP BT ĵņĽIJĿĽĮĿĮĺ
32. Ulysses’ Compass again TFUT POF GPS FBDI UBOL 5P HFU FBDI UBOLT FYQFDUFE TV Ǿ/)& WBMVFT BOE UIFO VTF UIF MPHJTUJD USBOTGPSN 4P /PX MFUT ĕU UIF NVMUJMFWFM NPEFM XIJDI BEBQUJWF UIBU JT SFRVJSFE UP FOBCMF BEBQUJWF QPPMJOH JT UP NBLF GVODUJPO PG JUT PXO QBSBNFUFST )FSF JT UIF NVMUJMFWF TJ ∼ #JOPNJBM(OJ, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αŁĮĻĸ ∼ /PSNBM(α, σ) α ∼ /PSNBM(, ) σ ∼ )BMG$BVDIZ(, ) /PUJDF UIBU UIF QSJPS GPS UIF αŁĮĻĸ JOUFSDFQUT JT OPX B ćJT JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF ( 0 ∞ Complete pooling All clusters same No pooling All clusters unrelated prior posterior 1.5  &9".1-& .6-5*-&7&- 5"%10-&4 DIBOHFT GSPN UIF QSFWJPVT NPEFM IJHIMJHIUFE JO CMVF 4J ∼ #JOPNJBM(/J, QJ) MPHJU(QJ) = αŁĮĻĸ[J] αK ∼ /PSNBM(¯ α, σ) ¯ α ∼ /PSNBM(, .) σ ∼ &YQPOFOUJBM() [pri /PUJDF UIBU UIF QSJPS GPS UIF UBOL JOUFSDFQUT JT OPX B GVODUJP σ :PV DBO TBZ ¯ α MJLF iCBS BMQIBw ćF CBS NFBOT BWFSBHF ćFT QSJPS JT XIFSF UIF iNVMUJw JO NVMUJMFWFM BSJTFT ćF (BVTTJBO E TUBOEBSE EFWJBUJPO σ JT UIF QSJPS GPS FBDI UBOLT JOUFSDFQU #VU UI BOE σ 4P UIFSF BSF UXP MFWFMT JO UIF NPEFM FBDI SFTFNCMJOH B TJN UIF PVUDPNF JT 4 UIF QBSBNFUFST BSF UIF WFDUPS α BOE UIF QSJPS J TFDPOE MFWFM UIF iPVUDPNFw WBSJBCMF JT UIF WFDUPS PG JOUFSDFQU QB BSF ¯ α BOE σ BOE UIFJS QSJPST BSF ¯ α ∼ /PSNBM(, .) BOE σ ∼ ćFTF UXP QBSBNFUFST ¯ α BOE σ BSF PęFO SFGFSSFE UP BT ĵņ
33. Regularizing distribution  &9".1-& .6-5*-&7&- 5"%10-&4  -3 -2 -1 0 1 2 3 4 0.00 0.10 0.20 0.30 log-odds survive Density 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 probability survive Density 'ĶĴłĿIJ ƉƋƊ ćF JOGFSSFE QPQVMBUJPO PG TVSWJWBM BDSPTT UBOLT -Fę  (BVTTJBO EJTUSJCVUJPOT PG UIF MPHPEET PG TVSWJWBM TBNQMFE GSPN UIF QPTUF SJPS PG (ǎǐǡǏ 3JHIU 4VSWJWBM QSPCBCJMJUJFT GPS  OFX TJNVMBUFE UBOLT BWFSBHJOH PWFS UIF QPTUFSJPS EJTUSJCVUJPO PO UIF MFę
34. • Simulate to demonstrate accuracy advantage • 60 ponds • 5, 10, 25, 35 tadpoles each of 15 pond n true.a s p.nopool p.partpool p.true 1 1 5 -3.089936132 1 0.2000000 0.32173203 0.04352429 2 2 5 0.267290817 5 1.0000000 0.91305884 0.56642768 3 3 5 0.896554101 4 0.8000000 0.79164823 0.71024085 4 4 5 1.934806220 5 1.0000000 0.91276066 0.87378044 5 5 5 -0.758682067 0 0.0000000 0.17692527 0.31893247 6 6 5 3.904836388 5 1.0000000 0.91337140 0.98025353 7 7 5 2.271914139 4 0.8000000 0.79349508 0.90652411 8 8 5 2.886101619 4 0.8000000 0.79557800 0.94715510 9 9 5 1.436457877 3 0.6000000 0.64219989 0.80790553 10 10 5 1.156079068 3 0.6000000 0.64414477 0.76061953 Ulysses’ Compass again
35. Raw proportion Multilevel estimate   .6-5*-&7&- .0%&-4 0.00 0.10 0.20 0.30 pond absolute error 1 10 20 30 40 50 60 tiny (5) small (10) medium (25) large (35) 'ĶĴłĿIJ ƉƊƋ &SSPS PG OPQPPMJOH BOE QBSUJBM QPPMJOH FTUJNBUFT GPS UIF TJN VMBUFE UBEQPMF QPOET ćF IPSJ[POUBM BYJT EJTQMBZT QPOE OVNCFS ćF WFSUJ DBM BYJT NFBTVSFT UIF BCTPMVUF FSSPS JO UIF QSFEJDUFE QSPQPSUJPO PG TVSWJWPST DPNQBSFE UP UIF USVF WBMVF VTFE JO UIF TJNVMBUJPO ćF IJHIFS UIF QPJOU
36. Raw proportion Multilevel estimate   .6-5*-&7&- .0%&-4 0.00 0.10 0.20 0.30 pond absolute error 1 10 20 30 40 50 60 tiny (5) small (10) medium (25) large (35) 'ĶĴłĿIJ ƉƊƋ &SSPS PG OPQPPMJOH BOE QBSUJBM QPPMJOH FTUJNBUFT GPS UIF TJN VMBUFE UBEQPMF QPOET ćF IPSJ[POUBM BYJT EJTQMBZT QPOE OVNCFS ćF WFSUJ DBM BYJT NFBTVSFT UIF BCTPMVUF FSSPS JO UIF QSFEJDUFE QSPQPSUJPO PG TVSWJWPST DPNQBSFE UP UIF USVF WBMVF VTFE JO UIF TJNVMBUJPO ćF IJHIFS UIF QPJOU avg raw error avg multilevel error
37. Raw proportion Multilevel estimate   .6-5*-&7&- .0%&-4 0.00 0.10 0.20 0.30 pond absolute error 1 10 20 30 40 50 60 tiny (5) small (10) medium (25) large (35) 'ĶĴłĿIJ ƉƊƋ &SSPS PG OPQPPMJOH BOE QBSUJBM QPPMJOH FTUJNBUFT GPS UIF TJN VMBUFE UBEQPMF QPOET ćF IPSJ[POUBM BYJT EJTQMBZT QPOE OVNCFS ćF WFSUJ DBM BYJT NFBTVSFT UIF BCTPMVUF FSSPS JO UIF QSFEJDUFE QSPQPSUJPO PG TVSWJWPST DPNQBSFE UP UIF USVF WBMVF VTFE JO UIF TJNVMBUJPO ćF IJHIFS UIF QPJOU