Tweet
Tweet

Slide 1

Slide 1 text

Python and MongoDB in Astronomy Dan Foreman-Mackey Center for Cosmology and Particle Physics Department of Physics @ NYU In collaboration with: David W. Hogg (NYU), Larry Widrow (Queen’s), Dustin Lang (Princeton), Jonathan Sick (Queen’s), Micha Gorelick (NYU) and many others...

Slide 2

Slide 2 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Astronomy 101 How to Study the Cosmos Python, MongoDB, etc. Case Studies

Slide 3

Slide 3 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Astronomy 101 How to Study the Cosmos Python, MongoDB, etc. Case Studies Andromeda The Milky Way The Internet

Slide 4

Slide 4 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com The Universe Galaxies Stars Planets

Slide 5

Slide 5 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com The Universe Galaxies Stars Planets What is the Universe Made of?

Slide 6

Slide 6 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com The Universe Galaxies Stars Planets What is the Universe Made of? Are there other Earth- like planets?

Slide 7

Slide 7 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com The Universe Galaxies Stars Planets What is the Universe Made of? Are there other Earth- like planets?

Slide 8

Slide 8 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of?

Slide 9

Slide 9 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Source: Wikipedia (Adam Evans)

Slide 10

Slide 10 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius

Slide 11

Slide 11 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius Observed

Slide 12

Slide 12 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius Observed WTF?

Slide 13

Slide 13 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius Observed WTF? ?

Slide 14

Slide 14 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius Observed WTF? ? ?

Slide 15

Slide 15 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius Observed WTF? ? ?

Slide 16

Slide 16 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Dark Matter

Slide 17

Slide 17 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Source: NASA / WMAP Science Team Time PyGotham Size of the Universe observable

Slide 18

Slide 18 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Atoms 4% Dark Matter 23% Dark Energy 73% Heavy Elements 0.03% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011)

Slide 19

Slide 19 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Atoms 4% Dark Matter 23% Dark Energy 73% Heavy Elements 0.03% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Source: DFM & Widrow (in prep)

Slide 20

Slide 20 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Atoms 4% Dark Matter 23% Dark Energy 73% Heavy Elements 0.03% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Source: http://apod.nasa.gov Source: DFM & Widrow (in prep)

Slide 21

Slide 21 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Atoms 4% Dark Matter 23% Dark Energy 73% Heavy Elements 0.03% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Source: http://apod.nasa.gov Source: DFM & Widrow (in prep)

Slide 22

Slide 22 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Credit: The Millennium Simulation Project

Slide 23

Slide 23 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Credit: The Millennium Simulation Project

Slide 24

Slide 24 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Credit: The Millennium Simulation Project

Slide 25

Slide 25 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy

Slide 26

Slide 26 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Credit: Jonathan Sick jonathansick.ca

Slide 27

Slide 27 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Credit: Jonathan Sick jonathansick.ca

Slide 28

Slide 28 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Credit: Jonathan Sick jonathansick.ca MegaCam: 340 MegaPixels

Slide 29

Slide 29 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy MegaCam: 340 MegaPixels Credit: NASA

Slide 30

Slide 30 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Imaging Source: NASA / ESA

Slide 31

Slide 31 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Imaging Source: NASA / ESA Spectroscopy

Slide 32

Slide 32 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Imaging Source: NASA / ESA Spectroscopy Spectroscopy Source: Riaud & Schneider (2007)

Slide 33

Slide 33 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Imaging Source: NASA / ESA Spectroscopy Spectroscopy Source: Riaud & Schneider (2007)

Slide 34

Slide 34 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy is Open a lot of

Slide 35

Slide 35 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy is Open a lot of and there’s a lot of it!

Slide 36

Slide 36 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy is Open Hubble SDSS 2MASS 1990– 2000– 1997–2001 sdss.org archive.stsci.edu/hst www.ipac.caltech.edu/2mass Pan-STARRS LSST Planned GAIA

Slide 37

Slide 37 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy is Open Hubble SDSS 2MASS 1990– 2000– 1997–2001 sdss.org archive.stsci.edu/hst www.ipac.caltech.edu/2mass Pan-STARRS LSST Planned GAIA

Slide 38

Slide 38 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy is Open Hubble SDSS 2MASS 1990– 2000– 1997–2001 sdss.org archive.stsci.edu/hst www.ipac.caltech.edu/2mass Pan-STARRS LSST Planned GAIA

Slide 39

Slide 39 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in?

Slide 40

Slide 40 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in?

Slide 41

Slide 41 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in?

Slide 42

Slide 42 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in?

Slide 43

Slide 43 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in?

Slide 44

Slide 44 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in? + Scientific Python Stack

Slide 45

Slide 45 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in? + Scientific Python Stack

Slide 46

Slide 46 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in? + Scientific Python Stack

Slide 47

Slide 47 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com What about ? Easy Pythonic Flexible Scalable

Slide 48

Slide 48 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies

Slide 49

Slide 49 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82

Slide 50

Slide 50 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 728 SESAR ET AL. Ses S07 Labela Ntot A 84 B 144 C 54 D 8 E 11 F 11 G 10 H 7 I 4 J 26 K 8 L 3 M 5 Source: Sesar et al. (2010)

Slide 51

Slide 51 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82

Slide 52

Slide 52 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82

Slide 53

Slide 53 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 800k “Fields” ~ 12TB Imaging data > 1M “Target Stars”

Slide 54

Slide 54 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 Time Photons/Brightness

Slide 55

Slide 55 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)]

Slide 56

Slide 56 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)] Stars

Slide 57

Slide 57 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)] p const ⌘ M Y i =1 [(1 P bad )p good + P bad p bad ] p var ⌘ M Y i =1 [(1 P bad )p var , good + P bad p bad ] Stars

Slide 58

Slide 58 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)] p const ⌘ M Y i =1 [(1 P bad )p good + P bad p bad ] p var ⌘ M Y i =1 [(1 P bad )p var , good + P bad p bad ] Stars Runs Runs

Slide 59

Slide 59 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)] p const ⌘ M Y i =1 [(1 P bad )p good + P bad p bad ] p var ⌘ M Y i =1 [(1 P bad )p var , good + P bad p bad ] Stars Runs Runs p good ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ ) “Constant & Good” p var , good ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ + ⌃2 var ) “Variable & Good” pbad ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ + ⌃2 bad ) “Bad”

Slide 60

Slide 60 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)] p const ⌘ M Y i =1 [(1 P bad )p good + P bad p bad ] p var ⌘ M Y i =1 [(1 P bad )p var , good + P bad p bad ] Stars Runs Runs p good ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ ) “Constant & Good” p var , good ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ + ⌃2 var ) “Variable & Good” pbad ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ + ⌃2 bad ) “Bad” Npars = Nstars + Nruns + 6 ⇥ = {~ f0,~ f⇤, , ⌘, ⌃2 var , Pvar, ⌃2 bad , Pbad }

Slide 61

Slide 61 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82

Slide 62

Slide 62 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82

Slide 63

Slide 63 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)]

Slide 64

Slide 64 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 f⇤ ( t ) = A0 + N X n=1 [ An sin( !t ) + Bn cos( !t )]

Slide 65

Slide 65 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies CFHT 4.7 Gigapixel mosaic of M31 Source: Jonathan Sick (Queen’s University)

Slide 66

Slide 66 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Source: Jonathan Sick (Queen’s University)

Slide 67

Slide 67 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Source: Jonathan Sick (Queen’s University)

Slide 68

Slide 68 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Source: Jonathan Sick (Queen’s University)

Slide 69

Slide 69 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies CFHT Source: Jonathan Sick (Queen’s University) MongoDB Persistent Metadata + GeoSpatial Indexing img1.fits img2.fits img4000.fits ... Flat-fielding Cosmic ray removal Sky subtraction Mosaic making ...

Slide 70

Slide 70 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Astrometry.net

Slide 71

Slide 71 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Astrometry.net

Slide 72

Slide 72 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com

Slide 73

Slide 73 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com

Slide 74

Slide 74 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com

Slide 75

Slide 75 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes ~2500 JPGs Lang & Hogg (2011)

Slide 76

Slide 76 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes

Slide 77

Slide 77 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes

Slide 78

Slide 78 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes

Slide 79

Slide 79 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes Source: Lang & Hogg (2011)

Slide 80

Slide 80 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes            Source: Lang & Hogg (2011)

Slide 81

Slide 81 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes            Source: Lang & Hogg (2011) github.com/dfm/MarkovPy

Slide 82

Slide 82 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes

Slide 83

Slide 83 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Number Crunching Data Management User Interaction And Much More... ( ) Growing Datasets

Slide 84

Slide 84 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Number Crunching Data Management User Interaction And Much More... ( ) Growing Datasets Easy!

Slide 85

Slide 85 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Number Crunching Data Management User Interaction And Much More... ( ) Growing Datasets Easy! Big Questions

Slide 86

Slide 86 text

Dan Foreman-Mackey CCPP@NYU dfm.github.com Dan Foreman-Mackey Center for Cosmology & Particle Physics (NYU) dfm.github.com @__dfm__