Python and MongoDB in Astronomy

Talk from PyGotham 2011.

Transcript

1. Python and MongoDB in Astronomy Dan Foreman-Mackey Center for Cosmology and Particle Physics Department of Physics @ NYU In collaboration with: David W. Hogg (NYU), Larry Widrow (Queen’s), Dustin Lang (Princeton), Jonathan Sick (Queen’s), Micha Gorelick (NYU) and many others...
2. Dan Foreman-Mackey CCPP@NYU dfm.github.com Astronomy 101 How to Study the Cosmos Python, MongoDB, etc. Case Studies
3. Dan Foreman-Mackey CCPP@NYU dfm.github.com Astronomy 101 How to Study the Cosmos Python, MongoDB, etc. Case Studies Andromeda The Milky Way The Internet
4. Dan Foreman-Mackey CCPP@NYU dfm.github.com The Universe Galaxies Stars Planets
5. Dan Foreman-Mackey CCPP@NYU dfm.github.com The Universe Galaxies Stars Planets What is the Universe Made of?
6. Dan Foreman-Mackey CCPP@NYU dfm.github.com The Universe Galaxies Stars Planets What is the Universe Made of? Are there other Earth- like planets?
7. Dan Foreman-Mackey CCPP@NYU dfm.github.com The Universe Galaxies Stars Planets What is the Universe Made of? Are there other Earth- like planets?
8. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of?
9. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Source: Wikipedia (Adam Evans)
10. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius
11. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius Observed
12. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius Observed WTF?
13. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius Observed WTF? ?
14. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius Observed WTF? ? ?
15. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Rotational Speed Radius Observed WTF? ? ?
16. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Dark Matter
17. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Source: NASA / WMAP Science Team Time PyGotham Size of the Universe observable
18. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Atoms 4% Dark Matter 23% Dark Energy 73% Heavy Elements 0.03% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011)
19. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Atoms 4% Dark Matter 23% Dark Energy 73% Heavy Elements 0.03% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Source: DFM & Widrow (in prep)
20. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Atoms 4% Dark Matter 23% Dark Energy 73% Heavy Elements 0.03% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Source: http://apod.nasa.gov Source: DFM & Widrow (in prep)
21. Dan Foreman-Mackey CCPP@NYU dfm.github.com What is the Universe Made of? Atoms 4% Dark Matter 23% Dark Energy 73% Heavy Elements 0.03% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Source: http://apod.nasa.gov Source: DFM & Widrow (in prep)
22. Dan Foreman-Mackey CCPP@NYU dfm.github.com Credit: The Millennium Simulation Project
23. Dan Foreman-Mackey CCPP@NYU dfm.github.com Credit: The Millennium Simulation Project
24. Dan Foreman-Mackey CCPP@NYU dfm.github.com Credit: The Millennium Simulation Project
25. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy
26. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Credit: Jonathan Sick jonathansick.ca
27. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Credit: Jonathan Sick jonathansick.ca
28. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Credit: Jonathan Sick jonathansick.ca MegaCam: 340 MegaPixels
29. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy MegaCam: 340 MegaPixels Credit: NASA
30. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Imaging Source: NASA / ESA
31. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Imaging Source: NASA / ESA Spectroscopy
32. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Imaging Source: NASA / ESA Spectroscopy Spectroscopy Source: Riaud & Schneider (2007)
33. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy Imaging Source: NASA / ESA Spectroscopy Spectroscopy Source: Riaud & Schneider (2007)
34. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy is Open a lot of
35. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy is Open a lot of and there’s a lot of it!
36. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy is Open Hubble SDSS 2MASS 1990– 2000– 1997–2001 sdss.org archive.stsci.edu/hst www.ipac.caltech.edu/2mass Pan-STARRS LSST Planned GAIA
37. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy is Open Hubble SDSS 2MASS 1990– 2000– 1997–2001 sdss.org archive.stsci.edu/hst www.ipac.caltech.edu/2mass Pan-STARRS LSST Planned GAIA
38. Dan Foreman-Mackey CCPP@NYU dfm.github.com Data in Astronomy is Open Hubble SDSS 2MASS 1990– 2000– 1997–2001 sdss.org archive.stsci.edu/hst www.ipac.caltech.edu/2mass Pan-STARRS LSST Planned GAIA
39. Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in?
40. Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in?
41. Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in?
42. Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in?
43. Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in?
44. Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in? + Scientific Python Stack
45. Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in? + Scientific Python Stack
46. Dan Foreman-Mackey CCPP@NYU dfm.github.com Where does Python fit in? + Scientific Python Stack
47. Dan Foreman-Mackey CCPP@NYU dfm.github.com What about ? Easy Pythonic Flexible Scalable
48. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies
49. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82
50. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 728 SESAR ET AL. Ses S07 Labela Ntot A 84 B 144 C 54 D 8 E 11 F 11 G 10 H 7 I 4 J 26 K 8 L 3 M 5 Source: Sesar et al. (2010)
51. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82
52. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82
53. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 800k “Fields” ~ 12TB Imaging data > 1M “Target Stars”
54. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 Time Photons/Brightness
55. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)]
56. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)] Stars
57. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)] p const ⌘ M Y i =1 [(1 P bad )p good + P bad p bad ] p var ⌘ M Y i =1 [(1 P bad )p var , good + P bad p bad ] Stars
58. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)] p const ⌘ M Y i =1 [(1 P bad )p good + P bad p bad ] p var ⌘ M Y i =1 [(1 P bad )p var , good + P bad p bad ] Stars Runs Runs
59. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)] p const ⌘ M Y i =1 [(1 P bad )p good + P bad p bad ] p var ⌘ M Y i =1 [(1 P bad )p var , good + P bad p bad ] Stars Runs Runs p good ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ ) “Constant & Good” p var , good ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ + ⌃2 var ) “Variable & Good” pbad ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ + ⌃2 bad ) “Bad”
60. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)] p const ⌘ M Y i =1 [(1 P bad )p good + P bad p bad ] p var ⌘ M Y i =1 [(1 P bad )p var , good + P bad p bad ] Stars Runs Runs p good ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ ) “Constant & Good” p var , good ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ + ⌃2 var ) “Variable & Good” pbad ⌘ N(Ci↵ |f0 i f⇤ ↵ , 2 i↵ + 2 i↵ + ⌃2 bad ) “Bad” Npars = Nstars + Nruns + 6 ⇥ = {~ f0,~ f⇤, , ⌘, ⌃2 var , Pvar, ⌃2 bad , Pbad }
61. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82
62. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82
63. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 p(X|⇥) = N Y ↵ =1 [(1 P var )p const (X↵ |⇥) + P var p var (X↵ |⇥)]
64. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies SDSS Variable Stars in Stripe 82 f⇤ ( t ) = A0 + N X n=1 [ An sin( !t ) + Bn cos( !t )]
65. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies CFHT 4.7 Gigapixel mosaic of M31 Source: Jonathan Sick (Queen’s University)
66. Dan Foreman-Mackey CCPP@NYU dfm.github.com Source: Jonathan Sick (Queen’s University)
67. Dan Foreman-Mackey CCPP@NYU dfm.github.com Source: Jonathan Sick (Queen’s University)
68. Dan Foreman-Mackey CCPP@NYU dfm.github.com Source: Jonathan Sick (Queen’s University)
69. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies CFHT Source: Jonathan Sick (Queen’s University) MongoDB Persistent Metadata + GeoSpatial Indexing img1.fits img2.fits img4000.fits ... Flat-fielding Cosmic ray removal Sky subtraction Mosaic making ...
70. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Astrometry.net
71. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Astrometry.net
72. Dan Foreman-Mackey CCPP@NYU dfm.github.com
73. Dan Foreman-Mackey CCPP@NYU dfm.github.com
74. Dan Foreman-Mackey CCPP@NYU dfm.github.com
75. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes ~2500 JPGs Lang & Hogg (2011)
76. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes
77. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes
78. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes
79. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes Source: Lang & Hogg (2011)
80. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes
    
         
    
       Source: Lang & Hogg (2011)
81. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes
    
         
    
       Source: Lang & Hogg (2011) github.com/dfm/MarkovPy
82. Dan Foreman-Mackey CCPP@NYU dfm.github.com Case Studies Crowdsourcing Comet Holmes
83. Dan Foreman-Mackey CCPP@NYU dfm.github.com Number Crunching Data Management User Interaction And Much More... ( ) Growing Datasets
84. Dan Foreman-Mackey CCPP@NYU dfm.github.com Number Crunching Data Management User Interaction And Much More... ( ) Growing Datasets Easy!
85. Dan Foreman-Mackey CCPP@NYU dfm.github.com Number Crunching Data Management User Interaction And Much More... ( ) Growing Datasets Easy! Big Questions
86. Dan Foreman-Mackey CCPP@NYU dfm.github.com Dan Foreman-Mackey Center for Cosmology & Particle Physics (NYU) dfm.github.com @__dfm__