Unlocking the Universe with Python and LSST

Invited talk from the 2013 RuPy conference, Budapest, Hungary.

1. Unlocking  the  Universe  with   Python  and  LSST   Jake  VanderPlas   RuPy,  Oct  11th  2013  
2. Then:  Edwin  Hubble  at   Mount  Wilson  Observatory   1990s  -­‐  Now:  the  Hubble  Space   Telescope  in  Low  Earth  Orbit…  
3. Today:  Survey  Astronomy  –   Sloan  Digital  Sky  Survey   SDSS  Telescope:  Apache  Point  NM  
4. Large   SynopQc   Survey   Telescope  
5. Large   SynopQc   Survey   Telescope   LSST in a sentence: An optical/near-IR multi-color survey of half the sky, based on ~1000 visits over a 10-year period
7. 8.4  meter  primary  mirror   3,000  Megapixel  CCD  camera   Field  of  View:  9.5  deg2   Largest  digital  camera  in  the   world!  
8. LSST:  a    10-­‐year,  full-­‐sky  digital  color  movie   -­‐  Full  southern  sky  every  3-­‐4  nights   -­‐  30,000  GB/night  data  stream   -­‐  Real-­‐Qme  transient  alerts   -­‐  Final  Catalog:  100s  of  Petabytes  
9. •  Adler  Planetarium   •  Argonne  NaQonal  Laboratory   •  Brookhaven  NaQonal  Laboratory  (BNL)     •  California  InsQtute  of  Technology   •  Carnegie  Mellon  University   •  Chile   •  Cornell  University   •  Drexel  University   •  Fermi  NaQonal  Accelerator  Laboratory     •  George  Mason  University   •  Google,  Inc.   •  Harvard-­‐Smithsonian  Center  for  Astrophysics   •  InsQtut  de  Physique  Nucleaire  et  de  Physique   des  ParQcules  (IN2P3)   •  Johns  Hopkins  University   •  Kavli  InsQtute  for  ParQcle  Astrophysics  and   Cosmology  (KIPAC)  -­‐  Stanford  University   •  Las  Cumbres  Observatory  Global  Telescope   Network,  Inc.   •  Lawrence  Livermore  NaQonal  Laboratory  (LLNL)   •  Los  Alamos  NaQonal  Laboratory  (LANL)   •  NaQonal  OpQcal  Astronomy  Observatory   •  NaQonal  Radio  Astronomy  Observatory   •  Northwestern  University   •  Princeton  University   •  Purdue  University   •  Research  CorporaQon  for  Science  Advancement   •  Rutgers  University   •  SLAC  NaQonal  Accelerator  Laboratory   •  Space  Telescope  Science  InsQtute   •  Texas  A  &  M  University   •  The  InsQtute  of  Physics  of  the  Academy  of   Sciences  of  the  Czech  Republic   •  The  Pennsylvania  State  University   •  The  University  of  Arizona   •  University  of  California  at  Davis   •  University  of  California  at  Irvine   •  University  of  Illinois  at  Urbana-­‐Champaign   •  University  of  Michigan   •  University  of  Pennsylvania   •  University  of  Piesburgh   •  University  of  Washington   •  Vanderbilt  and  Fisk  UniversiQes   LSST  Ins(tu(onal  Members:  
10. •  Adler  Planetarium   •  Argonne  NaQonal  Laboratory   •  Brookhaven  NaQonal  Laboratory  (BNL)     •  California  InsQtute  of  Technology   •  Carnegie  Mellon  University   •  Chile   •  Cornell  University   •  Drexel  University   •  Fermi  NaQonal  Accelerator  Laboratory     •  George  Mason  University   •  Google,  Inc.   •  Harvard-­‐Smithsonian  Center  for  Astrophysics   •  InsQtut  de  Physique  Nucleaire  et  de  Physique   des  ParQcules  (IN2P3)   •  Johns  Hopkins  University   •  Kavli  InsQtute  for  ParQcle  Astrophysics  and   Cosmology  (KIPAC)  -­‐  Stanford  University   •  Las  Cumbres  Observatory  Global  Telescope   Network,  Inc.   •  Lawrence  Livermore  NaQonal  Laboratory  (LLNL)   •  Los  Alamos  NaQonal  Laboratory  (LANL)   •  NaQonal  OpQcal  Astronomy  Observatory   •  NaQonal  Radio  Astronomy  Observatory   •  Northwestern  University   •  Princeton  University   •  Purdue  University   •  Research  CorporaQon  for  Science  Advancement   •  Rutgers  University   •  SLAC  NaQonal  Accelerator  Laboratory   •  Space  Telescope  Science  InsQtute   •  Texas  A  &  M  University   •  The  InsQtute  of  Physics  of  the  Academy  of   Sciences  of  the  Czech  Republic   •  The  Pennsylvania  State  University   •  The  University  of  Arizona   •  University  of  California  at  Davis   •  University  of  California  at  Irvine   •  University  of  Illinois  at  Urbana-­‐Champaign   •  University  of  Michigan   •  University  of  Pennsylvania   •  University  of  Piesburgh   •  University  of  Washington   •  Vanderbilt  and  Fisk  UniversiQes   LSST  Ins(tu(onal  Members:  
11. •  Adler  Planetarium   •  Argonne  NaQonal  Laboratory   •  Brookhaven  NaQonal  Laboratory  (BNL)     •  California  InsQtute  of  Technology   •  Carnegie  Mellon  University   •  Chile   •  Cornell  University   •  Drexel  University   •  Fermi  NaQonal  Accelerator  Laboratory     •  George  Mason  University   •  Google,  Inc.   •  Harvard-­‐Smithsonian  Center  for  Astrophysics   •  InsQtut  de  Physique  Nucleaire  et  de  Physique   des  ParQcules  (IN2P3)   •  Johns  Hopkins  University   •  Kavli  InsQtute  for  ParQcle  Astrophysics  and   Cosmology  (KIPAC)  -­‐  Stanford  University   •  Las  Cumbres  Observatory  Global  Telescope   Network,  Inc.   •  Lawrence  Livermore  NaQonal  Laboratory  (LLNL)   •  Los  Alamos  NaQonal  Laboratory  (LANL)   •  NaQonal  OpQcal  Astronomy  Observatory   •  NaQonal  Radio  Astronomy  Observatory   •  Northwestern  University   •  Princeton  University   •  Purdue  University   •  Research  CorporaQon  for  Science  Advancement   •  Rutgers  University   •  SLAC  NaQonal  Accelerator  Laboratory   •  Space  Telescope  Science  InsQtute   •  Texas  A  &  M  University   •  The  InsQtute  of  Physics  of  the  Academy  of   Sciences  of  the  Czech  Republic   •  The  Pennsylvania  State  University   •  The  University  of  Arizona   •  University  of  California  at  Davis   •  University  of  California  at  Irvine   •  University  of  Illinois  at  Urbana-­‐Champaign   •  University  of  Michigan   •  University  of  Pennsylvania   •  University  of  Piesburgh   •  University  of  Washington   •  Vanderbilt  and  Fisk  UniversiQes   LSST  Ins(tu(onal  Members:  
12. How  do  we  prepare  the   Astronomy  community   for  this  data  deluge?  
13. Credit:  John  Peterson   LSST  Image  SimulaQon   (Video  of  SimulaQon)  
14. 0.2”   OpQcal  Model                                          +Tracking                                                            +DiffracQon                            +Detector  PerturbaQons                 +Lens  PerturbaQons                  +Mirror  PerturbaQons                    +Detector                                                        +Dome  Seeing                 +Low  AlQtude                                          +Mid  AlQtude                                              +High  AlQtude                                        +PixelizaQon        Atmosphere                                                Atmosphere                                                Atmosphere   Peterson  et  al  2013  
15. ImSim  DescripQon   15  
16. LSST  SimulaQons:  Modeling  Everything  
17. Key  Component:  Difference  Imaging   OpQcal  Burst  in  the  Deep  Lens  Survey   hep://www.lsst.org/lsst/public/transient  
18. Key  Component:  Difference  Imaging   OpQcal  Burst  in  the  Deep  Lens  Survey   hep://www.lsst.org/lsst/public/transient   2  million  events  per  night,   ~500,000  are  random  fluctuaQons   several  thousand  are  “interesQng”  
19. ~  100  Supernovae  used  to  discover  dark  energy     By  contrast,  LSST  will  observe  millions   Supernova  in  Pinwheel  galaxy  (BJ  Fulton,  The  Guardian)  
20. LSST  and  Cosmology:   Supernova:  measure  of  the   rate  of  expansion   GravitaQonal  Lensing:   measure  of  geometry   Baryon  OscillaQons:  trace   of  primordial  fluctuaQons   Clustering:  measure  of  the   geometry  and  structure  growth  
21. LSST  and  The  Transient  Universe:   TransiQng  Planets   Supernovae   Gamma  Ray  Bursts   Quasars/AGN   Stellar  flares  &   Variability   Asteroids  &   Comets  
22. Rest  et  al  2005;  courtesey  of  A.  Becker   Large  Magellanic  Cloud  (CTIO)  
23. 2001  -­‐  2001  
24. 2001  -­‐  2002  
25. 2001  -­‐  2003  
26. 2001  -­‐  2004  
27. 2001  -­‐  2004   “Light  Echo”  from  the   famous  Supernova  1987A!  
28. Image  Differencing:  not  as  easy  as  it  sounds…   Rotate  &  align  with   “science  image”   Detect  sources   Model  point-­‐spread   funcQon   De-­‐convolve  &  subtract   –   =  
30. Base  ComputaQon  Layer:  C++  Object  Model.   User  Interface:  Python  Object  Model.   SWIG  binding  layer   Pipeline  Language  Model  
31. Why  Python?   Open  source  (free!)  &  cross-­‐plaqorm   Easy  wrapping  of  compiled  legacy  code   Dynamic,  InteracQve,  Easy  to  learn  and  use   (even  for  scienQsts!)  
32. Excellent  core  set  of  scienQfic  packages:   NumPy:  Array-­‐based  operaQons   SciPy:  ScienQfic  Python   Matplotlib:  PublicaQon-­‐quality  plosng   IPython:  InteracQve  CompuQng  
33. Excellent  core  set  of  scienQfic  packages:  
35. My  Soap-­‐box:  Reproducibility  
36. My  Soap-­‐box:  Reproducibility   Infamous  example:  Reinhart-­‐Rogoff  paper  
37. Infamous  example:  Reinhart-­‐Rogoff  paper   My  Soap-­‐box:  Reproducibility  
38. My  Soap-­‐box:  Reproducibility   Infamous  example:  Reinhart-­‐Rogoff  paper  
39. My  Soap-­‐box:  Reproducibility   Infamous  example:  Reinhart-­‐Rogoff  paper   heps://github.com/vincentarelbundock/Reinhart-­‐Rogoff  
40. If  it’s  not  reproducible,   it’s  not  science!  
41. hep://www.astroML.org   Our  Effort:  AstroML  –  Python  machine  learning  package  
42. hep://www.astroML.org   Our  Effort:  AstroML  –  Python  machine  learning  package  
43. hep://www.astroML.org   Our  Effort:  AstroML  –  Python  machine  learning  package  
44. hep://www.astroML.org   Our  Effort:  AstroML  –  Python  machine  learning  package  
45. Our  Effort:  AstroML  –  Python  machine  learning  package   hep://www.astroML.org   200+  Examples  of  real  analysis  on  real  data!   Already  several  papers  with  reproducible   analysis  through  AstroML  
46. •  LSST:  a  decade-­‐long  movie  of  the  enQre  southern   sky,  which  will  change  the  way  Astronomers  do  their   science.   •  Python:  quickly  becoming  the  tool  of  choice  for   much  data-­‐driven  research.   •  Reproducibility:  open-­‐source  tools  like  Python  &   IPython  notebook  will  be  vital  as  large  datasets   become  more  and  more  central  to  research.  
47. Email:  jakevdp@cs.washington.edu     Twieer:  @jakevdp     Github:  jakevdp     Web:  hep://www.vanderplas.com     Blog:  hep://jakevdp.github.io