Scientific Computing in [Operational] Meteorology

A combination of things I've learned over the years. Content comes from the paper, "Best Practices in Scientific Computing" (http://arxiv.org/abs/1210.0530), Software Carpentry (https://software-carpentry.org), and my own trial-and-error experiences.

Transcript

1. Scientific  Computing  in   [Operational]  Meteorology Patrick  Marsh  (@pmarshwx) NOAA/NWS  Storm  Prediction  Center
2. Things  I  Wish  I  Had  Known   10  Years  Ago Patrick  Marsh  (@pmarshwx) NOAA/NWS  Storm  Prediction  Center
3. A  Little  About  Me • BS  in  Physics  &  Mathematics  from  University  of  Arkansas • MS/PhD  in  Meteorology  from  University  of  Oklahoma • NSSL  Liaison  to  the  Hazardous  Weather  Testbed (2009-­‐2013) • VORTEX2  Operations  Center  (2009-­‐2010) • NWS  Storm  Prediction  Center  (2013-­‐????)
4. A  Little  About  Me • BS  in  Physics  &  Mathematics  from  University  of  Arkansas • MS/PhD  in  Meteorology  from  University  of  Oklahoma • NSSL  Liaison  to  the  Hazardous  Weather  Testbed (2009-­‐2013) • VORTEX2  Operations  Center  (2009-­‐2010) • NWS  Storm  Prediction  Center  (2013-­‐????) Everything  in  yellow  required/requires  significant  programming  
5. “Always  code  as  if  the  person  who  ends   up  maintaining  your  code  is  a  violent   psychopath  who  knows  where  you  live.”
6. Motivation • Software  is  lab  equipment  for  the  21st  Century • Scientists  spend  a  lot  of  time  writing  it • But  over  90%  are  self-­‐taught • We  don't  know  what  good looks  like
7. What  follows  is  derived  from  the  paper, Best  Practices  for  Scientific  Computing http://arxiv.org/abs/1210.0530 and Software  Carpentry https://software-­‐carpentry.org and Some  Meteorology-­‐specific  advice   learned  through  trial  and  error
8. Write  Programs  for  People, not  Computers
9. Write  Programs  for  People, not  Computers Code  that  is  hard  to  understand   makes  it  hard  to  know  if  it  is  working  correctly.
10. Write  Programs  for  People, not  Computers Code  that  is  hard  to  understand is  hard  to  use,  even  for  your  future  self.
11. Write  Programs  for  People, not  Computers A  program  should  not  require  its  readers  to  hold  more  than  a   handful  of  facts  in  memory  at  once.
12. Write  Programs  for  People, not  Computers Make  names  consistent,  distinctive,  and  meaningful.
13. Write  Programs  for  People, not  Computers Make  code  style  and  formatting  consistent.
14. Let  the  Computer  Do  the  Work
15. Let  the  Computer  Do  the  Work Make  the  computer  repeat  tasks.
16. Let  the  Computer  Do  the  Work Save  recent  commands  in  a  file  for  re-­‐use.
17. Make  Incremental  Changes
18. Make  Incremental  Changes Work  in  small  steps.
19. Make  Incremental  Changes Use  version  control.
20. Don’t  Repeat  Yourself (or  others)
21. Don’t  Repeat  Yourself (or  others) Anything  repeated  in  two  or  more  places  will  eventually  be   wrong  in  at  least  one.
22. Don’t  Repeat  Yourself (or  others) Every  piece  of  data  must  have  a  single,  authoritative   representation.
23. Don’t  Repeat  Yourself (or  others) Modularize  code  rather  than  copying  and  pasting.
24. Don’t  Repeat  Yourself (or  others) Re-­‐use  code  instead  of  rewriting  it.
25. Don’t  Repeat  Yourself (or  others) Look  for  methods/solutions  outside  of  Meteorology.
26. Don’t  Repeat  Yourself (or  others) Look  for  methods/solutions  outside  of  Meteorology. Model  Grids  ==  Images Model  Grid  Cell  ==  Image  Pixel
27. Plan  for  Mitsakes
28. Plan  for  Mitsakes Add  assertions  to  programs  to  check  their  operation.
29. Plan  for  Mitsakes Learn  to  use  unit  testing.
30. Plan  for  Mitsakes Turn  bugs  into  cases  for  testing.
31. Optimize  Software   Only  After  it  Works  Correctly
32. Optimize  Software   Only  After  it  Works  Correctly Even  experts  find  it  hard  to  predict  performance  bottlenecks
33. Optimize  Software   Only  After  it  Works  Correctly Use  a  profiler  to  identify  bottlenecks.
34. Optimize  Software   Only  After  it  Works  Correctly Write  code  in  the  highest-­‐level  language  possible.
35. Know  Version  Control Learn  Version  Control Utilize  Version  Control
36. Utilize  Version  Control Put  everything  that  has  been  created  manually  in  version   control.
37. Utilize  Version  Control Leave  out  things  generated  by  the  computer.
38. Utilize  Version  Control Git,  and  more  directly,  GitHub fundamentally  changed  my   programming  practices.
39. Utilize  Version  Control On  more  than  one  occasion,  version  control  has  saved  a   project.
40. Learn  Color  Theory
41. Learn  Color  Theory How  do  you  created  graphics  for  those  with   color  sensitivities?
42. Learn  GIS
43. Learn  GIS Just  do  it...
44. Learn  a  Multitude  of (Meteorological)  Data  Formats  
45. Learn  a  Multitude  of (Meteorological)  Data  Formats   Science  is  becoming  more  collaborative.  Inevitably,  your   collaborator  will  not  use  the  same  data  format  you  use.
46. Learn  a  Multitude  of (Meteorological)  Data  Formats   Research  models  (typically)  output  netcdf;   NWS  AWIPS  uses  grib2;  NWS  AWIPS  2  uses  HDF5; NWS  SPC  uses  GEMPAK; Harold  Brooks  uses  Microsoft  Excel.
47. It  Will  Always  Take  Longer  Than  You  Think
48. It  Will  Always  Take  Longer  Than  You  Think Hofstadter's  Law:  It  always  takes  longer  than  you  expect,  even   when  you  take  into  account  Hofstadter's  Law.
49. Don’t  Be  Afraid  to  Fail
50. Don’t  Be  Afraid  to  Fail Your  first  choice  is  rarely  the  optimal  choice…
51. Don’t  Be  Afraid  to  Fail Like  this  presentation.
52. Software  is  Never  Finished
53. Software  is  Never  Finished It  is  merely  released.