Empowering environmental science with open tools & open data

The Met Office is a world leading weather and climate centre with the largest operational supercomputer in Europe. With data growing rapidly every year we need to change our approach to analysing it.
This talk covers how we are working with a suite of open source tools to solve this problem. We have mixed popular projects like Jupyter Notebooks with new data processing libraries like Dask and our own bespoke weather analysis tools like Iris. We are also opening large volumes of data to the public, which through these tools can be manipulated quickly and easily from the comfort of a web browser.

Transcript

1. Empowering environmental science with open tools & open data Jacob Tomlinson Lead Engineer Met Office Informatics Lab
2. .@_jacobtomlinson. .linux. .bash. .cloud. .sysadmin. .web development. .docker. .javascript. .git. .public speaking. .CI/CD. .automation. .python.
7. Generating and moving data
9. Data is heavy
11. .Currently storing 40PB. .Archiving 200TB per day. .Will store 300PB by 2020. *Approx figures - early 2017
13. Does that make us unusual?
14. .Latitude. .Longitude.
15. .Latitude. .Longitude. .Altitude.
16. .Latitude. .Longitude. .Altitude. .Time.
17. .Latitude. .Longitude. .Altitude. .Time. .Variable.
18. .Latitude. .Longitude. .Altitude. .Time. .Variable. .Ensemble member.
19. .Latitude. .Longitude. .Altitude. .Time. .Variable. .Ensemble member. .Run.
21. Wikipedia - Z-order curve
22. Sounds like we are unusual...
23. ...but perhaps not
24. Neurobiology Product Design Medical Imaging Civil Engineering Astronomy Aeronautics
26. Processing data at scale
29. Iris
31. Laziness
35. Iris
42. Summary
43. .We generate lots of data. .We open up as much as we can. .We use open tools to analyse it..
44. Thank you Jacob Tomlinson www.informaticslab.co.uk @_jacobtomlinson