Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Updates and plans for more agile climate model ...

Julius Busecke
June 22, 2024
14

Updates and plans for more agile climate model analysis in the cloud

Slides for "2024-06-06_Updates and plans for more agile climate model analysis in the cloud" presented at m2lines Annual Meeting 2024 on June 06, 2024 by Julius Busecke.

Julius Busecke

June 22, 2024
Tweet

More Decks by Julius Busecke

Transcript

  1. @M2LInES https://m2lines.github.io The future - finite times Climate Process Team:

    “Ocean Transport & Eddy Energy” to implement, assess, improve, & unify recent work on energetically-consistent ocean eddy momentum & tracer parametrizations in ocean-only and coupled climate models to improve model fidelity. https://ocean-eddy-cpt.github.io/ Postdoctoral positions available: Climate Process Team on Ocean Transport and Eddy Energy Funded by the National Oceanographic and Atmospheric Administration and the National Science Foundat Multiple postdoctoral research positions are available as part of a multi-institution Climate Pro Team (CPT) on Ocean Transport and Eddy Energy. The CPT aims to survey, improve, and unify advances in energy-, flow-, and scale-aware parameterizations of mesoscale eddies, in process stu and global ocean models; constrain parameters and parameterized fluxes through a synthes up-to-date observations of ocean energetics and transport; and implement and assess schemes w IPCC-class climate models at NCAR, NOAA-GFDL, and DOE-LANL. The expectation is modernized, energetically-consistent mesoscale eddy parameterizations will significantly re climate model biases in ocean currents, stratification, and transport. • New York University (Supervised by Laure Zanna): Unification of buoyancy and tracer closu Assessment and parameterization of vertical energy structure; Parameterization of the grey zone. More information and application at ​https://apply.interfolio.com/68119​. • University of Colorado, Boulder (Supervised by Ian Grooms): Assessment of 2D eddy energ equation; parameterization of eddy energy transport; parameterizing dissipation in the eddy energy equation. More information and application are at https://jobs.colorado.edu/jobs/JobDetail/?jobId=20799​. • Woods Hole Oceanographic Institution (Supervised by Sylvia Cole): Characterizing scale-dependent EKE from observations; quasi-3D eddy buoyancy and momentum statistics from observations; analysis of vertical eddy structure in observations; synthesis of observati More information and application are at https://careers.whoi.edu/opportunities/view-all-openings/science-research/​ (position 19-08-0 • Princeton University (Supervised by Alistair Adcroft): Implementation and assessment of ext parameterizations of mesoscale eddies in process, idealized and global ocean models; consistent and optimized formulation of closures; development and assessment of improved and unified closures; evaluation of new closures in climate models. More information and application at ​https://www.princeton.edu/acad-positions/position/13701​. Applications must include a cover letter, a CV with a list of publications, a statement of rese interests, and contact information of 3 references. Applicants wishing to be considered for positio multiple institutions should indicate this in their cover letter, and submit a separate application to position of interest. For more information email Laure Zanna (​[email protected]​), or any o collaborating PIs listed above. Funded by the National Oceanographic and Atmospheric Administration and the National Science Foundation Multiple postdoctoral research positions are available as part of a multi-institution Climate Process Team (CPT) on Ocean Transport and Eddy Energy. The CPT aims to survey, improve, and unify new advances in energy-, flow-, and scale-aware parameterizations of mesoscale eddies, in process studies and global ocean models; constrain parameters and parameterized fluxes through a synthesis of up-to-date observations of ocean energetics and transport; and implement and assess schemes within IPCC-class climate models at NCAR, NOAA-GFDL, and DOE-LANL. The expectation is that modernized, energetically-consistent mesoscale eddy parameterizations will significantly reduce climate model biases in ocean currents, stratification, and transport. • New York University (Supervised by Laure Zanna): Unification of buoyancy and tracer closures; Assessment and parameterization of vertical energy structure; Parameterization of the grey zone. More information and application at ​https://apply.interfolio.com/68119​. • University of Colorado, Boulder (Supervised by Ian Grooms): Assessment of 2D eddy energy equation; parameterization of eddy energy transport; parameterizing dissipation in the eddy energy equation. More information and application are at https://jobs.colorado.edu/jobs/JobDetail/?jobId=20799​. • Woods Hole Oceanographic Institution (Supervised by Sylvia Cole): Characterizing scale-dependent EKE from observations; quasi-3D eddy buoyancy and momentum statistics from observations; analysis of vertical eddy structure in observations; synthesis of observations. More information and application are at https://careers.whoi.edu/opportunities/view-all-openings/science-research/​ (position 19-08-09). • Princeton University (Supervised by Alistair Adcroft): Implementation and assessment of extant parameterizations of mesoscale eddies in process, idealized and global ocean models; consistent and optimized formulation of closures; development and assessment of improved and unified closures; evaluation of new closures in climate models. More information and application at ​https://www.princeton.edu/acad-positions/position/13701​. Applications must include a cover letter, a CV with a list of publications, a statement of research interests, and contact information of 3 references. Applicants wishing to be considered for positions in multiple institutions should indicate this in their cover letter, and submit a separate application to each position of interest. For more information email Laure Zanna (​[email protected]​), or any of the collaborating PIs listed above. Julius Busecke June 6 Updates and Plans for more agile climate model analysis in the cloud 1
  2. (Open) Science does not have to be slow - Agile

    Science 2 Idea 💡 Result ✅
  3. (Open) Science does not have to be slow - Agile

    Science 2 Idea 💡 Result ✅ Tech/Infrastructure limited Understanding limited
  4. (Open) Science does not have to be slow - Agile

    Science 2 Idea 💡 Result ✅ Tech/Infrastructure limited Understanding limited • Open/Fast Access to data • Scalable Compute • Community OSS tools • Reproducible Results • Hire Engineers?
  5. (Open) Science does not have to be slow - Agile

    Science 2 Idea 💡 Result ✅ Tech/Infrastructure limited Understanding limited • Open/Fast Access to data • Scalable Compute • Community OSS tools • Reproducible Results • Hire Engineers? • Collaboration
  6. LEAP-Pangeo • Shared cloud based infrastructure between LEAP and m2lines

    • Built with Open Source Tools by 2i2c on top of commodity cloud resources (Google) • Everyone at m2lines can get access • Modular design: Use only what you need 3 https://leap-stc.github.io/intro.html
  7. Analysis-Ready Cloud-Optimized (ARCO) data Analysis-Ready: • Think in “Datasets” not

    “data f iles” • No need for tedious homogenizing / cleaning steps • Curated and cataloged Chunked appropriately for analysis Rich metadata Everything in one dataset object
  8. Analysis-Ready Cloud-Optimized (ARCO) data Analysis-Ready: • Think in “Datasets” not

    “data f iles” • No need for tedious homogenizing / cleaning steps • Curated and cataloged Cloud Optimized: • Compatible with object storage 
 (access via HTTP) • Supports lazy access and intelligent subsetting • Integrates with high-level analysis libraries and distributed frameworks
  9. Scalable Interactive Compute 6 • JupyterHub Interface • Copy and

    paste from the catalog for fast startup • Standard Software Environments • Share your code with collaborators without setup needed
  10. Scalable Interactive Compute 6 • JupyterHub Interface • Copy and

    paste from the catalog for fast startup • Standard Software Environments • Share your code with collaborators without setup needed • Dask Gateway for scalable compute • Ideal for data analytics on many model runs with new parametrization 😋
  11. What is new? • Fully customizable software environment 7 Build

    your fully customized docker image! Or use speci fi c versions of the pangeo images
  12. What is new? • Fully customizable software environment • LEAP-Pangeo

    Catalog (URL subject to change) 7 catalog.leap.carbonplan.org
  13. What is new? • Fully customizable software environment • LEAP-Pangeo

    Catalog (URL subject to change) • Pangeo-Forge Based Data Ingestion Pipelines 7 Domain Expertise: 
 How to fi nd, clean, and homogenize data 🤓 Data Scientist
  14. What is new? • Fully customizable software environment • LEAP-Pangeo

    Catalog (URL subject to change) • Pangeo-Forge Based Data Ingestion Pipelines • Example: Pangeo CMIP6 Cloud data 7
  15. Open Science beyond single communities 8 🧑💻 👩💻 👩💻 👨💻

    🧑💻 🖥 💽 👩💻 👩💻 👨💻 🧑💻 🖥 👩💻 👩💻 👨💻 🧑💻 🖥 💽 😀 🧑💻 💽 💽 Data maintained by the community but publicly accessible from outside Data maintained by the community but only accessible from community resources
  16. Open Science beyond single communities 9 🧑💻 👩💻 👩💻 👨💻

    🧑💻 🖥 💽 👩💻 👩💻 👨💻 🧑💻 🖥 👩💻 👩💻 👨💻 🧑💻 🖥 💽 🧑💻 💽 💽 Data maintained by the community but publicly accessible from outside Data maintained by the community but only accessible from community resources 😡 🚫
  17. Open Science beyond single communities 10 🧑💻 👩💻 👩💻 👨💻

    🧑💻 🖥 💽 👩💻 👩💻 👨💻 🧑💻 🖥 👩💻 👩💻 👨💻 🧑💻 🖥 💽 🧑💻 💽 💽 Data maintained by the community but publicly accessible from outside Data maintained by the community but only accessible from community resources 💽 💵⏳
  18. Open Science beyond single communities 11 👩💻 👩💻 👨💻 🧑💻

    🖥 💽 👩💻 👩💻 👨💻 🧑💻 🖥 👩💻 👩💻 👨💻 🧑💻 🖥 💽 🧑💻 💽 💽 Data maintained by the community but publicly accessible from outside Data maintained by the community but only accessible from community resources 😀
  19. Open Science beyond single communities 12 👩💻 👩💻 👨💻 🧑💻

    🖥 💽 👩💻 👩💻 👨💻 🧑💻 🖥 👩💻 👩💻 👨💻 🧑💻 🖥 💽 🧑💻 💽 💽 Data maintained by the community but publicly accessible from outside Data maintained by the community but only accessible from community resources 😀
  20. A brave new science world 13 👩💻 👩💻 👨💻 🧑💻

    🖥 💽 👩💻 👩💻 👨💻 🧑💻 🖥 👩💻 👩💻 👨💻 🧑💻 🖥 💽 🧑💻 💽 💽 😀 😀 😀
  21. Actual Science produced on LEAP Pangeo 14 Have you used

    LEAP-Pangeo for a publication? Please let me know.
  22. Coming Soon 15 1PB public storage no egress operated with

    100% non- ff energy authenticated access available Public Storage based on AWS S3 Exploring ingestion at the moment Fully Public Storage Options Globus Pangeo-Forge Recipes
  23. How to get started 17 Need Data? Lets get it

    into the cloud! https://github.com/leap-stc/data-management https://github.com/leap-stc/cmip6-leap-feedstock Check out the Hub! leap.2i2c.cloud Apply for LEAP membership: https://leap.columbia.edu/research-home/leap-pangeo/
  24. Tell me how to accelerate your amazing work! • I

    am always happy to chat about science and technical challenges (and successes! ) • There are many things to work on. What do you think is the most important/ urgent? 18
  25. I ❤ Feedback + Questions 19 @JuliusBusecke jbusecke [email protected] juliusbusecke.com

    @JuliusBusecke @JuliusBusecke @[email protected] @codeandcurrents.bsky.social Made on LEAP-Pangeo ☝
  26. I ❤ Feedback + Questions 19 @JuliusBusecke jbusecke [email protected] juliusbusecke.com

    @JuliusBusecke @JuliusBusecke @[email protected] @codeandcurrents.bsky.social Made on LEAP-Pangeo ☝