LLNL Open Source - Speaker Deck

Slide 1

Slide 1 text

LLNL-PRES-752119 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC LLNL Open Source Univ. of California – Santa Cruz Ian Lee Lawrence Livermore National Laboratory February 19, 2019

Slide 2

Slide 2 text

LLNL-PRES-752119 2 software.llnl.gov https://www.ted.com/talks/simon_sinek_why_good_leaders_make_you_feel_safe

Slide 3

Slide 3 text

LLNL-PRES-752119 3 software.llnl.gov Django Con 2015 – Opening Keynote https://pbs.twimg.com/profile_images/444296717446287360/VGK7qhQa_400x400.jpeg https://upload.wikimedia.org/wikipedia/commons/thumb/5/50/18F_logo.svg/2000px-18F_logo.svg.png https://2015.djangocon.us/site_media/static/images/logo.png

Slide 4

Slide 4 text

LLNL-PRES-752119 4 software.llnl.gov Why am I passionate about Open Source? Leave things better than you found them.

Slide 5

Slide 5 text

LLNL-PRES-752119 5 software.llnl.gov

Slide 6

Slide 6 text

LLNL-PRES-752119 6 software.llnl.gov

Slide 7

Slide 7 text

LLNL-PRES-752119 7 software.llnl.gov

Slide 8

Slide 8 text

LLNL-PRES-752119 8 software.llnl.gov https://upload.wikimedia.org/wikipedia/commons/a/a8/U.S._National_labs_map.jpg

Slide 9

Slide 9 text

LLNL-PRES-752119 9 software.llnl.gov http://www.ex-astris-scientia.org/articles/new_enterprise/enterprise-warpcore.jpg

Slide 10

Slide 10 text

LLNL-PRES-752119 10 software.llnl.gov 1960s 1970s 1980s 1990s 2000s 2010s Pioneering simulations of particle tracking CDC 3600 CDC 7600 Ozone mixing models CRAY 1 ASCI Blue- Pacific Helping the medical community plan radiation treatment Unprecedented dislocation dynamics simulations BlueGene Breakthrough visualizations of mixing fluids Dynamics in three dimensions Global climate modeling Detailed predictions of ecosystems Petascale and exascale computing

Slide 11

Slide 11 text

LLNL-PRES-752119 11 software.llnl.gov § 3 out of 16 #1 systems over last 20 years Top500.org ASCI White Nov 2000 – Nov 2001 BlueGene/L Nov 2004 – Nov 2007 Sequoia June 2012 https://www.top500.org/resources/top-systems/

Slide 12

Slide 12 text

LLNL-PRES-752119 12 software.llnl.gov Sierra

Slide 13

Slide 13 text

LLNL-PRES-752119 13 software.llnl.gov TOSS – Tri-Lab Operating System Software § Built on Red Hat Enterprise Linux — Not an HPC distribution § Adds LLNL developed additions and patches to support HPC — Low Latency Interconnect: Infiniband — Parallel File System: Lustre — Resource Manager: SLURM § Work closely with open communities Components not in TOSS Supported Linux Commodity Hardware Platform Kernel, Infiniband, Message Passing Interface Batch Scheduler (MOAB) User Environment Lustre File Systems Compiler & Development Tools Resource Manager (SLURM) TOSS Components HPSS Hopper LLNL-PRES-550311 TOSS is a software stack for HPC – large, interconnected clusters!

Slide 14

Slide 14 text

LLNL-PRES-752119 14 software.llnl.gov § Began as simple resource manager — Now scalable to 1.6M+ cores (sequoia) § Launch and manage parallel jobs — Large, parallel jobs, often MPI § Queuing and scheduling of jobs — Much more work than resources http://slurm.schedmd.com http://slurm.schedmd.com http://www.ibm.com/developerworks/library/l-slurm-utility/figure3.gif

Slide 15

Slide 15 text

LLNL-PRES-752119 15 software.llnl.gov § Family of projects used to build site-customized resource management systems § flux-core — Implements the communication layer and lowest level services and interfaces § flux-sched — Consists of an engine that handles all the functionality common to scheduling § capacitor — A bulk execution manager using flux-core, handles running and monitoring 1000’s of jobs http://flux-framework.github.io

Slide 16

Slide 16 text

LLNL-PRES-752119 16 software.llnl.gov ZFS on Linux § ZFS is an open source filesystem and volume manager designed to address the limitations of existing storage solutions § 2011: Available for Linux § Ten LLNL filesystems, totaling ~ 100PB § Ships in Ubuntu 16.04 https://zfsonlinux.org

Slide 17

Slide 17 text

LLNL-PRES-752119 17 software.llnl.gov § Handles combinatorial explosion of ABI-incompatible packages § All versions coexist, binaries work regardless of user’s environment § Familiar syntax, reminiscent of brew, yum, etc $ spack install mpileaks unconstrained $ spack install [email protected] @ custom version $ spack install [email protected] %[email protected] % custom compiler $ spack install [email protected] %[email protected] +threads +/- build option $ spack install [email protected] os=SuSE11 os= $ spack install [email protected] os=CNL10 os= $ spack install [email protected] os=CNL10 target=haswell target= SPACK https://spack.io

Slide 18

Slide 18 text

LLNL-PRES-752119 18 software.llnl.gov § Manages the first-ever decentralized database for handling climate science data § Multiple petabytes of data at dozens of federated sites worldwide § International collaboration for the software that powers most global climate change research https://github.com/ESGF https://esgf.llnl.gov

Slide 19

Slide 19 text

LLNL-PRES-752119 19 software.llnl.gov VisIt § Originally developed to visualize and analyze the results of terascale simulations § Interactive, scalable, visualization, animation and analysis tool § Powerful, easy to use GUI § Distributed and parallel architecture allows handling extremely large data sets interactively https://visit.llnl.gov

Slide 20

Slide 20 text

LLNL-PRES-752119 20 software.llnl.gov https://computation.llnl.gov/casc

Slide 21

Slide 21 text

LLNL-PRES-752119 21 software.llnl.gov

Slide 22

Slide 22 text

LLNL-PRES-752119 22 software.llnl.gov https://software.llnl.gov

Slide 23

Slide 23 text

LLNL-PRES-752119 23 software.llnl.gov LLNL Open Source Presence https://software.llnl.gov/explore 475

Slide 24

Slide 24 text

LLNL-PRES-752119 24 software.llnl.gov LLNL Open Source Engagement https://software.llnl.gov/explore

Slide 25

Slide 25 text

LLNL-PRES-752119 25 software.llnl.gov

Slide 26

Slide 26 text

LLNL-PRES-752119 26 software.llnl.gov Science & Technology Review “Our large collection of software is a precious Laboratory asset, one that benefits both Lawrence Livermore, and in many cases, the public at large.” - Bruce Hendrickson Associate Director, Computation https://str.llnl.gov/2018-01/comjan18

Slide 27

Slide 27 text

LLNL-PRES-752119 27 software.llnl.gov https://www.exascaleproject.org/more-on-the-software-that-underpins-the-exascale-computing-project/

Slide 28

Slide 28 text

LLNL-PRES-752119 28 software.llnl.gov https://osti.gov/doecode

Slide 29

Slide 29 text

LLNL-PRES-752119 29 software.llnl.gov § “Federal Source Code Policy: Achieving Efficiency, Transparency, and Innovation through Reuseable and Open Source Software” — “Agencies shall make custom-developed code available for Government-wide reuse and make their code inventories discoverable at https://www.code.gov (“Code.gov”) […]” — “[…] establishes a pilot program that requires agencies, when commissioning new custom software, to release at least 20 percent of new custom-developed code as Open Source Software (OSS) […]” Federal Source Code Policy https://sourcecode.cio.gov https://code.gov & https://sourcecode.cio.gov

Slide 30

Slide 30 text

LLNL-PRES-752119 30 software.llnl.gov https://code.gov

Slide 31

Slide 31 text

LLNL-PRES-752119 31 software.llnl.gov Public US Government GitHub Data Scrape § 252 US Government Orgs — U.S. Federal (137) — U.S. Military and Intelligence (12) — U.S. Research Labs (103) § 8716 Open Source Repositories https://github.com/LLNL/scraper/pull/3 LLNL 5% Other US Government 95%

Slide 32

Slide 32 text

LLNL-PRES-752119 32 software.llnl.gov US Government Organizations on GitHub https://government.github.com/community/

Slide 33

Slide 33 text

LLNL-PRES-752119 33 software.llnl.gov OMB M-15-13: Require Secure Connections https://www.whitehouse.gov/sites/default/files/omb/memoranda/2015/m-15-13.pdf

Slide 34

Slide 34 text

LLNL-PRES-752119 34 software.llnl.gov https://https.cio.gov

Slide 35

Slide 35 text

LLNL-PRES-752119 35 software.llnl.gov 35 https://18f.gsa.gov/2015/03/17/for-public-comment-the-https-only-standard/

Slide 36

Slide 36 text

LLNL-PRES-752119 36 software.llnl.gov https://github.com/gsa/https/issues?utf8=✓&q=label%3A%22Public+Comment%22

Slide 37

Slide 37 text

LLNL-PRES-752119 37 software.llnl.gov https://github.com/GSA/https/pull/108

Slide 38

Slide 38 text

LLNL-PRES-752119 38 software.llnl.gov https://18f.gsa.gov/2017/01/06/open-source-collaboration-across-agencies-to-improve-https-deployment/

Slide 39

Slide 39 text

LLNL-PRES-752119 39 software.llnl.gov https://github.com/dhs-ncats/pshtt

Slide 40

Slide 40 text

LLNL-PRES-752119 40 software.llnl.gov https://18f.gsa.gov/2017/01/06/open-source-collaboration-across-agencies-to-improve-https-deployment/

Slide 41

Slide 41 text

LLNL-PRES-752119 41 software.llnl.gov https://18f.gsa.gov/2017/01/06/open-source-collaboration-across-agencies-to-improve-https-deployment/

Slide 42

Slide 42 text

LLNL-PRES-752119 42 software.llnl.gov https://18f.gsa.gov/2017/01/06/open-source-collaboration-across-agencies-to-improve-https-deployment/

Slide 43

Slide 43 text

LLNL-PRES-752119 43 software.llnl.gov https://18f.gsa.gov/2017/05/25/from-launch-to-landing-how-nasa-took-control-of-its-https-mission/

Slide 44

Slide 44 text

LLNL-PRES-752119 44 software.llnl.gov https://pulse.cio.gov

Slide 45

Slide 45 text

LLNL-PRES-752119 45 software.llnl.gov https://pulse.cio.gov/https/agencies/

Slide 46

Slide 46 text

LLNL-PRES-752119 46 software.llnl.gov https://pulse.cio.gov/https/agencies/

Slide 47

Slide 47 text

LLNL-PRES-752119 47 software.llnl.gov https://pulse.cio.gov/https/domains/#q=Department%20of%20Energy

Slide 48

Slide 48 text

LLNL-PRES-752119 48 software.llnl.gov https://pulse.cio.gov/https/domains/#q=Department%20of%20Energy

Slide 49

Slide 49 text

LLNL-PRES-752119 49 software.llnl.gov https://18f.gsa.gov/2017/05/25/from-launch-to-landing-how-nasa-took-control-of-its-https-mission/

Slide 50

Slide 50 text

LLNL-PRES-752119 50 software.llnl.gov https://cyber.dhs.gov

Slide 51

Slide 51 text

LLNL-PRES-752119 51 software.llnl.gov https://github.com/dhs-ncats/cyber.dhs.gov

Slide 52

Slide 52 text

LLNL-PRES-752119 52 software.llnl.gov https://arstechnica.com/tech-policy/2018/11/how-i-changed-the-law-with-a-github-pull-request/

Slide 53

Slide 53 text

Thank You! [email protected] @IanLee1521 // @LLNL_OpenSource https://speakerdeck.com/IanLee1521

Slide 54

Slide 54 text

This document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Rference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.