2017 OLCF Users Meeting

Transcript

1. ORNL is managed by UT-Battelle 
 for the US Department of Energy Preparing FLASH for Exascale Simulations of the Universe’s Most Powerful Explosions Bronson Messer Scientific Computing & Theoretical Physics Groups Oak Ridge National Laboratory Department of Physics & Astronomy University of Tennessee Tom Papatheodore NVIDIA
2. Outline • An overview of Type Ia Supernovae – What they are – Detonations in degenerate, high-gravity environments • FLASH – Why accelerating nuclear kinetics is important • Summary This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Co-I’s: Alan Calder†, Sean Couch‡, Petros Tzeferacos§ †Department of Physics, Stony Brook University ‡Department of Physics, Michigan State University
 §Department of Astronomy and Astrophysics, University of Chicago
3. October 2006
6. Supernova types
7. Supernova types
8. Supernova types
9. Supernova types
10. Type Ia supernovae • Brightness rivals that of the host galaxy (L ~ 1043 erg/s) • Larger amounts of radioactive 56Ni produced than in CCSNe • Radioactivity powers the light curve (“Arnett’s Law”) • Not associated with star-forming regions (unlike CCSNe) • No compact remnant - star is completely disrupted • Likely event − the accretion-induced thermonuclear explosion of a white dwarf
11. Type Ia supernovae • Brightness rivals that of the host galaxy (L ~ 1043 erg/s) • Larger amounts of radioactive 56Ni produced than in CCSNe • Radioactivity powers the light curve (“Arnett’s Law”) • Not associated with star-forming regions (unlike CCSNe) • No compact remnant - star is completely disrupted • Likely event − the accretion-induced thermonuclear explosion of a white dwarf
12. Type Ia supernova cosmology • SNe Ia are ‘standardizable’ candles –Robust lightcurve - variations can be corrected with a single-parameter function (Phillips relation) • Distant Ia’s appear dimmer than expected in a Universe without a ‘dark energy’ component. Perlmutter et al. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Discovery of Dark Energy Type Ia supernovae appear dimmer in the Universe with non-zero ! " This led to discovery that rate of expansion of universe is accelerating – and thus to discovery of dark energy - - 2011 Nobel Prize (Perlmutter, Schmidt, & Riess)
13. Ia progenitors and burning modes • Accretion from companion (non-compact) star (single degenerate - SD) or WD-WD mergers (double degenerate - DD) possible • Chandrasekhar mass (MCh) or sub-Chandrasekhar mass (sub-MCh) WD –all the permutations are possible, but observational constraints suggest some more likely • Two modes of burning –deflagration - subsonic –detonation - supersonic • + DDT - deflagration-to-detonation transition
14. Ia progenitors and burning modes • Accretion from companion (non-compact) star (single degenerate - SD) or WD-WD mergers (double degenerate - DD) possible • Chandrasekhar mass (MCh) or sub-Chandrasekhar mass (sub-MCh) WD –all the permutations are possible, but observational constraints suggest some more likely • Two modes of burning –deflagration - subsonic –detonation - supersonic • + DDT - deflagration-to-detonation transition
15. The disparity of scales in the Type Ia problem - subgrid models and LES white dwarf r ~ 108-9 cm Typical cell in large-scale simulations Δx ~ 1 km Typical thermal width of flame ~10-3 - 10 cm
16. White d collision dilution. detonat The ans !  Rising plumes create shear which drives turbulence !  After the turbulence cascades down one order of magnitude in length scale it is Kolmogorov and isotropic. On small scales, the flame sees this turbulence and is moved around by it. L 107 cm s-1 v shear 107 cm s-1 ρ~109 gm cm-3 η~109 gm cm-1 s-1 Re= ρvL η ~1014 The Reynolds Number is very large Spontaneous transition to detonation? The disparity of scales in the Type Ia problem - subgrid models and LES white dwarf r ~ 108-9 cm Typical cell in large-scale simulations Δx ~ 1 km Typical thermal width of flame ~10-3 - 10 cm
17. Detonation - cellular structure formation • Deflagrations make their own turbulence (RT), but detonations are also subject to instabilities. • These instabilities can increase the burning length(time) for a given species. –becomes important at lower densities where these burning lengths are already O(RWD) • Network size, resolution, and dimensionality all impact the formation of cellular structures Papatheodore & Messer (2014) 0.5 16O + 0.5 12C 5e7 g/cm3
18. Detonation - cellular structure formation • Deflagrations make their own turbulence (RT), but detonations are also subject to instabilities. • These instabilities can increase the burning length(time) for a given species. –becomes important at lower densities where these burning lengths are already O(RWD) • Network size, resolution, and dimensionality all impact the formation of cellular structures Papatheodore & Messer (2014) 0.5 16O + 0.5 12C 5e7 g/cm3
19. 3D detonations ~40,000 cores roughly 30 cm on a side 16O 28Si
20. 3D detonations ~40,000 cores roughly 30 cm on a side 16O 28Si
21. FLASH • FLASH is a publicly available, component-based, massively parallel, adaptive mesh refinement (AMR) code that has been used on a variety of parallel platforms. • The code has been used to simulate a variety of phenomenon, including thermonuclear and core-collapse supernovae, galaxy cluster formation, classical novae, the formation of proto-planetary disks, and high-energy-density physics. FLASH’s multi-physics and AMR capabilities make it an ideal numerical laboratory for investigations of nucleosynthesis in supernovae. • Research supported by the DOE SC/NP seeks to understand the nuclear processes that have shaped the cosmos, including the origin of the elements, the evolution of stars, and the detonation of supernovae. • In particular, measurements made at the under-construction Facility for Rare Isotope Beams (FRIB) – coupled with simulations of the late-time evolution of supernovae – will help determine how the elements from iron to uranium are created. Targeted for CAAR 1.Nuclear kinetics (burn unit) threading and vectorization, including Jacobian formation and solution using GPU-enabled libraries 2.Equation of State (EOS) threading and vectorization 3.Hydrodynamics module performance
22. FLASH • FLASH is a publicly available, component-based, massively parallel, adaptive mesh refinement (AMR) code that has been used on a variety of parallel platforms. • The code has been used to simulate a variety of phenomenon, including thermonuclear and core-collapse supernovae, galaxy cluster formation, classical novae, the formation of proto-planetary disks, and high-energy-density physics. FLASH’s multi-physics and AMR capabilities make it an ideal numerical laboratory for investigations of nucleosynthesis in supernovae. • Research supported by the DOE SC/NP seeks to understand the nuclear processes that have shaped the cosmos, including the origin of the elements, the evolution of stars, and the detonation of supernovae. • In particular, measurements made at the under-construction Facility for Rare Isotope Beams (FRIB) – coupled with simulations of the late-time evolution of supernovae – will help determine how the elements from iron to uranium are created. Targeted for CAAR 1.Nuclear kinetics (burn unit) threading and vectorization, including Jacobian formation and solution using GPU-enabled libraries 2.Equation of State (EOS) threading and vectorization 3.Hydrodynamics module performance
23. Hydrodynamics: Solves reactive Euler equations using Piecewise-Parabolic Method Nuclear Burning: Backward Euler time evolution and LU decomposition for solution of ODEs Equation of State: Table interpolation (quintic Hermite polynomials) Self-Gravity: Multipole solution of Poisson equation Targeting Summit Architecture • Off-load work to GPUs ⟶ Solution of ODEs (nuclear burning) ⟶ Flux reconstruction (hydrodynamics) ⟶ Table interpolation (equation of state) • Large reaction networks require increased memory per node • Shared memory between CPUs and GPUs will reduce data transfer time S. Couch, MSU FLASH characteristics
24. Why is the burner important? — e.g., 44Ti 44Ti decay offers a unique window to study supernova rates. •Gamma rays penetrate the entire galactic disk with little extinction •44Ti γ-rays reflect the current rate of supernovae, mean decay time scale of τ = 58 years; this present- day snapshot has not yet fed back into chemical evolution •The origin of cosmic 44Ca occurs mainly through 44Ti nucleosynthesis Tsygankov et al (2016) doi.org/10.1093/mnras/stw549 Murchison No. 1, 1996 PROOF OF A
25. Why the burner?
26. Why the burner?
27. Why do α-networks over-produce 44Ti? n H He Li Be B C N O F Ne Na Mg Al Si P S Cl Ar K Ca Sc Ti V Cr Mn Fe Co Ni Cu Zn 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 neutrino_p_process (zone_01) Timestep = 0 Time (sec) = 0.000E+00 Density (g/cm^3) = 4.214E+06 Temperature (T9) = 8.240E+00 nucastrodata.org nucastrodata.org Max: 2.35E-01 Min: 1.00E-25 Abundance Aprox13: 13-species α-chain X150: 150-species network Proton Number Neutron Number 44Ti
28. Why do α-networks over-produce 44Ti? n H He Li Be B C N O F Ne Na Mg Al Si P S Cl Ar K Ca Sc Ti V Cr Mn Fe Co Ni Cu Zn 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 neutrino_p_process (zone_01) Timestep = 0 Time (sec) = 0.000E+00 Density (g/cm^3) = 4.214E+06 Temperature (T9) = 8.240E+00 nucastrodata.org nucastrodata.org Max: 2.35E-01 Min: 1.00E-25 Abundance Aprox13: 13-species α-chain X150: 150-species network Proton Number Neutron Number 44Ti
29. Why do α-networks over-produce 44Ti? n H He Li Be B C N O F Ne Na Mg Al Si P S Cl Ar K Ca Sc Ti V Cr Mn Fe Co Ni Cu Zn 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 neutrino_p_process (zone_01) Timestep = 0 Time (sec) = 0.000E+00 Density (g/cm^3) = 4.214E+06 Temperature (T9) = 8.240E+00 nucastrodata.org nucastrodata.org Max: 2.35E-01 Min: 1.00E-25 Abundance Aprox13: 13-species α-chain X150: 150-species network Proton Number Neutron Number
30. Building a new framework for stellar astrophysics simulation, ExaStar - CLASH
31. Summary •Type Ia supernovae are the thermonuclear explosions of white dwarfs •Modeling these events requires multiphysics codes capable of stellar hydrodynamics, nuclear burning, accurate determination of the gravitational field, and degenerate equations of state. •Accelerating the FLASH code will enable detailed explosion simulations with predictions that can be compared directly to observations. •Details of implementation thus far in Tom’s talk…