of Energy Preparing FLASH for Exascale Simulations of the Universe’s Most Powerful Explosions Bronson Messer Scientific Computing & Theoretical Physics Groups Oak Ridge National Laboratory Department of Physics & Astronomy University of Tennessee Tom Papatheodore NVIDIA
they are – Detonations in degenerate, high-gravity environments • FLASH – Why accelerating nuclear kinetics is important • Summary This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Co-I’s: Alan Calder†, Sean Couch‡, Petros Tzeferacos§ †Department of Physics, Stony Brook University ‡Department of Physics, Michigan State University §Department of Astronomy and Astrophysics, University of Chicago
galaxy (L ~ 1043 erg/s) • Larger amounts of radioactive 56Ni produced than in CCSNe • Radioactivity powers the light curve (“Arnett’s Law”) • Not associated with star-forming regions (unlike CCSNe) • No compact remnant - star is completely disrupted • Likely event − the accretion-induced thermonuclear explosion of a white dwarf
galaxy (L ~ 1043 erg/s) • Larger amounts of radioactive 56Ni produced than in CCSNe • Radioactivity powers the light curve (“Arnett’s Law”) • Not associated with star-forming regions (unlike CCSNe) • No compact remnant - star is completely disrupted • Likely event − the accretion-induced thermonuclear explosion of a white dwarf
–Robust lightcurve - variations can be corrected with a single-parameter function (Phillips relation) • Distant Ia’s appear dimmer than expected in a Universe without a ‘dark energy’ component. Perlmutter et al. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Discovery of Dark Energy Type Ia supernovae appear dimmer in the Universe with non-zero ! " This led to discovery that rate of expansion of universe is accelerating – and thus to discovery of dark energy - - 2011 Nobel Prize (Perlmutter, Schmidt, & Riess)
star (single degenerate - SD) or WD-WD mergers (double degenerate - DD) possible • Chandrasekhar mass (MCh) or sub-Chandrasekhar mass (sub-MCh) WD –all the permutations are possible, but observational constraints suggest some more likely • Two modes of burning –deflagration - subsonic –detonation - supersonic • + DDT - deflagration-to-detonation transition
star (single degenerate - SD) or WD-WD mergers (double degenerate - DD) possible • Chandrasekhar mass (MCh) or sub-Chandrasekhar mass (sub-MCh) WD –all the permutations are possible, but observational constraints suggest some more likely • Two modes of burning –deflagration - subsonic –detonation - supersonic • + DDT - deflagration-to-detonation transition
create shear which drives turbulence ! After the turbulence cascades down one order of magnitude in length scale it is Kolmogorov and isotropic. On small scales, the flame sees this turbulence and is moved around by it. L 107 cm s-1 v shear 107 cm s-1 ρ~109 gm cm-3 η~109 gm cm-1 s-1 Re= ρvL η ~1014 The Reynolds Number is very large Spontaneous transition to detonation? The disparity of scales in the Type Ia problem - subgrid models and LES white dwarf r ~ 108-9 cm Typical cell in large-scale simulations Δx ~ 1 km Typical thermal width of flame ~10-3 - 10 cm
turbulence (RT), but detonations are also subject to instabilities. • These instabilities can increase the burning length(time) for a given species. –becomes important at lower densities where these burning lengths are already O(RWD) • Network size, resolution, and dimensionality all impact the formation of cellular structures Papatheodore & Messer (2014) 0.5 16O + 0.5 12C 5e7 g/cm3
turbulence (RT), but detonations are also subject to instabilities. • These instabilities can increase the burning length(time) for a given species. –becomes important at lower densities where these burning lengths are already O(RWD) • Network size, resolution, and dimensionality all impact the formation of cellular structures Papatheodore & Messer (2014) 0.5 16O + 0.5 12C 5e7 g/cm3
adaptive mesh refinement (AMR) code that has been used on a variety of parallel platforms. • The code has been used to simulate a variety of phenomenon, including thermonuclear and core-collapse supernovae, galaxy cluster formation, classical novae, the formation of proto-planetary disks, and high-energy-density physics. FLASH’s multi-physics and AMR capabilities make it an ideal numerical laboratory for investigations of nucleosynthesis in supernovae. • Research supported by the DOE SC/NP seeks to understand the nuclear processes that have shaped the cosmos, including the origin of the elements, the evolution of stars, and the detonation of supernovae. • In particular, measurements made at the under-construction Facility for Rare Isotope Beams (FRIB) – coupled with simulations of the late-time evolution of supernovae – will help determine how the elements from iron to uranium are created. Targeted for CAAR 1.Nuclear kinetics (burn unit) threading and vectorization, including Jacobian formation and solution using GPU-enabled libraries 2.Equation of State (EOS) threading and vectorization 3.Hydrodynamics module performance
adaptive mesh refinement (AMR) code that has been used on a variety of parallel platforms. • The code has been used to simulate a variety of phenomenon, including thermonuclear and core-collapse supernovae, galaxy cluster formation, classical novae, the formation of proto-planetary disks, and high-energy-density physics. FLASH’s multi-physics and AMR capabilities make it an ideal numerical laboratory for investigations of nucleosynthesis in supernovae. • Research supported by the DOE SC/NP seeks to understand the nuclear processes that have shaped the cosmos, including the origin of the elements, the evolution of stars, and the detonation of supernovae. • In particular, measurements made at the under-construction Facility for Rare Isotope Beams (FRIB) – coupled with simulations of the late-time evolution of supernovae – will help determine how the elements from iron to uranium are created. Targeted for CAAR 1.Nuclear kinetics (burn unit) threading and vectorization, including Jacobian formation and solution using GPU-enabled libraries 2.Equation of State (EOS) threading and vectorization 3.Hydrodynamics module performance
Backward Euler time evolution and LU decomposition for solution of ODEs Equation of State: Table interpolation (quintic Hermite polynomials) Self-Gravity: Multipole solution of Poisson equation Targeting Summit Architecture • Off-load work to GPUs ⟶ Solution of ODEs (nuclear burning) ⟶ Flux reconstruction (hydrodynamics) ⟶ Table interpolation (equation of state) • Large reaction networks require increased memory per node • Shared memory between CPUs and GPUs will reduce data transfer time S. Couch, MSU FLASH characteristics
offers a unique window to study supernova rates. •Gamma rays penetrate the entire galactic disk with little extinction •44Ti γ-rays reflect the current rate of supernovae, mean decay time scale of τ = 58 years; this present- day snapshot has not yet fed back into chemical evolution •The origin of cosmic 44Ca occurs mainly through 44Ti nucleosynthesis Tsygankov et al (2016) doi.org/10.1093/mnras/stw549 Murchison No. 1, 1996 PROOF OF A
B C N O F Ne Na Mg Al Si P S Cl Ar K Ca Sc Ti V Cr Mn Fe Co Ni Cu Zn 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 neutrino_p_process (zone_01) Timestep = 0 Time (sec) = 0.000E+00 Density (g/cm^3) = 4.214E+06 Temperature (T9) = 8.240E+00 nucastrodata.org nucastrodata.org Max: 2.35E-01 Min: 1.00E-25 Abundance Aprox13: 13-species α-chain X150: 150-species network Proton Number Neutron Number 44Ti
B C N O F Ne Na Mg Al Si P S Cl Ar K Ca Sc Ti V Cr Mn Fe Co Ni Cu Zn 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 neutrino_p_process (zone_01) Timestep = 0 Time (sec) = 0.000E+00 Density (g/cm^3) = 4.214E+06 Temperature (T9) = 8.240E+00 nucastrodata.org nucastrodata.org Max: 2.35E-01 Min: 1.00E-25 Abundance Aprox13: 13-species α-chain X150: 150-species network Proton Number Neutron Number 44Ti
B C N O F Ne Na Mg Al Si P S Cl Ar K Ca Sc Ti V Cr Mn Fe Co Ni Cu Zn 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 neutrino_p_process (zone_01) Timestep = 0 Time (sec) = 0.000E+00 Density (g/cm^3) = 4.214E+06 Temperature (T9) = 8.240E+00 nucastrodata.org nucastrodata.org Max: 2.35E-01 Min: 1.00E-25 Abundance Aprox13: 13-species α-chain X150: 150-species network Proton Number Neutron Number
dwarfs •Modeling these events requires multiphysics codes capable of stellar hydrodynamics, nuclear burning, accurate determination of the gravitational field, and degenerate equations of state. •Accelerating the FLASH code will enable detailed explosion simulations with predictions that can be compared directly to observations. •Details of implementation thus far in Tom’s talk…
bronsonmesser
tagtag
konakalab
aikinohara
olachinkei
trycycle
voginip
rmaruy
tasusu
sergeychernyshev
aemeredith
marktimemedia
davidcarrasco
sferik
kimpetersen
panda_program
mza
.png){kind=avatar}
helenjbeal
notwaldorf
maltzj
