Wir schaffen Wissen – heute für morgen Laboratory for Reactor Physics and Systems Behaviour NURETH-16 Conference, Chicago, USA September 2, 2015 A Methodology for Global Sensitivity Analysis of Transient Code Output Applied to a Reflood Experiment Model using TRACE D. Wicaksono, O. Zerkak, and A. Pautz 

NURETH16/02.09.2015 D. Wicaksono (Page 2/ 22) psi.ch/stars Revisiting OECD/NEA PREMIUM Benchmark Background and Motivation Problem: Simulation FEBA Reflood SETF in thermal- hydraulics code (OECD/NEA PREMIUM Benchmark) Goal: Derive the uncertainty of code (reflood) model parameters using experimental data (temperature) Steps Involved: Parameter Selection, Model Parameters Uncertainty Quantification, Uncertainty Propagation Inverted Annular Film Boiling (IAFB) Dispersed Flow (DFFB) Transitionary (Inverted Slug) TRACE Post-CHF Closure Laws in [mm] H = 4.114 [m] 26 Parameters to specify the TRACE model: 1. (4) boundary conditions (backpressure, inlet flow, etc.) 2. (9) material properties (conductivity, heat capacity, etc.) 3. (2) spacer grid model (heat transfer enhancement, pressure drop correlation) 4. (10) Post-CHF closure relations describing transfer terms of phases (IAFB wall HTC, DFFB Interf. Drag,…) 5. (1) quench temperature 

NURETH16/02.09.2015 D. Wicaksono (Page 3/ 22) psi.ch/stars Revisiting OECD/NEA PREMIUM Benchmark Background and Motivation Problem: Simulation FEBA Reflood SETF in thermal- hydraulics code (OECD/NEA PREMIUM Benchmark) Goal: Derive the uncertainty of code (reflood) model parameters using experimental data (temperature) Steps Involved: Parameter Selection, Model Parameters Uncertainty Quantification, Uncertainty Propagation in [mm] H = 4.114 [m] FEBA Take away message from the Project: • Methodology that has large uncertainty range (minor update) on the parameter able to cover experimental results of other facility • Methodology that updated the prior distribution of a few selected parameters failed to cover experimental results of other facility • Symptom of over-fitting 

NURETH16/02.09.2015 D. Wicaksono (Page 4/ 22) psi.ch/stars How do parameters perturbation affect code output that is a function? Background and Motivation 1. How to characterize the variation of time-dependent code output? 2. How such variations can be decomposed and apportioned to the inputs variation? Reflood Curves at mid asssembly height of 100 Monte-Carlo Samples Problem: Simulation FEBA Reflood SETF in thermal- hydraulics code (OECD/NEA PREMIUM Benchmark) Goal: Derive the uncertainty of code (reflood) model parameters using experimental data (temperature) Steps Involved: Parameter Selection, Model Parameters Uncertainty Quantification, Uncertainty Propagation To investigate how the prior distribution affects the model output and gain better understanding on (TRACE reflood) model behavior 

NURETH16/02.09.2015 D. Wicaksono (Page 5/ 22) psi.ch/stars Describing (Functional) Output Variations 

NURETH16/02.09.2015 D. Wicaksono (Page 6/ 22) psi.ch/stars Suppose: { }, = 1, … , = 0, … − 1 • Realizations (Code runs) • Output given at each = 0, … − 1 • Total data points × Describing the variation in functional output: Discrete Dataset Describing Variation in Functional Output (1) 1 Using a predefined scalar function • Relevant to the application • Problem-specific (max., min., saddle point, etc.) = 1 ෍ =1 − ത 2 Variations of the output described by the variance of some quantities of interest (QoI) Max. Temperature Time of Quenching (time at maximum negative curvature) Which QoI to choose? 

NURETH16/02.09.2015 D. Wicaksono (Page 7/ 22) psi.ch/stars Describing the variation in functional output Describing Variation in Functional Output (3) How to have parsimonious description of functional data that capture the shape variation? 

NURETH16/02.09.2015 D. Wicaksono (Page 8/ 22) psi.ch/stars Describing the variation in functional output: Covariance in Functional Dataset Describing Variation in Functional Output (4) 2 Using functional principal component analysis on functional dataset Functional Dataset: { }, = 1, … , ∈ [ , ] • Realizations (Code runs) • Output given at continuous • Total data points functions , = 1 ෍ =1 ( − ത ()) ⋅ ( − ത ()) (Co)variance in functional sense: Covariance a surface; hard to interpret Functional Principal Component Analysis (fPCA) 

NURETH16/02.09.2015 D. Wicaksono (Page 9/ 22) psi.ch/stars Lower Dimension Projections and its relevance for data analysis fPCA: Eigenvalues and fPC scores KL is an optimal orthogonal basis functions expansion and functional Principal Component (fPC) scores contains the random character of each curves = ത + ෍ =1 ∞ ⋅ fPC scores, calculated for each reflood curve Quantities of Interest describing functional variation • Karhunen-Loève Transform (KLT) of the covariance function reads: , = ෍ =1 ∞ ⋅ ⋅ () • : Eigenvalues • : Orthogonal Eigenfunctions (fPC) = න[ − ത ] ⋅ 

NURETH16/02.09.2015 D. Wicaksono (Page 10/ 22) psi.ch/stars Decomposing Output Variance 

NURETH16/02.09.2015 D. Wicaksono (Page 11/ 22) psi.ch/stars Code output variance can be decomposed into its contributing inputs variances Variance Decomposition Consider a model : ∈ ℝ ↦ ∈ ℝ with -dimensional inputs The High-Dimensional Model Representation (Rabitz et al.) reads: = + ෍ =1 + ෍ 1≤<≤ , + ⋯ + 12… () Univariate Bivariate Higher-order For random variable inputs with independent uniform distributions { ~ 0,1 ; = 1, … }, the Hoeffding–Sobol’ Decomposition gives the variance of : [] = ෍ =1 ~ + ෍ 1≤<≤ [~ , ] + ⋯ 1st-Order (Main) Effect (Partial variance) 2nd-Order Effect (Partial variance) Higher- order Effect Constant 

NURETH16/02.09.2015 D. Wicaksono (Page 12/ 22) psi.ch/stars Sobol’ Indices gives quantitative measure of sensitivity globally with less model assumption Sobol’ Indices 1 , 2 , 3 = 1 + 2 + 3 1 , 2 , 3 = 1 3 + 2 0.8 + 3 20 1 , 2 , 3 = 1 ⋅ 2 + 1 ⋅ 3 + 1 ⋅ 2 ⋅ 3 Additive, Linear Additive, Non-Linear Interacting, Non-Linear Examples: Three kinds of 3-parameter model 2 Sensitivity measures from the decomposition: Sobol’ Indices Definition Meaning Main-effect (1st –order) ≡ ~ Output variance cut if fixed additive effect of each inputs separately Total Effect ≡ 1 − ~ ~ Output variance left if other than fixed total effect of with all its interactions 

NURETH16/02.09.2015 D. Wicaksono (Page 13/ 22) psi.ch/stars Estimating Sobol’ indices by MC simulation is expensive, yet (can be more) affordable Estimating Sobol’ Indices ≡ ~ = ׭ ~ ′ , ⋅ ~ , ~ ′ − ׬ 2 ׬ 2 − ׬ 2 2 + 1-dimensional integral! Variance and Expectation can be written in integral form Monte-Carlo (MC) estimation by Sobol’ –Saltelli method (sampling - resampling): required runs = × + . Improvement by: = 1’000 # of parameters () 5 10 26 Code runs 7’000 12’000 28’000 1 Parameters Screening 2 Using Better MC Estimator A better estimator 

NURETH16/02.09.2015 D. Wicaksono (Page 14/ 22) psi.ch/stars Only 10 parameters (of 26) are influential in the reflood temperature transient Notes on Screening • Parameter screening using the Morris Screening method • Closure laws, boundary conditions, and spacer grid model parameters are relatively influential (10 parameters) • 16 parameters are excluded from further analysis • Generate 2’000 MC samples on the 10-parameter model What does influential mean? Check run with 50 MC samples for each parameter set 

NURETH16/02.09.2015 D. Wicaksono (Page 15/ 22) psi.ch/stars Simulation experiment with in-house scripting and open-source tools Simulation Experiment Design of Experiment Sensitivity Analysis Post-processor Functional Data Analysis Post-processor • Simple Random Sampling (SRS) • Latin Hypercube Sampling (LHS) • Sobol’ Sequence • Morris factorial design TRACE Launcher • Basis function expansion (B-spline) • Curve registration • Functional Principal Component Analysis • Sobol’ Indices (1st- and total) • Elementary effects • … • Parallel Execution • Output management Results Reporting Simulation experiment on TRACE reflood model to decompose mid-assembly temperature variance: • Sobol’ Indices estimated using Janon et al. estimator (1st) and Jansen est. (total) • 2’000 Sobol’ sequence samples on the 10-parameter model (24’000 TRACE runs) • 10’000 bootstrap samples for MC error assessment 

NURETH16/02.09.2015 D. Wicaksono (Page 16/ 22) psi.ch/stars Results 

NURETH16/02.09.2015 D. Wicaksono (Page 17/ 22) psi.ch/stars For 2 relevant static quantities in reflood, TRACE model is additive (minor interaction) Sensitivity Analysis of Static Quantities 48% 12% 8% 8% Spacer Grid 18% 24% 22% 20% Other 1st Order 10% Other 1st Order 15% Interaction 7% Interaction 9% Max. Temperature σ = 60 [K], σ = % Quenching Time σ = 61 [s], σ = % (66%) DFFB Closure Laws Bootstrap samples boxplot (CI: Confidence Interval) Median 50% 95% CI 1 Using a predefined scalar function 

NURETH16/02.09.2015 D. Wicaksono (Page 18/ 22) psi.ch/stars The eigenfunctions describe the functional variation of a functional dataset fPCA: Eigenfunctions Eigenfunction [K] Temperature [K] 1 , 2 ≈ ෍ =1 3 ⋅ 1 ⋅ 2 Perturbation (±) around the mean by the eigenfunction assists in their interpretation Temperature Ramp Temperature Descent Quenching 55% 32% 5% KL Transform in finite series Using fPCA on functional dataset 2 

NURETH16/02.09.2015 D. Wicaksono (Page 19/ 22) psi.ch/stars Temperature ramp variation is driven by DFFB and Grid HT parameters (no interact.) Sensitivity Analysis of 1st fPC scores • The model is additive with respect to temperature raise • Given temperature data, these parameters (and their uncertainties) by structure of the model can be estimated 21% 21% 25% 20% Other Main-Effect 5% Interaction 8% DFFB Parameters (66%) Sum of Main-Effect Indices = 92% 

NURETH16/02.09.2015 D. Wicaksono (Page 20/ 22) psi.ch/stars Temperature descent variation is driven by the interactions of all model parameters Sensitivity Analysis of 2nd fPC Scores Small 1st-order effect: σ = % Boundary Conditions Physical Model Parameters Δ indices = Total Interaction Effect • The model is interacting w.r.t to this functional behavior • Parameters estimation might suffer from non-identifiability (non-uniqueness) 

(Page 21/ 22) Conclusions How do parameters perturbation affect code output that is a function? 1. Using a scalar function (problem specific but relevant) 2. Using functional principal component analysis (parsimonious, but less straightforward) Combining fPC and Variance-Based sensitivity analysis gives more complete description of model behavior; functional output variation and attributing these variations to the responsible input … and apply them to the OECD/NEA Benchmark Problem using TRACE… • Reducing the variation of the max. temperature and time of quenching is possible in terms of the structure of the model • Parameter interaction exists for some form of functional variation (possible non-identifiability issue, but which with which not known yet) Decompose them using Sobol’-Saltelli Method 

Thanks a lot for your attention Acknowledgments: • Dr. Gregory Perret, LRS – Paul Scherrer Institut • Dr. Carl Adamsson, Vattenfall AB • Swiss Federal Nuclear Safety Inspectorate (ENSI) • Swiss Federal Office of Energy (BFM) 

NURETH16/02.09.2015 D. Wicaksono (Page 23/ 22) psi.ch/stars References 1. P. Ihle and K. Rust, “FEBA-Flooding Experiments with Blocked Arrays,” Evaluation Report, Kernforschungzentrum Karlsruhe, 1994 2. US NRC, “TRACE v5.0p3 Theory Manual,” United States Regulatory Commission, Washington D.C., 2012 3. Perez et al., “Uncertainty and sensitivity Analysis of a LBLOCA in a PWR NPP: Results of the Phase V of the BEMUSE programme,” Nucl. Eng. Des., vol. 241, pp. 4206-4222, 2011 4. J. Ramsay and B. W. Silverman, “Functional Data Analysis,” 2nd Edition, New York: Springer Science+Business Media, LLC, 2005. 5. G. Li et al., “High Dimensional Model Representations,” J. Phys. Chem. A, 105(33), pp. 7765-7777 (2001) 6. I. M. Sobol’, “Global sensitivity indices for nonlinear mathematical models and their monte carlo estimates,” Math. Comput. Simul., 55, pp. 271-280, 2001 7. T. Homma and A. Saltelli, “Importance measures in global sensitivity analysis of nonlinear models,” Reliab. Eng. Sys. Saf., 52(1), pp. 1-17, 1996 8. A. Saltelli, “Making best use of model evaluations to compute sensitivity indices,” Comput. Phys. Commun., 181(2), pp. 259-270, 2010 

NURETH16/02.09.2015 D. Wicaksono (Page 24/ 22) psi.ch/stars References 9. A. Janon et al, “Asymptotic normality and efficiency of two Sobol’ index estimators,” ESAIM Probab. Stat., 18, pp. 342-364, 2014 10. M. D. Morris, “Factorial Sampling Plans for Preliminary Computational Experiments,” Technometrics, vol. 33, no. 2, pp. 161 – 174, 1991. 11. F. Campolongo, J. Cariboni, and A. Saltelli, “An effective screening design for sensitivity analysis for large models,” Environmental Modelling & Software, vol. 22, pp. 1509 – 1518, 2007. 12. Ramsay et al., “fda: Functional Data Analysis,” R Package version 2.4.0, The Comprehensive R Archive Network (CRAN), 2013 13. R Core Team, “R: a Language and Environment for Statistical Computing,” R Foundation for Statistical Computing, Vienna, Austria, 2014, http://www.R- project.org