Seminar talk at Cold Spring Harbor Laboratory (CSHL)

Transcript

1. HIGH-DIMENSIONAL SURROGATE MODELING FOR EXPENSIVE COMPUTER CODES A machine learning approach Rohit Tripathy PhD candidate, Predictive Science Lab Purdue University, West Lafayette, IN https://rohittripathy.netlify.com/ (Advised by Prof. Ilias Bilionis)
2. OUTLINE 1. What is uncertainty quantification (UQ) ? 2. UQ for expensive, high-dimensional systems. 3. Approach 1 – Linear low-rank structure within Gaussian processes. 4. Approach 2 – Linear low-rank structure within deep neural networks. 5. Approach 3 – Generalized nonlinear reduction 6. Concluding remarks.
3. • Identifying and describing sources of uncertainties. • Calibrating models/experiments. • Propagating uncertainties in the inputs. • Optimizing the output of scientific/engineering processes characterized by uncertain parameters. INTRODUCTION WHAT IS UQ?
4. Hertzian contact model dynamical system EXAMPLES Ref. - Tripathy, Rohit, Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223. Fig.: Propagation of solitary waves (solitons) in granular crystals
5. NACA12 airfoil Turbulent flow CFD solver O ptim ization under uncertainty Ref. - Seshadri, Pranay, et al. "A density-matching approach for optimization under uncertainty." Computer Methods in Applied Mechanics and Engineering 305 (2016): 562-578. EXAMPLES
6. Catalytic conversion of nitrate to nitrogen – calibration of reaction rate coefficients Identification of pollutant source– calibration of source term in system of PDEs Ref. - Tsilifis, Panagiotis, et al. "Computationally efficient variational approximations for Bayesian inverse problems." Journal of Verification, Validation and Uncertainty Quantification 1.3 (2016): 031004. EXAMPLES
7. UQ IN HIGH-DIMENSIONAL AND EXPENSIVE COMPUTATIONAL MODELS 7
8. NUMERICAL SIMULATOR INPUT PARAMETERS QUANTITY OF INTEREST f q 8 MATHEMATICAL SETUP • f is expensive to evaluate. • is high-dimensional, i.e. ∈ ℝ!, D ≫ 1. • q is scalar.
9. 9 Running example – steady-state elliptic PDE MATHEMATICAL SETUP
10. FORWARD PROBLEM (PROPAGATION) INVERSE PROBLEM (CALIBRATION) FORMALIZATION 10
11. THE SURROGATE APPROACH • Perform a finite number of simulations. • Solve the UQ problem with the surrogate. • Replace model with an approximation: 11 • Repeated evaluation of the likelihood is expensive. THIS IS DIFFICULT IN HIGH- DIMENSIONS !
12. REF: https://speakerdeck.com/paulcon/active-subspaces-emerging-ideas-for-dimension-reduction-in-parameter-studies-2 CURSE OF DIMENSIONALITY 12
13. APPROACH 1: Gaussian process surrogate with built-in dimensionality reduction 13
14. 14 GAUSSIAN PROCESS REGRESSION Statistical model: Prior GP: Data: Posterior GP: Posterior mean: Posterior variance: Prior GP Bayes rule Posterior GP
15. 15 GPR - INFERENCE INFERENCE PROBLEM – Estimate the posterior distribution of the hyperparameters. Ref. - Rasmussen, Carl Edward. "Gaussian processes in machine learning." Advanced lectures on machine learning. Springer, Berlin, Heidelberg, 2004. 63-71.. Hyperparameter prior: Hyperparameter posterior: Approximation of the posterior at the mode: Marginal likelihood:
16. CLASSICAL ACTIVE SUBSPACE Ref. - Constantine, Paul G., Eric Dow, and Qiqi Wang. "Active subspace methods in theory and practice: applications to kriging surfaces." SIAM Journal on Scientific Computing 36.4 (2014): A1500-A1524. Probability Measure: Sample gradients: Centered covariance: Spectral decomposition: Final surrogate: How do we get these? 16
17. GRADIENT-FREE ACTIVE SUBSPACE RECOVERY Data: Surrogate: Optimize: 17 Tripathy, Rohit, Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223.
18. GRADIENT-FREE ACTIVE SUBSPACE RECOVERY Data: Surrogate: Hyperparameters Gaussian Process Bayes rule Modified kernel Any standard kernel Likelihood noise Negative log marginal likelihood 18
19. PDE: Uncertainty: Quantity of interest: Ω Γ1 Γ2 19 Tripathy, Rohit, Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223. STOCHASTIC ELLIPTIC PDE
20. Surrogate modelling task: We need to construct the map that links the vector of uncertain parameters to the quantity of interest q. Find: With data**: High dimensional inputs, small sample set. **Data source: https://github.com/paulcon/as-data-sets/tree/master/Elliptic_PDE STOCHASTIC ELLIPTIC PDE 20 Tripathy, Rohit, Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223.
21. Fig: Comparison of 1-d active subspace from gradient-based and gradient-free approach. Active subspace recovered using gradients Active subspace recovered without access to gradients (our approach) NOTICE : We recover the right active subspace upto arbitrary rotations ! 21 Tripathy, Rohit, Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223. STOCHASTIC ELLIPTIC PDE
22. Fig: Comparison of predictive accuracy between gradient-based and gradient-free approach. Surrogate model predicted vs observed values on test dataset (gradient-based approach) Surrogate model predicted vs observed values on test dataset (gradient-free approach) 22 Tripathy, Rohit, Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223. STOCHASTIC ELLIPTIC PDE
23. APPROACH 2: Active Subspace Recovery With DNNs 23
24. Fig.: Schematic of a DNN. DEEP NEURAL NETWORKS M-layer hidden layer feedforward network: where: 24
25. Fig.: Schematic of a DNN DEEP NEURAL NETWORKS Trainable parameters: Training data: Mini-batch data loss: Discrepancy / log likelihood Regularizer / Log prior Stochastic Gradient descent update: 25
26. ACTIVE SUBSPACE - REVISITED Ref. - Constantine, Paul G., Eric Dow, and Qiqi Wang. "Active subspace methods in theory and practice: applications to kriging surfaces." SIAM Journal on Scientific Computing 36.4 (2014): A1500-A1524. Probability Measure: Sample gradients: Centered covariance: Spectral decomposition: Final surrogate: 26
27. ACTIVE SUBSPACE RECOVERY Orthogonal projection GRAM-SCHMIDT ORTHOGONALIZATION W = h(Q) 2 RD⇥d <latexit sha1_base64="9S+6LBM+NNL49YKd9aan/pE89oE=">AAACH3icbVBNS8NAEN3Ur1q/oh69LBahXkoiol6Egh701or9gCaWzXbTLt1swu5GKCH/xIt/xYsHRcRb/42bNoK2Phh482aGmXlexKhUljUxCkvLK6trxfXSxubW9o65u9eSYSwwaeKQhaLjIUkY5aSpqGKkEwmCAo+Rtje6yurtRyIkDfm9GkfEDdCAU59ipLTUM8+cAKmh5yftFF7CYeUnbaTH0KEcznIvuUsfkmvoKBoQCftpzyxbVWsKuEjsnJRBjnrP/HL6IY4DwhVmSMqubUXKTZBQFDOSlpxYkgjhERqQrqYc6T1uMv0vhUda6UM/FDq4glP190SCAinHgac7s3PlfC0T/6t1Y+VfuAnlUawIx7NFfsygCmFmFuxTQbBiY00QFlTfCvEQCYSVtrSkTbDnX14krZOqbVXtxmm5dpvbUQQH4BBUgA3OQQ3cgDpoAgyewAt4A+/Gs/FqfBifs9aCkc/sgz8wJt83TKJ7</latexit> <latexit sha1_base64="9S+6LBM+NNL49YKd9aan/pE89oE=">AAACH3icbVBNS8NAEN3Ur1q/oh69LBahXkoiol6Egh701or9gCaWzXbTLt1swu5GKCH/xIt/xYsHRcRb/42bNoK2Phh482aGmXlexKhUljUxCkvLK6trxfXSxubW9o65u9eSYSwwaeKQhaLjIUkY5aSpqGKkEwmCAo+Rtje6yurtRyIkDfm9GkfEDdCAU59ipLTUM8+cAKmh5yftFF7CYeUnbaTH0KEcznIvuUsfkmvoKBoQCftpzyxbVWsKuEjsnJRBjnrP/HL6IY4DwhVmSMqubUXKTZBQFDOSlpxYkgjhERqQrqYc6T1uMv0vhUda6UM/FDq4glP190SCAinHgac7s3PlfC0T/6t1Y+VfuAnlUawIx7NFfsygCmFmFuxTQbBiY00QFlTfCvEQCYSVtrSkTbDnX14krZOqbVXtxmm5dpvbUQQH4BBUgA3OQQ3cgDpoAgyewAt4A+/Gs/FqfBifs9aCkc/sgz8wJt83TKJ7</latexit> <latexit sha1_base64="9S+6LBM+NNL49YKd9aan/pE89oE=">AAACH3icbVBNS8NAEN3Ur1q/oh69LBahXkoiol6Egh701or9gCaWzXbTLt1swu5GKCH/xIt/xYsHRcRb/42bNoK2Phh482aGmXlexKhUljUxCkvLK6trxfXSxubW9o65u9eSYSwwaeKQhaLjIUkY5aSpqGKkEwmCAo+Rtje6yurtRyIkDfm9GkfEDdCAU59ipLTUM8+cAKmh5yftFF7CYeUnbaTH0KEcznIvuUsfkmvoKBoQCftpzyxbVWsKuEjsnJRBjnrP/HL6IY4DwhVmSMqubUXKTZBQFDOSlpxYkgjhERqQrqYc6T1uMv0vhUda6UM/FDq4glP190SCAinHgac7s3PlfC0T/6t1Y+VfuAnlUawIx7NFfsygCmFmFuxTQbBiY00QFlTfCvEQCYSVtrSkTbDnX14krZOqbVXtxmm5dpvbUQQH4BBUgA3OQQ3cgDpoAgyewAt4A+/Gs/FqfBifs9aCkc/sgz8wJt83TKJ7</latexit> 1. 2. 3. 4. Deep Neural Network Network weights/biases 27
28. STOCHASTIC ELLIPTIC PDE B.C.s: PDE: Uncertain diffusion coefficient: Quantity of interest: Exponential kernel: 28
29. STOCHASTIC ELLIPTIC PDE Table : Test set RMSE comparison for long and short correlation length cases. Short correlation length ( 0.01 ) Long correlation length ( 1.0 ) 29
30. APPROACH 3: Deep Neural Network (DNN) surrogate modeling framework 30
31. To be determined from data: • L: number of layers • d: number of nodes in last layer before output PARAMETERIZED ARCHITECTURE 31 Width of kth layer: Uniquely determined by L and d Deep Neural Network Network weights/biases D
32. MODEL SELECTION - WORKFLOW 32 Generate grid of L and d Bayesian global optimization for λ at all grid locations Use best estimate of λ, L and d to train final network. EMBARASSINGLY PARALLELIZABLE ! We need to pick: 1. The number of layers, L. 2. The size of the active subspace encoding, d. 3. The regularization constant, λ.
33. 33 u = 0, 8x1 = 1, u = 1, 8x1 = 0, @u @n = 0, 8x2 = 1. PDE: Boundary conditions: Uncertain diffusion: log Exponential covariance: STOCHASTIC ELLIPTIC PDE SETUP
34. STOCHASTIC ELLIPTIC PDE DATA 34 FEM FEM
35. 35 o2000 random conductivities generated by: • Sampling length-scales [1]. • Sampling the conductivity on a 32x32 grid. oFor each one of the sampled conductivities, we solved the PDE on a 32x32 grid. u(x, ✓) <latexit sha1_base64="ErE6xeO1oqTds87eAIYJzhu3d/o=">AAACJXicbVDLSgMxFM3UV62vqks3wSIoSJkRQRcuim5cKlgtdErJZO60oZkHyR2xDPMzbvwVNy4sIrjyV8zUgtp6IOTk3EfuPV4ihUbb/rBKc/MLi0vl5crK6tr6RnVz61bHqeLQ5LGMVctjGqSIoIkCJbQSBSz0JNx5g4sifncPSos4usFhAp2Q9SIRCM7QSN3q2Z4bMux7QfaQV9L9n8chzdxx+0yBn1PXi6Wvh6G5Mhf7gCzPD7rVml23x6CzxJmQGpngqlsduX7M0xAi5JJp3XbsBDsZUyi4hLziphoSxgesB21DIxaC7mTjMXK6ZxSfBrEyJ0I6Vn9XZCzUxYQms9hCT8cK8b9YO8XgtJOJKEkRIv79UZBKijEtLKO+UMBRDg1hXAkzK+V9phhHY2zFmOBMrzxLbo/qjl13ro9rjfOJHWWyQ3bJPnHICWmQS3JFmoSTR/JMXsnIerJerDfr/Tu1ZE1qtskfWJ9fggumgw==</latexit> <latexit sha1_base64="ErE6xeO1oqTds87eAIYJzhu3d/o=">AAACJXicbVDLSgMxFM3UV62vqks3wSIoSJkRQRcuim5cKlgtdErJZO60oZkHyR2xDPMzbvwVNy4sIrjyV8zUgtp6IOTk3EfuPV4ihUbb/rBKc/MLi0vl5crK6tr6RnVz61bHqeLQ5LGMVctjGqSIoIkCJbQSBSz0JNx5g4sifncPSos4usFhAp2Q9SIRCM7QSN3q2Z4bMux7QfaQV9L9n8chzdxx+0yBn1PXi6Wvh6G5Mhf7gCzPD7rVml23x6CzxJmQGpngqlsduX7M0xAi5JJp3XbsBDsZUyi4hLziphoSxgesB21DIxaC7mTjMXK6ZxSfBrEyJ0I6Vn9XZCzUxYQms9hCT8cK8b9YO8XgtJOJKEkRIv79UZBKijEtLKO+UMBRDg1hXAkzK+V9phhHY2zFmOBMrzxLbo/qjl13ro9rjfOJHWWyQ3bJPnHICWmQS3JFmoSTR/JMXsnIerJerDfr/Tu1ZE1qtskfWJ9fggumgw==</latexit> <latexit sha1_base64="ErE6xeO1oqTds87eAIYJzhu3d/o=">AAACJXicbVDLSgMxFM3UV62vqks3wSIoSJkRQRcuim5cKlgtdErJZO60oZkHyR2xDPMzbvwVNy4sIrjyV8zUgtp6IOTk3EfuPV4ihUbb/rBKc/MLi0vl5crK6tr6RnVz61bHqeLQ5LGMVctjGqSIoIkCJbQSBSz0JNx5g4sifncPSos4usFhAp2Q9SIRCM7QSN3q2Z4bMux7QfaQV9L9n8chzdxx+0yBn1PXi6Wvh6G5Mhf7gCzPD7rVml23x6CzxJmQGpngqlsduX7M0xAi5JJp3XbsBDsZUyi4hLziphoSxgesB21DIxaC7mTjMXK6ZxSfBrEyJ0I6Vn9XZCzUxYQms9hCT8cK8b9YO8XgtJOJKEkRIv79UZBKijEtLKO+UMBRDg1hXAkzK+V9phhHY2zFmOBMrzxLbo/qjl13ro9rjfOJHWWyQ3bJPnHICWmQS3JFmoSTR/JMXsnIerJerDfr/Tu1ZE1qtskfWJ9fggumgw==</latexit> STOCHASTIC ELLIPTIC PDE DATA [1] - Rohit K Tripathy and Ilias Bilionis. Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification. Journal of Computational Physics, 375:565–588, 2018.
36. 36 STOCHASTIC ELLIPTIC PDE MODEL SELECTION Lowest Validation error (and selected DNN) Bayesian Global Optimization Negative Validation Error A different DNN trained with ADAM.
37. 37 STOCHASTIC ELLIPTIC PDE PREDICTION Fig.: (From left column to right) (a) Input random field; (b) True (FEM) solution; (c) Predicted solution. Fig.: (Top) Histogram of R2 across test dataset. (Bottom) Histogram of relative errors across test dataset.
38. PROPAGATING INPUT UNCERTAINTY SPDE WITH FIXED LENGTHSCALE 38 Fig.: Comparison of solution pdf at x = (0.484,0.484) obtained from MCS* and DNN surrogate. Fig. : Comparison of Monte Carlo* (left) mean and variance and surrogate (right) mean and variance for the PDE solution. FEM Surrogate
39. PROPAGATING INPUT UNCERTAINTY SPDE WITH RANDOM LENGTHSCALE 39 Fig.: Comparison of solution pdf at x = (0.484,0.484) obtained from MCS* and DNN surrogate. Fig. : Comparison of Monte Carlo* (left) mean and variance and surrogate (right) mean and variance for the PDE solution.
40. CONCLUDING REMARKS § Monte Carlo approach to UQ – infeasible due to slow convergence. § Surrogate approach to UQ – constrained by high-dimensionality and cost of numerical simulator. § Strategy – find suitable low-rank approximation of the stochastic parameter space: • Optimal linear projections for capturing directions of maximal variation of QoI gradient (active subspace). • Nonlinear manifold capturing QoI variability (deep neural networks). § Limitations with proposed approaches – • Scalability concerns with GPs. • Incorporation of rigorous priors (continuity/smoothness etc.) in DNNs. • Incorporation of physical constraints. 40
41. CONCLUDING REMARKS § Ongoing/future work – • Encoding physical laws into statistical models (physics-informed machine learning). • Synthesizing information from multiple sources of varying levels of fidelity (multifidelity surrogate modeling & uncertainty quantification). • Exploiting group-theoretic structure in dynamical systems. • Developing surrogates robust to non-stationarities, discontinuities etc. 41
42. REFERENCES 42 [1] Rohit Tripathy, Ilias Bilionis, and Marcial Gonzalez. Gaussian processes with built-in dimensionality re- duction: Applications to high-dimensional uncertainty propagation. Journal of Computational Physics, 321:191–223, 2016. [2] Rohit K Tripathy and Ilias Bilionis. Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification. Journal of Computational Physics, 375:565–588, 2018. [3] Rohit Tripathy and Ilias Bilionis. Deep active subspaces–a scalable method for high-dimensional uncertainty propagation. arXiv preprint arXiv:1902.10527, 2019. [4] Rohit Tripathy and Ilias Bilionis. Learning deep neural network (DNN) surrogate models for UQ. SIAM UQ, 2018. [5] Sharmila Karumuri, Rohit Tripathy, Ilias Bilionis, and Jitesh Panchal. Simulator-free solution of high- dimensional stochastic elliptic partial differential equations using deep neural networks. arXiv preprint arXiv:1902.05200, 2019.
43. 43 THANK YOU !