Seminar talk at Cold Spring Harbor Laboratory (CSHL)

Seminar talk at Cold Spring Harbor Laboratory (CSHL)

A3135db419cf364a0d04e4bf80d0b142?s=128

Rohit Tripathy

January 15, 2020
Tweet

Transcript

  1. 1.

    HIGH-DIMENSIONAL SURROGATE MODELING FOR EXPENSIVE COMPUTER CODES A machine learning

    approach Rohit Tripathy PhD candidate, Predictive Science Lab Purdue University, West Lafayette, IN https://rohittripathy.netlify.com/ (Advised by Prof. Ilias Bilionis)
  2. 2.

    OUTLINE 1. What is uncertainty quantification (UQ) ? 2. UQ

    for expensive, high-dimensional systems. 3. Approach 1 – Linear low-rank structure within Gaussian processes. 4. Approach 2 – Linear low-rank structure within deep neural networks. 5. Approach 3 – Generalized nonlinear reduction 6. Concluding remarks.
  3. 3.

    • Identifying and describing sources of uncertainties. • Calibrating models/experiments.

    • Propagating uncertainties in the inputs. • Optimizing the output of scientific/engineering processes characterized by uncertain parameters. INTRODUCTION WHAT IS UQ?
  4. 4.

    Hertzian contact model dynamical system EXAMPLES Ref. - Tripathy, Rohit,

    Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223. Fig.: Propagation of solitary waves (solitons) in granular crystals
  5. 5.

    NACA12 airfoil Turbulent flow CFD solver O ptim ization under

    uncertainty Ref. - Seshadri, Pranay, et al. "A density-matching approach for optimization under uncertainty." Computer Methods in Applied Mechanics and Engineering 305 (2016): 562-578. EXAMPLES
  6. 6.

    Catalytic conversion of nitrate to nitrogen – calibration of reaction

    rate coefficients Identification of pollutant source– calibration of source term in system of PDEs Ref. - Tsilifis, Panagiotis, et al. "Computationally efficient variational approximations for Bayesian inverse problems." Journal of Verification, Validation and Uncertainty Quantification 1.3 (2016): 031004. EXAMPLES
  7. 8.

    NUMERICAL SIMULATOR INPUT PARAMETERS QUANTITY OF INTEREST f q 8

    MATHEMATICAL SETUP • f is expensive to evaluate. • is high-dimensional, i.e. ∈ ℝ!, D ≫ 1. • q is scalar.
  8. 11.

    THE SURROGATE APPROACH • Perform a finite number of simulations.

    • Solve the UQ problem with the surrogate. • Replace model with an approximation: 11 • Repeated evaluation of the likelihood is expensive. THIS IS DIFFICULT IN HIGH- DIMENSIONS !
  9. 14.

    14 GAUSSIAN PROCESS REGRESSION Statistical model: Prior GP: Data: Posterior

    GP: Posterior mean: Posterior variance: Prior GP Bayes rule Posterior GP
  10. 15.

    15 GPR - INFERENCE INFERENCE PROBLEM – Estimate the posterior

    distribution of the hyperparameters. Ref. - Rasmussen, Carl Edward. "Gaussian processes in machine learning." Advanced lectures on machine learning. Springer, Berlin, Heidelberg, 2004. 63-71.. Hyperparameter prior: Hyperparameter posterior: Approximation of the posterior at the mode: Marginal likelihood:
  11. 16.

    CLASSICAL ACTIVE SUBSPACE Ref. - Constantine, Paul G., Eric Dow,

    and Qiqi Wang. "Active subspace methods in theory and practice: applications to kriging surfaces." SIAM Journal on Scientific Computing 36.4 (2014): A1500-A1524. Probability Measure: Sample gradients: Centered covariance: Spectral decomposition: Final surrogate: How do we get these? 16
  12. 17.

    GRADIENT-FREE ACTIVE SUBSPACE RECOVERY Data: Surrogate: Optimize: 17 Tripathy, Rohit,

    Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223.
  13. 18.

    GRADIENT-FREE ACTIVE SUBSPACE RECOVERY Data: Surrogate: Hyperparameters Gaussian Process Bayes

    rule Modified kernel Any standard kernel Likelihood noise Negative log marginal likelihood 18
  14. 19.

    PDE: Uncertainty: Quantity of interest: Ω Γ1 Γ2 19 Tripathy,

    Rohit, Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223. STOCHASTIC ELLIPTIC PDE
  15. 20.

    Surrogate modelling task: We need to construct the map that

    links the vector of uncertain parameters to the quantity of interest q. Find: With data**: High dimensional inputs, small sample set. **Data source: https://github.com/paulcon/as-data-sets/tree/master/Elliptic_PDE STOCHASTIC ELLIPTIC PDE 20 Tripathy, Rohit, Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223.
  16. 21.

    Fig: Comparison of 1-d active subspace from gradient-based and gradient-free

    approach. Active subspace recovered using gradients Active subspace recovered without access to gradients (our approach) NOTICE : We recover the right active subspace upto arbitrary rotations ! 21 Tripathy, Rohit, Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223. STOCHASTIC ELLIPTIC PDE
  17. 22.

    Fig: Comparison of predictive accuracy between gradient-based and gradient-free approach.

    Surrogate model predicted vs observed values on test dataset (gradient-based approach) Surrogate model predicted vs observed values on test dataset (gradient-free approach) 22 Tripathy, Rohit, Ilias Bilionis, and Marcial Gonzalez. "Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation." Journal of Computational Physics 321 (2016): 191-223. STOCHASTIC ELLIPTIC PDE
  18. 24.
  19. 25.

    Fig.: Schematic of a DNN DEEP NEURAL NETWORKS Trainable parameters:

    Training data: Mini-batch data loss: Discrepancy / log likelihood Regularizer / Log prior Stochastic Gradient descent update: 25
  20. 26.

    ACTIVE SUBSPACE - REVISITED Ref. - Constantine, Paul G., Eric

    Dow, and Qiqi Wang. "Active subspace methods in theory and practice: applications to kriging surfaces." SIAM Journal on Scientific Computing 36.4 (2014): A1500-A1524. Probability Measure: Sample gradients: Centered covariance: Spectral decomposition: Final surrogate: 26
  21. 27.

    ACTIVE SUBSPACE RECOVERY Orthogonal projection GRAM-SCHMIDT ORTHOGONALIZATION W = h(Q)

    2 RD⇥d <latexit sha1_base64="9S+6LBM+NNL49YKd9aan/pE89oE=">AAACH3icbVBNS8NAEN3Ur1q/oh69LBahXkoiol6Egh701or9gCaWzXbTLt1swu5GKCH/xIt/xYsHRcRb/42bNoK2Phh482aGmXlexKhUljUxCkvLK6trxfXSxubW9o65u9eSYSwwaeKQhaLjIUkY5aSpqGKkEwmCAo+Rtje6yurtRyIkDfm9GkfEDdCAU59ipLTUM8+cAKmh5yftFF7CYeUnbaTH0KEcznIvuUsfkmvoKBoQCftpzyxbVWsKuEjsnJRBjnrP/HL6IY4DwhVmSMqubUXKTZBQFDOSlpxYkgjhERqQrqYc6T1uMv0vhUda6UM/FDq4glP190SCAinHgac7s3PlfC0T/6t1Y+VfuAnlUawIx7NFfsygCmFmFuxTQbBiY00QFlTfCvEQCYSVtrSkTbDnX14krZOqbVXtxmm5dpvbUQQH4BBUgA3OQQ3cgDpoAgyewAt4A+/Gs/FqfBifs9aCkc/sgz8wJt83TKJ7</latexit> <latexit sha1_base64="9S+6LBM+NNL49YKd9aan/pE89oE=">AAACH3icbVBNS8NAEN3Ur1q/oh69LBahXkoiol6Egh701or9gCaWzXbTLt1swu5GKCH/xIt/xYsHRcRb/42bNoK2Phh482aGmXlexKhUljUxCkvLK6trxfXSxubW9o65u9eSYSwwaeKQhaLjIUkY5aSpqGKkEwmCAo+Rtje6yurtRyIkDfm9GkfEDdCAU59ipLTUM8+cAKmh5yftFF7CYeUnbaTH0KEcznIvuUsfkmvoKBoQCftpzyxbVWsKuEjsnJRBjnrP/HL6IY4DwhVmSMqubUXKTZBQFDOSlpxYkgjhERqQrqYc6T1uMv0vhUda6UM/FDq4glP190SCAinHgac7s3PlfC0T/6t1Y+VfuAnlUawIx7NFfsygCmFmFuxTQbBiY00QFlTfCvEQCYSVtrSkTbDnX14krZOqbVXtxmm5dpvbUQQH4BBUgA3OQQ3cgDpoAgyewAt4A+/Gs/FqfBifs9aCkc/sgz8wJt83TKJ7</latexit> <latexit sha1_base64="9S+6LBM+NNL49YKd9aan/pE89oE=">AAACH3icbVBNS8NAEN3Ur1q/oh69LBahXkoiol6Egh701or9gCaWzXbTLt1swu5GKCH/xIt/xYsHRcRb/42bNoK2Phh482aGmXlexKhUljUxCkvLK6trxfXSxubW9o65u9eSYSwwaeKQhaLjIUkY5aSpqGKkEwmCAo+Rtje6yurtRyIkDfm9GkfEDdCAU59ipLTUM8+cAKmh5yftFF7CYeUnbaTH0KEcznIvuUsfkmvoKBoQCftpzyxbVWsKuEjsnJRBjnrP/HL6IY4DwhVmSMqubUXKTZBQFDOSlpxYkgjhERqQrqYc6T1uMv0vhUda6UM/FDq4glP190SCAinHgac7s3PlfC0T/6t1Y+VfuAnlUawIx7NFfsygCmFmFuxTQbBiY00QFlTfCvEQCYSVtrSkTbDnX14krZOqbVXtxmm5dpvbUQQH4BBUgA3OQQ3cgDpoAgyewAt4A+/Gs/FqfBifs9aCkc/sgz8wJt83TKJ7</latexit> 1. 2. 3. 4. Deep Neural Network Network weights/biases 27
  22. 29.

    STOCHASTIC ELLIPTIC PDE Table : Test set RMSE comparison for

    long and short correlation length cases. Short correlation length ( 0.01 ) Long correlation length ( 1.0 ) 29
  23. 31.

    To be determined from data: • L: number of layers

    • d: number of nodes in last layer before output PARAMETERIZED ARCHITECTURE 31 Width of kth layer: Uniquely determined by L and d Deep Neural Network Network weights/biases D
  24. 32.

    MODEL SELECTION - WORKFLOW 32 Generate grid of L and

    d Bayesian global optimization for λ at all grid locations Use best estimate of λ, L and d to train final network. EMBARASSINGLY PARALLELIZABLE ! We need to pick: 1. The number of layers, L. 2. The size of the active subspace encoding, d. 3. The regularization constant, λ.
  25. 33.

    33 u = 0, 8x1 = 1, u = 1,

    8x1 = 0, @u @n = 0, 8x2 = 1. PDE: Boundary conditions: Uncertain diffusion: log Exponential covariance: STOCHASTIC ELLIPTIC PDE SETUP
  26. 35.

    35 o2000 random conductivities generated by: • Sampling length-scales [1].

    • Sampling the conductivity on a 32x32 grid. oFor each one of the sampled conductivities, we solved the PDE on a 32x32 grid. u(x, ✓) <latexit sha1_base64="ErE6xeO1oqTds87eAIYJzhu3d/o=">AAACJXicbVDLSgMxFM3UV62vqks3wSIoSJkRQRcuim5cKlgtdErJZO60oZkHyR2xDPMzbvwVNy4sIrjyV8zUgtp6IOTk3EfuPV4ihUbb/rBKc/MLi0vl5crK6tr6RnVz61bHqeLQ5LGMVctjGqSIoIkCJbQSBSz0JNx5g4sifncPSos4usFhAp2Q9SIRCM7QSN3q2Z4bMux7QfaQV9L9n8chzdxx+0yBn1PXi6Wvh6G5Mhf7gCzPD7rVml23x6CzxJmQGpngqlsduX7M0xAi5JJp3XbsBDsZUyi4hLziphoSxgesB21DIxaC7mTjMXK6ZxSfBrEyJ0I6Vn9XZCzUxYQms9hCT8cK8b9YO8XgtJOJKEkRIv79UZBKijEtLKO+UMBRDg1hXAkzK+V9phhHY2zFmOBMrzxLbo/qjl13ro9rjfOJHWWyQ3bJPnHICWmQS3JFmoSTR/JMXsnIerJerDfr/Tu1ZE1qtskfWJ9fggumgw==</latexit> <latexit sha1_base64="ErE6xeO1oqTds87eAIYJzhu3d/o=">AAACJXicbVDLSgMxFM3UV62vqks3wSIoSJkRQRcuim5cKlgtdErJZO60oZkHyR2xDPMzbvwVNy4sIrjyV8zUgtp6IOTk3EfuPV4ihUbb/rBKc/MLi0vl5crK6tr6RnVz61bHqeLQ5LGMVctjGqSIoIkCJbQSBSz0JNx5g4sifncPSos4usFhAp2Q9SIRCM7QSN3q2Z4bMux7QfaQV9L9n8chzdxx+0yBn1PXi6Wvh6G5Mhf7gCzPD7rVml23x6CzxJmQGpngqlsduX7M0xAi5JJp3XbsBDsZUyi4hLziphoSxgesB21DIxaC7mTjMXK6ZxSfBrEyJ0I6Vn9XZCzUxYQms9hCT8cK8b9YO8XgtJOJKEkRIv79UZBKijEtLKO+UMBRDg1hXAkzK+V9phhHY2zFmOBMrzxLbo/qjl13ro9rjfOJHWWyQ3bJPnHICWmQS3JFmoSTR/JMXsnIerJerDfr/Tu1ZE1qtskfWJ9fggumgw==</latexit> <latexit sha1_base64="ErE6xeO1oqTds87eAIYJzhu3d/o=">AAACJXicbVDLSgMxFM3UV62vqks3wSIoSJkRQRcuim5cKlgtdErJZO60oZkHyR2xDPMzbvwVNy4sIrjyV8zUgtp6IOTk3EfuPV4ihUbb/rBKc/MLi0vl5crK6tr6RnVz61bHqeLQ5LGMVctjGqSIoIkCJbQSBSz0JNx5g4sifncPSos4usFhAp2Q9SIRCM7QSN3q2Z4bMux7QfaQV9L9n8chzdxx+0yBn1PXi6Wvh6G5Mhf7gCzPD7rVml23x6CzxJmQGpngqlsduX7M0xAi5JJp3XbsBDsZUyi4hLziphoSxgesB21DIxaC7mTjMXK6ZxSfBrEyJ0I6Vn9XZCzUxYQms9hCT8cK8b9YO8XgtJOJKEkRIv79UZBKijEtLKO+UMBRDg1hXAkzK+V9phhHY2zFmOBMrzxLbo/qjl13ro9rjfOJHWWyQ3bJPnHICWmQS3JFmoSTR/JMXsnIerJerDfr/Tu1ZE1qtskfWJ9fggumgw==</latexit> STOCHASTIC ELLIPTIC PDE DATA [1] - Rohit K Tripathy and Ilias Bilionis. Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification. Journal of Computational Physics, 375:565–588, 2018.
  27. 36.

    36 STOCHASTIC ELLIPTIC PDE MODEL SELECTION Lowest Validation error (and

    selected DNN) Bayesian Global Optimization Negative Validation Error A different DNN trained with ADAM.
  28. 37.

    37 STOCHASTIC ELLIPTIC PDE PREDICTION Fig.: (From left column to

    right) (a) Input random field; (b) True (FEM) solution; (c) Predicted solution. Fig.: (Top) Histogram of R2 across test dataset. (Bottom) Histogram of relative errors across test dataset.
  29. 38.

    PROPAGATING INPUT UNCERTAINTY SPDE WITH FIXED LENGTHSCALE 38 Fig.: Comparison

    of solution pdf at x = (0.484,0.484) obtained from MCS* and DNN surrogate. Fig. : Comparison of Monte Carlo* (left) mean and variance and surrogate (right) mean and variance for the PDE solution. FEM Surrogate
  30. 39.

    PROPAGATING INPUT UNCERTAINTY SPDE WITH RANDOM LENGTHSCALE 39 Fig.: Comparison

    of solution pdf at x = (0.484,0.484) obtained from MCS* and DNN surrogate. Fig. : Comparison of Monte Carlo* (left) mean and variance and surrogate (right) mean and variance for the PDE solution.
  31. 40.

    CONCLUDING REMARKS § Monte Carlo approach to UQ – infeasible

    due to slow convergence. § Surrogate approach to UQ – constrained by high-dimensionality and cost of numerical simulator. § Strategy – find suitable low-rank approximation of the stochastic parameter space: • Optimal linear projections for capturing directions of maximal variation of QoI gradient (active subspace). • Nonlinear manifold capturing QoI variability (deep neural networks). § Limitations with proposed approaches – • Scalability concerns with GPs. • Incorporation of rigorous priors (continuity/smoothness etc.) in DNNs. • Incorporation of physical constraints. 40
  32. 41.

    CONCLUDING REMARKS § Ongoing/future work – • Encoding physical laws

    into statistical models (physics-informed machine learning). • Synthesizing information from multiple sources of varying levels of fidelity (multifidelity surrogate modeling & uncertainty quantification). • Exploiting group-theoretic structure in dynamical systems. • Developing surrogates robust to non-stationarities, discontinuities etc. 41
  33. 42.

    REFERENCES 42 [1] Rohit Tripathy, Ilias Bilionis, and Marcial Gonzalez.

    Gaussian processes with built-in dimensionality re- duction: Applications to high-dimensional uncertainty propagation. Journal of Computational Physics, 321:191–223, 2016. [2] Rohit K Tripathy and Ilias Bilionis. Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification. Journal of Computational Physics, 375:565–588, 2018. [3] Rohit Tripathy and Ilias Bilionis. Deep active subspaces–a scalable method for high-dimensional uncertainty propagation. arXiv preprint arXiv:1902.10527, 2019. [4] Rohit Tripathy and Ilias Bilionis. Learning deep neural network (DNN) surrogate models for UQ. SIAM UQ, 2018. [5] Sharmila Karumuri, Rohit Tripathy, Ilias Bilionis, and Jitesh Panchal. Simulator-free solution of high- dimensional stochastic elliptic partial differential equations using deep neural networks. arXiv preprint arXiv:1902.05200, 2019.