Gradient-free active subspace recovery using deep neural networks - application to high-dimensional uncertainty quantification

Transcript

1. GRADIENT-FREE ACTIVE SUBSPACE RECOVERY USING DEEP NEURAL NETWORKS - APPLICATION TO HIGH-DIMENSIONAL UNCERTAINTY QUANTIFICATION Rohit Tripathy and Ilias Bilionis Predictive Science Lab http://www.predictivesciencelab.org/ Purdue University West Lafayette, IN, USA 1
2. UNCERTAINTY PROPAGATION - f is some scalar quantity of interest. - Obtained numerically through the solution of a set of PDEs. - Inputs x – uncertain and high dimensional. - Interested in quantifying the uncertainty in f. Image sources: [1] - Left image. [2] - Right image.. 2
3. UNCERTAINTY PROPAGATION Input uncertainty: QoI density: QoI mean: QoI variance: Formally: 3
4. • Do a finite number of simulations. • Replace model with an approximation: • The surrogate is usually cheap to evaluate. • Solve the UQ problem with the surrogate. THE SURROGATE IDEA y ⇡ ˆ f(x) 4
5. TECHNIQUES FOR DIMENSIONALITY REDUCTION • Truncated Karhunen-Loeve Expansion (also known as Linear Principal Component analysis)[1]. • Kernel PCA[2]. (Non-linear model reduction). • Latent variable models ( GPLVM[3], VAE[4] etc.) References: [1]- Ghanem and Spanos. Stochastic finite elements: a spectral approach (2003). [2]-Ma and Zabaras. Kernel principal component analysis for stochastic input model generation. (2011). [3]- Lawrence. Gaussian process latent variable models for visualisation of high dimensional data. (2004). [4]- Kingma and Ba. Auto-encoding variational bayes. (2013). 5
6. CLASSIC ACTIVE SUBSPACE 2. Sample gradients: 3. Empirical covariance: 4. Eigendecomposition: 5. Active-inactive subspace separation: 6. Projection: 7. Regression: 1. Input prob. measure: Ref. - Constantine, Paul G., Eric Dow, and Qiqi Wang. "Active subspace methods in theory and practice: applications to kriging surfaces." SIAM Journal on Scientific Computing 36.4 (2014): A1500-A1524. 6
7. ACTIVE SUBSPACE RECOVERY f(x) = g(z) = g(WT x) <latexit sha1_base64="vHFD1s+rYs+3erHirY/5CHTtg6I=">AAACMnicbVDLSsNAFJ3UV62vqEs3g0Wom5KIoBuh4MbuKvQFTSyTyaQdOnkwMxFryDe58UsEF7pQxK0f4SRtobYeGDiccy/3zHEiRoU0jDetsLK6tr5R3Cxtbe/s7un7B20RxhyTFg5ZyLsOEoTRgLQklYx0I06Q7zDScUbXmd+5J1zQMGjKcURsHw0C6lGMpJL6et2rWD6SQ8dLHtJTeAUHFTgTHlM4URIrP5Rw4qYzs5PeNVM4t9vXy0bVyAGXiTklZTBFo6+/WG6IY58EEjMkRM80ImkniEuKGUlLVixIhPAIDUhP0QD5RNhJniSFJ0pxoRdy9QIJc3V+I0G+EGPfUZNZRLHoZeJ/Xi+W3qWd0CCKJQnw5JAXMyhDmPUHXcoJlmysCMKcqqwQDxFHWKqWS6oEc/HLy6R9VjWNqnl7Xq7Vp3UUwRE4BhVgggtQAzegAVoAgyfwCj7Ap/asvWtf2vdktKBNdw7BH2g/vzynqj4=</latexit> <latexit sha1_base64="vHFD1s+rYs+3erHirY/5CHTtg6I=">AAACMnicbVDLSsNAFJ3UV62vqEs3g0Wom5KIoBuh4MbuKvQFTSyTyaQdOnkwMxFryDe58UsEF7pQxK0f4SRtobYeGDiccy/3zHEiRoU0jDetsLK6tr5R3Cxtbe/s7un7B20RxhyTFg5ZyLsOEoTRgLQklYx0I06Q7zDScUbXmd+5J1zQMGjKcURsHw0C6lGMpJL6et2rWD6SQ8dLHtJTeAUHFTgTHlM4URIrP5Rw4qYzs5PeNVM4t9vXy0bVyAGXiTklZTBFo6+/WG6IY58EEjMkRM80ImkniEuKGUlLVixIhPAIDUhP0QD5RNhJniSFJ0pxoRdy9QIJc3V+I0G+EGPfUZNZRLHoZeJ/Xi+W3qWd0CCKJQnw5JAXMyhDmPUHXcoJlmysCMKcqqwQDxFHWKqWS6oEc/HLy6R9VjWNqnl7Xq7Vp3UUwRE4BhVgggtQAzegAVoAgyfwCj7Ap/asvWtf2vdktKBNdw7BH2g/vzynqj4=</latexit> <latexit sha1_base64="vHFD1s+rYs+3erHirY/5CHTtg6I=">AAACMnicbVDLSsNAFJ3UV62vqEs3g0Wom5KIoBuh4MbuKvQFTSyTyaQdOnkwMxFryDe58UsEF7pQxK0f4SRtobYeGDiccy/3zHEiRoU0jDetsLK6tr5R3Cxtbe/s7un7B20RxhyTFg5ZyLsOEoTRgLQklYx0I06Q7zDScUbXmd+5J1zQMGjKcURsHw0C6lGMpJL6et2rWD6SQ8dLHtJTeAUHFTgTHlM4URIrP5Rw4qYzs5PeNVM4t9vXy0bVyAGXiTklZTBFo6+/WG6IY58EEjMkRM80ImkniEuKGUlLVixIhPAIDUhP0QD5RNhJniSFJ0pxoRdy9QIJc3V+I0G+EGPfUZNZRLHoZeJ/Xi+W3qWd0CCKJQnw5JAXMyhDmPUHXcoJlmysCMKcqqwQDxFHWKqWS6oEc/HLy6R9VjWNqnl7Xq7Vp3UUwRE4BhVgggtQAzegAVoAgyfwCj7Ap/asvWtf2vdktKBNdw7BH2g/vzynqj4=</latexit> Orthogonal projection GRAM-SCHMIDT ORTHOGONALIZATION W = h(Q) 2 RD⇥d <latexit sha1_base64="9S+6LBM+NNL49YKd9aan/pE89oE=">AAACH3icbVBNS8NAEN3Ur1q/oh69LBahXkoiol6Egh701or9gCaWzXbTLt1swu5GKCH/xIt/xYsHRcRb/42bNoK2Phh482aGmXlexKhUljUxCkvLK6trxfXSxubW9o65u9eSYSwwaeKQhaLjIUkY5aSpqGKkEwmCAo+Rtje6yurtRyIkDfm9GkfEDdCAU59ipLTUM8+cAKmh5yftFF7CYeUnbaTH0KEcznIvuUsfkmvoKBoQCftpzyxbVWsKuEjsnJRBjnrP/HL6IY4DwhVmSMqubUXKTZBQFDOSlpxYkgjhERqQrqYc6T1uMv0vhUda6UM/FDq4glP190SCAinHgac7s3PlfC0T/6t1Y+VfuAnlUawIx7NFfsygCmFmFuxTQbBiY00QFlTfCvEQCYSVtrSkTbDnX14krZOqbVXtxmm5dpvbUQQH4BBUgA3OQQ3cgDpoAgyewAt4A+/Gs/FqfBifs9aCkc/sgz8wJt83TKJ7</latexit> <latexit sha1_base64="9S+6LBM+NNL49YKd9aan/pE89oE=">AAACH3icbVBNS8NAEN3Ur1q/oh69LBahXkoiol6Egh701or9gCaWzXbTLt1swu5GKCH/xIt/xYsHRcRb/42bNoK2Phh482aGmXlexKhUljUxCkvLK6trxfXSxubW9o65u9eSYSwwaeKQhaLjIUkY5aSpqGKkEwmCAo+Rtje6yurtRyIkDfm9GkfEDdCAU59ipLTUM8+cAKmh5yftFF7CYeUnbaTH0KEcznIvuUsfkmvoKBoQCftpzyxbVWsKuEjsnJRBjnrP/HL6IY4DwhVmSMqubUXKTZBQFDOSlpxYkgjhERqQrqYc6T1uMv0vhUda6UM/FDq4glP190SCAinHgac7s3PlfC0T/6t1Y+VfuAnlUawIx7NFfsygCmFmFuxTQbBiY00QFlTfCvEQCYSVtrSkTbDnX14krZOqbVXtxmm5dpvbUQQH4BBUgA3OQQ3cgDpoAgyewAt4A+/Gs/FqfBifs9aCkc/sgz8wJt83TKJ7</latexit> <latexit sha1_base64="9S+6LBM+NNL49YKd9aan/pE89oE=">AAACH3icbVBNS8NAEN3Ur1q/oh69LBahXkoiol6Egh701or9gCaWzXbTLt1swu5GKCH/xIt/xYsHRcRb/42bNoK2Phh482aGmXlexKhUljUxCkvLK6trxfXSxubW9o65u9eSYSwwaeKQhaLjIUkY5aSpqGKkEwmCAo+Rtje6yurtRyIkDfm9GkfEDdCAU59ipLTUM8+cAKmh5yftFF7CYeUnbaTH0KEcznIvuUsfkmvoKBoQCftpzyxbVWsKuEjsnJRBjnrP/HL6IY4DwhVmSMqubUXKTZBQFDOSlpxYkgjhERqQrqYc6T1uMv0vhUda6UM/FDq4glP190SCAinHgac7s3PlfC0T/6t1Y+VfuAnlUawIx7NFfsygCmFmFuxTQbBiY00QFlTfCvEQCYSVtrSkTbDnX14krZOqbVXtxmm5dpvbUQQH4BBUgA3OQQ3cgDpoAgyewAt4A+/Gs/FqfBifs9aCkc/sgz8wJt83TKJ7</latexit> 1. 2. 3. 4. 7
8. SYNTHETIC EXAMPLE 1-D active subspace 2-D active subspace 8 f(x) = g(z) = ↵ + T z + zT z | {z } Link Function , z = WT x, x 2 R50 <latexit sha1_base64="xrvF/1tvERvN3/IbB9mHquUcBCE=">AAACzHicbVFdixMxFM2MH7uOX1UffQkWYUUpM6KsLwsLgrggssp2u9Dplps004YmmSHJSGvIqz/QN5/9I2ampY673jBwzrnncu/cSyrBjU3TX1F84+at23v7d5K79+4/eNh79PjclLWmbEhLUeoLAoYJrtjQcivYRaUZSCLYiCzfN/nRN6YNL9WZXVdsImGueMEp2CBNe7+Lg1yCXZDCrfwLfITnO/695Ri7vG3jiKiZx3mtZkwTDZS5HES1APwS54RZuDzDf0sbcUeazBykhK7BT13LtHSfuFrm+EOtaDNUSIX3Ku+aj3Zk5Dt9Vr5jW4XhuNow4r76S/c29UmS4Gmvnw7SNvB1kG1BH23jdNr7mc9KWkumLBVgzDhLKztxoC2ngvkkrw2rgC5hzsYBKpDMTFy7JY+fB2WGi1KHT1ncqt0KB9KYtSTB2YxqruYa8X+5cW2LdxPHVVVbpuimUVELbEvcXBbPuGbUinUAQDUPs2K6gHAoG+6fhCVkV3/5Ojh/PcjSQfblTf/4ZLuOffQUPUMHKEOH6Bh9RKdoiGh0EpXRKlrHn2Mbu9hvrHG0rXmC/on4xx/Zbd9R</latexit> <latexit sha1_base64="xrvF/1tvERvN3/IbB9mHquUcBCE=">AAACzHicbVFdixMxFM2MH7uOX1UffQkWYUUpM6KsLwsLgrggssp2u9Dplps004YmmSHJSGvIqz/QN5/9I2ampY673jBwzrnncu/cSyrBjU3TX1F84+at23v7d5K79+4/eNh79PjclLWmbEhLUeoLAoYJrtjQcivYRaUZSCLYiCzfN/nRN6YNL9WZXVdsImGueMEp2CBNe7+Lg1yCXZDCrfwLfITnO/695Ri7vG3jiKiZx3mtZkwTDZS5HES1APwS54RZuDzDf0sbcUeazBykhK7BT13LtHSfuFrm+EOtaDNUSIX3Ku+aj3Zk5Dt9Vr5jW4XhuNow4r76S/c29UmS4Gmvnw7SNvB1kG1BH23jdNr7mc9KWkumLBVgzDhLKztxoC2ngvkkrw2rgC5hzsYBKpDMTFy7JY+fB2WGi1KHT1ncqt0KB9KYtSTB2YxqruYa8X+5cW2LdxPHVVVbpuimUVELbEvcXBbPuGbUinUAQDUPs2K6gHAoG+6fhCVkV3/5Ojh/PcjSQfblTf/4ZLuOffQUPUMHKEOH6Bh9RKdoiGh0EpXRKlrHn2Mbu9hvrHG0rXmC/on4xx/Zbd9R</latexit> <latexit sha1_base64="xrvF/1tvERvN3/IbB9mHquUcBCE=">AAACzHicbVFdixMxFM2MH7uOX1UffQkWYUUpM6KsLwsLgrggssp2u9Dplps004YmmSHJSGvIqz/QN5/9I2ampY673jBwzrnncu/cSyrBjU3TX1F84+at23v7d5K79+4/eNh79PjclLWmbEhLUeoLAoYJrtjQcivYRaUZSCLYiCzfN/nRN6YNL9WZXVdsImGueMEp2CBNe7+Lg1yCXZDCrfwLfITnO/695Ri7vG3jiKiZx3mtZkwTDZS5HES1APwS54RZuDzDf0sbcUeazBykhK7BT13LtHSfuFrm+EOtaDNUSIX3Ku+aj3Zk5Dt9Vr5jW4XhuNow4r76S/c29UmS4Gmvnw7SNvB1kG1BH23jdNr7mc9KWkumLBVgzDhLKztxoC2ngvkkrw2rgC5hzsYBKpDMTFy7JY+fB2WGi1KHT1ncqt0KB9KYtSTB2YxqruYa8X+5cW2LdxPHVVVbpuimUVELbEvcXBbPuGbUinUAQDUPs2K6gHAoG+6fhCVkV3/5Ojh/PcjSQfblTf/4ZLuOffQUPUMHKEOH6Bh9RKdoiGh0EpXRKlrHn2Mbu9hvrHG0rXmC/on4xx/Zbd9R</latexit>
9. ELLIPTIC PDE B.C.s: PDE: Uncertain diffusion coefficient: Quantity of interest: Exponential kernel: 9
10. Uncertainty propagation problem: What are the statistical properties of the quantity of interest q given the uncertainty in the diffusion coefficient, a? We need a `surrogate’ that maps xi’s to q, i.e., Karhunen Loeve decomposition of diffusion coeff. a: Find: With data**: High dimensional inputs, small sample set. ELLIPTIC PDE 10
11. ELLIPTIC PDE 11 Table : Test set RMSE comparison for long and short correlation length cases. Short correlation length ( 0.01 ) Long correlation length ( 1.0 )
12. CONCLUSION 12 • Classical approach to active subspace requires computation of gradients. • Simple reparameterization of the first layer of a DNN – gradient-free recovery. • Useful in a wide variety of applications where model QoIs have shown to possess ridge structure. • Drawback – Reparameterization makes SGD in the Stiefel manifold more challenging. Potential cure – better initialization scheme.