Slide 10
Slide 10 text
APPROACHES TO SCALE GAUSSIAN PROCESSES
10
§ Inducing point/Pseudo-input methods (Sparse Gaussian process regression)
• ``Summarize” N training data points with M training data points (or pseudo-inputs) – different choices of pseudo-
inputs lead to different Sparse GP strategies (M << N).
• Reduce training cost from O(N3) to O(NM2) and storage cost from O(N2) to O(NM).
• Do not require special structure in the data, i.e., meant to be usable ‘off-the-shelf’.
• Key examples – Snelson, Gahramani (2006)[1], Titsias (2009)[2] (and many others).
§ Structure-exploiting approaches:
• Exploit the fact the dataset is generated with some meaningful structure (such as grid inputs).
• Main types of structure – Kronecker[3] and Toeplitz[4].
• Kronecker structure arises when inputs are on a multidimensional lattice; Toeplitz structure arises when the data
lies on a regularly spaced 1D – making such methods highly useful for some scientific applications (thinking
surrogate models for PDE solvers) but not so much for general purpose datasets.
Ref.:
1. Snelson, Edward and Ghahramani, Zoubin. Sparse Gaussian processes using pseudo-inputs. In Advances in neural information processing systems (NIPS), volume 18, pp. 1257. MIT Press, 2006.
2. Titsias, Michalis. Variational learning of inducing variables in sparse Gaussian processes. In Artificial Intelligence and Statistics, pp. 567-574. 2009.
3. Saatçi, Yunus. Scalable inference for structured Gaussian process models. PhD thesis., University of Cambridge, 2012.
4. Cunningham, John P., Krishna V. Shenoy, and Maneesh Sahani. Fast Gaussian process methods for point process intensity estimation. In Proceedings of the 25th international conference on
Machine learning, pp. 192-199. ACM, 2008.