Semidefinite Programming: Exploiting Data Structure

progress report on some simple exploitations used to speed up SDP solvers in specific domains

Transcript

1. Semidefinite Programming: Exploiting Data Structure Matthew Barga 20 December 2012 1 / 12
2. 1 Recap 2 Investigations 3 Conclusion 2 / 12
3. SDP and multikernel SVM SDP Problem In multikernel SVM construction, we have a linear objective to maximize in order to get a combined kernel matrix Target optimization problem is a set of linear combinations of fixed kernel matrices that are computed beforehand K = i µi Ki , µ ≥ 0, K 0 2 / 12
4. Last meeting Negative values can and do come up in cholesky factorization in multiple kernel learning SVM In the software packages I have come across, all simply mark the factorization as failed if a negative value is encountered in the radical SHOGUN uses cholesky factorization from “Numerical Recipes in C++” Why? 3 / 12
5. Considerations Most MKL routines do not require high precision libraries (depending on application) rather, large number libraries or routines to deal with large dimensional data are needed more precision needed in some SVM variations (regression, etc.) SHOGUN supports up to 96-bit floating point precision 4 / 12
6. Connection to multikernel SVM Many implementations relax to a Quadratically Constrained Quadratic Programming problem by introducing some further constraints (SHOGUN, etc.) 5 / 12
7. Connection to multikernel SVM Without restriction on µ, complexity is O((m + ntr )2n2) With added restriction to µ, the optimization problem becomes a QCQP problem, which is a special case of SOCP Complexity is O(mn3) SDPARA used similar constraints to achieve an equivalent complexity 6 / 12
8. Bottlenecks Schur complement matrix Cholesky factorization 7 / 12
9. Exploitations of data structure Three exploitations are most common ‘chordal’ structure: sparsity patterns (some automatic optimization in solvers for this) low rank: schur complement can be formed more efficiently algebraic symmetry: data matrices belong to matrix *-algebra 8 / 12
10. Exploitations of data structure Control theory is one of the most targeted areas of SDP optimization There are two large exploitations that are most commonly used 1 a specific type of constraint that when removed, greatly simplifies calculation through use of Lagrangian relaxation, the optimization problem is decomposed into smaller ones that are solved independently 2 low rank structure in basis matrices (from the input data) by exploiting this structure can efficiently determine search direction 9 / 12
11. Summary Most multikernel problems reduce to QCQP problem The only current exploitation of structure in MKL-SVM problems is sparsity 10 / 12
12. Current and Future Work Relate restrictions in SDPARA and for SVM multikernel (same computational complexity) Understanding nature of translation between SVM and SDP/QCQP Automated analysis and recognition of patterns in input data? How does exploitation relate to memory optimization/ communication? Application domains Alternatives to SDP: replacement of SDP with other optimizations bypass the solver; optimize the classifier and kernel combination factors together 11 / 12
13. Cited Works 12 / 12