Introduction to Sparse Modeling

227382dbd5e033db211c159edf32853c?s=47 Hacarus Inc.
September 19, 2019

Introduction to Sparse Modeling

Presentation Slides at 如何開發可解釋的AI
https://www.eventbrite.com/e/ai-aihacarus-tickets-68588170063

227382dbd5e033db211c159edf32853c?s=128

Hacarus Inc.

September 19, 2019
Tweet

Transcript

  1. 如何開發可解釋的AI Introduction to Sparse Modeling Takashi Someda CTO, Hacarus Inc.

    September 19th, 2019 LIGHTWEIGHT & EXPLAINABLE AI
  2. About Me Takashi Someda, @tksmd Director/CTO at HACARUS Inc. Master’s

    degree of Information Science at Kyoto University Sun Microsystems K.K. → Founded own startup at Kyoto → Established Kyoto Branch of Nulab Inc. → Current
  3. Company and Team RAISED: USD 3.7 Million (Series A) FOUNDED:

    Jan, 2014 EMPLOYEES: 49 LOCATIONS: Kyoto (HQ), Tokyo, Manila CUSTOMER SUPPORT IN: Japanese, English, German, Tagalog, Chinese and Swedish ACADEMIC PARTNERS
  4. What HACARUS is doing Sparse Modeling Based AI On-Premises Cloud

    FPGA / Major CPUs Medical Manufacturing
  5. Today’s Takeaways • Basic concept of Sparse Modeling • Several

    concrete use cases / examples • Guide to Python implementation
  6. Blackbox Problem Blackbox AI Target Result Blackbox AI cannot answer

    questions like • Why do I get the result ? • When does it succeed/fail ? • How can I correct the result ? difficult to use AI even if it shows fascinating performance
  7. Approach to explainable AI • Post-hoc explain a given AI

    model • Individual prediction explanations • Global prediction explanations • Build an interpretable model • Logistic regression, Decision trees and so on For more details, refer to Explainable AI in Industry (KDD 2019 Tutorial)
  8. History of Sparse Modeling Year Paper Author 1996 Regression Shrinkage

    and Selection via Lasso R. Tibshirani 1996 Sparse Coding B. Olshausen 2006 Compressed Sensing D.L. Donoho 2018 Multi-Layer Convolutional Sparse Modeling M. Elad
  9. ZUBSHFUWBSJBCMF 9JOQVU = $ + & & + ( (

    + ) ) + * * + + + ⇒ find w to satisfy the condition above between y and X Linear Regression in sales forecast area(m^2) Distance from station(km) Regional Population Competitors Shop ID Products Revenue
  10. ⇒ Analytical solution cannot be found 884 = $ +

    33& + 1.7( + 41) + 0* + 38+ 554 = $ + 36& + 4.7( + 23) + 0* + 36+ 677 = $ + 29& + 3.2( + 46) + 0* + 36+ 478 = $ + 43& + 3.7( + 25) + 5* + 30+ If no sufficient data exists.. area(m^2) Distance from station(km) Regional Population Competitors Shop ID Products Revenue
  11. • Use minimum number of input features ⇒ Set as

    many weight values to 0 as possible Assume most of variables are irrelevant area(m^2) Distance from station(km) Regional Population Competitors Shop ID Products Revenue
  12. • Problem Settings • Output y can be expressed as

    linear combination of x with observation noise ε where x is m dimensional and sample size of y is n = & & + ⋯ + < < + Basic approach to the problem • Least squares method • Minimize least square errors of y and multipliers of x and estimated w min 1 2 − (
  13. What if data is not sufficient ? • Additional constraint

    will be introduced as regularization term • Objective function can be changed to the following form min 1 2 − ( + C ⇒ Regularization parameter λ controls effectiveness of regularization Introduce Regularization
  14. • L0 norm optimization • Use minimum number of x

    to satisfy equation • Find w to minimize number of non-zero elements • Combinational optimization • NP-hard and not feasible L L0 and L1 norm optimization • L1 norm optimization • Find w to minimize sum of its absolute values • Global solution can (still) be reached • Solved within practical time Relax constraint Least Absolute Shrinkage and Selection Operator
  15. 1. Initialize D = 1, … , with random value

    2. Update I D = K L MN L O , Where (D) = − ∑TUD T T and S is soft thresholding function 3. Repeat 2 until converging condition is satisfied Algorithm for Lasso Coordinate Descent
  16. • Shrink given value x by comparing it with S

    , = W − , ( ≥ ) 0, (− < < ) + , ( ≤ −) Soft thresholding operator − (, )
  17. # Soft thresholding operator def soft_threshold(X, thresh): return np.where(np.abs(X) <=

    thresh, 0, X - thresh * np.sign(X)) # Coordinate descent w_cd = np.zeros(n_features) for _ in range(n_iter): for j in range(n_features): w_cd[j] = 0.0 r_j = y - np.dot(X, w_cd) w_cd[j] = soft_threshold(np.dot(X[:, j], r_j) / n_samples, alpha) Code: Coordinate Descent
  18. X is 1,000 dimensional input features Only 20 features out

    of 1,000 are relevant to output y Only 100 samples are available Example: Numerical experiment
  19. Other Lasso variants Fused Lasso Trend Filtering constraint difference of

    two neighboring values Constraint two-fold differential of three neighboring values
  20. 1. Extract patches from images 2. Learn dictionary to express

    every patches 3. Every patches should be represented as sparse combination of dictionary basis Y: Image A: Dictionary [ ] ] X: coefficient Dictionary Learning
  21. Dictionary (8x8, 64 basis) Sparse Coding (green is 0) Example:

    Image Reconstruction Sparse Encode Reconstruction
  22. # extract patches patches = extract_simple_patches_2d(img, patch_size) # normalize patches

    patches = patches.reshape(patches.shape[0], -1).astype(np.float64) intercept = np.mean(patches, axis=0) patches -= intercept patches /= np.std(patches, axis=0) # dictionary learning model = MiniBatchDictionaryLearning(n_components=n_basis, alpha=1, n_iter=n_iter, n_jobs=1) model.fit(patches) # reconstruction reconstructed_patches = np.dot(code, model.components_) reconstructed_patches = reconstructed_patches.reshape(len(patches), *patch_size) reconstructed = reconstruct_from_simple_patches_2d(reconstructed_patches, img.shape) Code: Image Reconstruction
  23. Advanced use case of dictionary learning Inpainting Super Resolution

  24. Example: Inspection of solar panel cells Comparison of proposed methods(paper

    below) and SPECTRO by HACARUS https://arxiv.org/abs/1807.02894 Paper (SVM) Paper (CNN) SPECTRO Dataset 800 800 60 Training time 30 mins 5 hours 19 secs Inference time 8 mins 20 secs 10 secs Accuracy 85% 86% 90% Monocrystalline modules
  25. Summary

  26. Sparse Modeling in a nutshell • Small dataset, explainable, lightweight

    • Applicable to table, image and time series data • Has been developed over 20 years and still evolving
  27. More Python resources for Sparse Modeling https://github.com/hacarus/spm-image/

  28. Thank you !! Visit HACARUS website