Upgrade to Pro — share decks privately, control downloads, hide ads and more …

2017 - Rachel Thomas - Using Randomness to make code much faster

Db2ee812bdc6fd057f8f4209c08b6f63?s=47 PyBay
August 12, 2017

2017 - Rachel Thomas - Using Randomness to make code much faster

Description
An introduction to randomized linear algebra (a recently developed field with huge implications for scientific computing) in Python with a detailed case study of randomized Singular Value Decomposition (SVD). We will look at the applications of using randomized SVD to find the topics of documents and to identify the background in a surveillance video.

Abstract
Linear algebra lies at the heart of much of data science, including deep learning, regression, and recommendation systems, and is also widely used in engineering, finance, and much more. The basic linear algebra techniques of matrix products and decompositions have super-linear runtimes, and therefore speeding them up is of vital importance. Counter-intuitively, recent advances have shown that the key to doing this is to take advantage of random matrices. We will see how to use random matrices to dramatically speed up the widely used singular value decomposition, a method that is used in least squares regression, PCA, general matrix inverses, and more.

Attendees will learn how language and video data can be represented as matrices with numpy. I will explain what Singular Value Decomposition (SVD) is conceptually and how randomized SVD gives us a huge improvement in speed. We will see applications in Python of using randomized SVD to find the topics of a group of documents and identify the background in a surveillance video. I will introduce all math concepts needed so there are no prerequisites, although familiarity with data processing will be helpful.

Bio
Rachel Thomas is co-founder of fast.ai, which is making deep learning more accessible, and a researcher-in-residence at University of San Francisco Data Institute. Rachel has a mathematics PhD from Duke and has previously worked as a quant, a data scientist + backend engineer at Uber, and a full-stack software instructor at Hackbright.

Rachel was selected by Forbes as one of 20 "Incredible Women Advancing A.I. Research." She co-created the course "Practical Deep Learning for Coders," which is available for free at course.fast.ai and >50,000 students have started it. Her writing has made the front page of Hacker News 4x, the top 5 list on Medium, and been translated into Chinese, Spanish, & Portuguese. She is on twitter @math_rachel

Video: https://www.youtube.com/watch?v=7i6kBz1kZ-A

Db2ee812bdc6fd057f8f4209c08b6f63?s=128

PyBay

August 12, 2017
Tweet

Transcript

  1. Using randomness to make code much faster Rachel Thomas fast.ai

    & USF Data Institute @math_rachel
  2. My Background Swarthmore: Math + CS Duke: Math PhD Quant

    Uber Data Scientist & Engineer Hackbright Instructor fast.ai co-founder & USF researcher @math_rachel data science blog: fast.ai https://medium.com/@racheltho
  3. Computers treat pictures and words as numbers Source: Adam Geitgey

  4. If you think you’re not good at math… “A Mathematician’s

    Lament”--Lockhart
  5. None
  6. Computational Linear Algebra • This talk is based on a

    course I created for USF MS in Analytics students & released for FREE: github.com/fastai/numerical-linear-algebra How do we get computers to do matrix computations with acceptable speed and acceptable accuracy?
  7. That’s random… • Random Data Structures • Bloom Filters •

    HyperLogLog • Locality-Sensitive Hashing • Skip Lists • Count-min sketch • Min-hash • Random Algorithms • Karger's algorithm (minimum cut of a graph) • Randomized regression • Monte Carlo simulation • Randomized LU decomposition (Gaussian Elimination) • Randomized SVD
  8. Word Embeddings • Words are represented as high-dimensional vectors Source:

    Adrian Colyer Source: Rachel Thomas
  9. Demo #1 The demos are available on github: https://github.com/fastai/randomized-SVD

  10. Surveillance Video

  11. One moment in time:  Too tall to fit on

    the slide
  12. The whole surveillance video:

  13. This is the background:

  14. These are the people walking around:

  15. is the sum of these two: This matrix…

  16. Matrix Decompositions • We use matrix multiplication and addition to

    put matrices together. • We use matrix decompositions to take them apart!
  17. Factorization • Multiplication: 2 * 2 * 3 * 3

    * 2 * 2  144 • Factorization is the “opposite” of multiplication: 144  2 * 2 * 3 * 3 * 2 * 2 • Factorization is much harder than multiplication… (which is good, because it’s the heart of encryption).
  18. Matrix Decomposition • Taking matrices apart • Harder than putting

    them together • One application:
  19. Parts of Faces Source: Essid & Ozerov

  20. Term-Document Matrix source: Schütze & Lioma

  21. Source: Essid & Ozerov Identify Topics for Documents

  22. Corrupted Data Source: Jean Kossaifi, Robust Tensor PCA with TensorLy

  23. Demo #2 The demos are available on github: https://github.com/fastai/randomized-SVD

  24. SVD on a big matrix is slow  • For

    our video matrix (19,200 x 6,000): 57 seconds • Let’s do SVD on a smaller matrix! 100 1,000 10,000 100 0.006 0.009 0.043 1000 0.004 0.259 0.992 10000 0.019 0.984 218.726
  25. Punchline: big speedup!

  26. None
  27. Applications of SVD • Principal Component Analysis • Data compression

    • Pseudo-inverse • Collaborative Filtering • Topic Modeling • Background Removal • Removing Corrupted Data
  28. SVD is just 1 of many randomized algorithms • Random

    Data Structures • Bloom Filters • HyperLogLog • Locality-Sensitive Hashing • Skip Lists • Count-min sketch • Min-hash • Random Algorithms • Karger's algorithm (minimum cut of a graph) • Randomized regression • Monte Carlo simulation • Randomized LU decomposition (Gaussian Elimination) • Randomized SVD
  29. You often don’t need to be super accurate • When

    your measurements aren’t that accurate • When you have a ton of data • In machine learning, less accuracy can help reduce overfitting • Run approximate algorithms multiple times to increases probability of correct answer • Depending on the application
  30. Resources • Word Embeddings Workshop • Computational Linear Algebra

  31. Questions? @math_rachel fast.ai