Large Scale Machine Learning @ BigDive 4

Transcript

1. LARGE SCALE MACHINE LEARNING RECIPES SCIKIT L E A R N valeriomaggio@gmail.com @leriomaggio +ValerioMaggio
2. ESSENCE OF MACHINE LEARNING
3. ALGORITHMS ON DATA (huge amount of)
4. ALGORITHMS SPEEDUP & SPEEDUP ?
5. MULTIPLE PROCESSING IN PYTHON?
6. MULTIPLE PROCESSING IN PYTHON!!!!
7. PARALLELIZATION Single Machine Multiple Machines
8. THREADS ARE EVIL (at least in Python)
10. http://www.slideshare.net/dabeaz/in-search-of-the-perfect-global-interpreter-lock
11. http://www.slideshare.net/dabeaz/in-search-of-the-perfect-global-interpreter-lock
12. SINGLE MACHINE MULTIPLE CORES
13. • Very Simple API (part of the std. Library) • Cross-Platform support (even Windows!) • Improved in Python 3.4 • Py 3.4.3: spawn instead of fork on Unix • Some support for shared memory • Support for synchronisation (lock) multiprocessing
14. multiprocessing
15. multiprocessing (-) • Vary bad documentation • (Almost) No Docstring • Bad support for KeyboardInterrupt • Very tricky to use the shared memory values with NumPy
16. • transparent disk-caching of the output values and lazy re-evaluation (memoization) • easy simple parallel computing • logging and tracing the execution
17. No-GIL Alternative
18. meets
19. meets
20. meets
21. meets
22. MULTIPLE MACHINE MULTIPLE CORES
23. • Parallel Processing Library • Interactive Exploratory Shell • Multicore and Distributed • (load-balancing features)
25. INTERACTIVE
26. PROGRAMMATIC
32. LOOKING FORWARD TO 
 USING IT, RIGHT?
33. SCALING UP ML USE C A S E S
34. PARALLEL ML USE CASES • Stateless Feature Extraction • Model Assessment with CrossValidation • Model Selection with Grid Search • Ensemble Methods • In-Loop Averaged Models
35. • Stateless Feature Extraction • Model Assessment with CrossValidation • Model Selection with Grid Search • Ensemble Methods • In-Loop Averaged Models EMBARRASSINGLY PARALLEL ML 
 USE CASES
36. • Stateless Feature Extraction • Model Assessment with CrossValidation • Model Selection with Grid Search • Ensemble Methods • In-Loop Averaged Models IPC PARALLEL ML 
 USE CASES
37. SCALING TEXT FEATURE EXTRACTION USE C A S E 
 # 1
40. SCALABILITY ISSUES • Builds an In-Memory Vocabulary from text tokens to integer feature indices • A Big Python dict: slow to (un)pickle • Large Corpus: ~106 tokens • Vocabulary == Statefulness == Sync barrier • No easy way to run in parallel
41. THE HASHING TRICK • Replace the Python dict by a hash function • Does not need any memory storage • Hashing is stateless: can run in parallel!
42. THE HASHING TRICK
43. HASHING VECTORIZER • (+) Possibility to implement streaming and parallel text classification • (-) What about collisions • (-) No Inverse Document Frequency • (-) No easy way to retrieve the original term
44. SENTIMENT140 http://help.sentiment140.com/for-students
45. SENTIMENT140
46. PARALLEL TEXT CLASSIFICATION USE C A S E 
 # 2
47. TEXT CLASSIFICATION All Labels to Predict All Text Data
48. TEXT CLASSIFICATION IN PARALLEL Labels 1 Text Data 1 Labels 2 Text Data 2 Labels 3 Text Data 3
49. TEXT CLASSIFICATION IN PARALLEL Labels 1 Text Data 1 Labels 2 Text Data 2 Labels 3 Text Data 3 Labels 1 Vec Data 1 Labels 2 Vec Data 2 Labels 3 Vec Data 3 vec vec vec
50. TEXT CLASSIFICATION IN PARALLEL Labels 1 Text Data 1 Labels 2 Text Data 2 Labels 3 Text Data 3 Labels 1 Vec Data 1 Labels 2 Vec Data 2 Labels 3 Vec Data 3 vec vec vec clf1 clf2 clf3
51. TEXT CLASSIFICATION IN PARALLEL clf1 clf2 clf3 + + clf = Collect and Average
52. TEXT CLASSIFICATION IN PARALLEL clf1 clf2 clf3 + + clf = Collect and Average Classifier: Perceptron Model
53. RUNNING
55. GATHERING
56. AVERAGING
57. MODEL NON-LINEAR FEATURES
58. MODEL SELECTION Find parameters of the model maximizing the CV score USE C A S E 
 # 3
59. CROSS VALIDATION All Labels to Predict All Text Data
60. CROSS VALIDATION Labels 1 Text Data 1 Labels 2 Text Data 2 Labels 3 Text Data 3
61. CROSS VALIDATION Train Labels 1 Train Data 1 Train Labels 2 Train Data 2 Test Labels 3 Test Data 3
62. Train Labels 1 Train Data 1 Train Labels 2 Train Data 2 Test Labels 3 Test Data 3 Train Labels 1 Train Data 1 Train Labels 3 Train Data 3 Test Labels 2 Test Data 2 Train Labels 2 Train Data 2 Train Labels 3 Train Data 3 Test Labels 1 Test Data 1
66. WHAT IF DATA DO NOT FIT INTO MEMORY?
71. OUT-OF-CORE LEARNING • (Def.): The task of training a machine learning model on a dataset that does not fit in the main memory. • a feature extraction layer with fixed output dimensionality • knowing the list of all classes in advance • a machine learning algorithm that supports incremental learning (a.k.a. Online Learning) • the partial_fit method in scikit-learn.
74. THANKS FOR YOUR KIND ATTENTION valeriomaggio@gmail.com @leriomaggio +ValerioMaggio