Error Analysis in Machine Learning

I spoke about how to perform Error analysis on machine learning and over how to reduce error rate of the learning algorithm.

I have also discussed what transfer, multi-task and end-to-end learning are

Transcript

1. ERROR ANALYSIS WEEK 3 COURSE 2 30 MAY 2019
2. OUTLINE • ERROR ANALYSIS • TRANSFER LEARNING • MULTI TASK LEARNING • END-TO-END LEARNING
3. WHAT IS ERROR ANALYSIS? • Is the process of manually examining dev set examples that your algorithm misclassified. • Will help you understand the cause of the problem. • This will give you a better insight on what to do next.
4. CONT. Example. • Look at your dev example of your cat classifier App • So, the question is should you try make your cat classifier do better on dogs?
5. CONT. ERROR ANALYSIS: • Get a sample of 100 mislabeled dev set examples. • Look at these examples manually, and count what fraction of them are dog images. OUTPUT: • 5 % of images are dogs. • 50 % of images are dogs.
6. EVALUATE MULTIPLE IDEAS IN PARALLEL • Your team may have several ideas for improving the cat detector: • Fix picture of dogs being recognized as cats • Fix great cats (lion, Panther, etc. . . ) being misrecognized • Improve performance on blurry images
7. CONT.
8. CLEANING UP MISLABELED DATA • During error analysis, you might notice that some examples in your dev set are mislabeled ( the pictures were already mislabeled by a human ). • Should you correct the labels in your dev set ?
9. CONT. Over all error: 10 % 2% Error due to incorrect Label: 0.6% 0.6% Error due to other causes: 9.4% 1.4%
10. CONT. Additional guideline to consider. • Make sure you dev and test sets come from same distribution • Consider examining examples your algorithm got right as well as once it got wrong.
11. BUILD YOUR FIRS SYSTEM QUICKLY & ITERATE • If you are tackling a brand Machine learning problem, it is advised that to build your first system quickly and then iterate. • Speech recognition example. • Noisy background (Café noise, car noise) •Accented speech •Far from microphone •Stuttering, etc.
12. CONT. Build your system quickly step 1: set up dev/test set matric (set up target). step 2: Build your initial system quickly. step 3: Use bias/Variance analysis and error analysis to prioritize next steps.
13. TRAINING AND TESTING ON DIFFERENT DIST. • Cat App Example •What can you do ? 10,000 images 200,000 images You care about this data
14. CONT. Option 1 : •Put all data sets together •Randomly shuffle them and divide to train, dev and test • Advantage is all your sets come from the same distribution. • Disadvantage is most of your dev set images come from the web data Train 205,000 images dev 2.5k test 2.5k
15. CONT. Option 2: • put all the data that came from the web to train sets or you could add some from mobile app. •Make your dev and test sets come from the mobile app • Disadvantage is all your train set now come from the web Train 205k dev 2.5k test 2.5k 205k from web, 5k from mobile app All from mobile app
16. BIAS AND VARIANCE WITH MISMATCHED DATA DIST. • Estimating bias and variance of your learning algorithm will also help you prioritize on what to next. • Analyzing bias and variance change when your training sets comes from different distribution than dev/test sets will help you identify your error. • Cat classifier App Example •Training error – 1% •Dev error – 10%
17. CONT. • You can conclude two problems. 1. May be you have large variance problem 2. Distribution problem and your algorithm is just doing right • To identify the problem introduce Training-Dev set. •Training-Dev set : Same distribution as training set, but not used for training Test set Dev set Training/Dev set Training set
18. CONT. • Lets say : •Training error : 1 % 1% •Train/Dev error : 9% 1.5% •Dev set error : 10% 10% Dev set Training/Dev set Training set This shows high variance problem This shows high data mismatch problem Test set
19. ADDRESSING DATA MISMATCH PROBLEM • Carryout manual error analysis to try to understand difference between training and dev/test sets. • Make training data more similar; or collect more data similar to dev/test sets.
20. TRANSFER LEARNING • Sometimes you can take knowledge that some neural network has learned from one task and apply that knowledge to tackle separate task. This is called Transfer Learning. • Image recognition Example (X,Y) Y ˆ (w,b) Radiology diagnosis Image recognition Then, Swap new input (X,Y) from radiology and train you NN
21. WHEN TO USE TL? • Task A and Task B has the same input X. • You have a lot more data for Task A than task B.
22. MULTI TASK LEARNING • Is to make your single neural network to do several things simultaneously at the same time. • Autonomous driving Example •Features you need to identify: •Pedestrian •Cars •Stop sign •Traffic Lights
23. CONT. This makes sense usually if data you have for each task is quite similar [4,1] vector
24. END-TO-END DEEP LEARNING • Most learning system require multiple stages of processing, In End- to-End deep learning; it takes all those multiple stages and replace it with single neural network. • Speech recognition Example Audio Features Phonemes Words Transcript X Y MFCC ˆ Traditional DL: Audio Transcript Y ˆ X E2E DL:
25. CONT. Challenges of E2E DL: • Requires lots of data before it works well. If you have smaller data traditional pipelines works better. • If your data is not big enough, you could take some stages to your E2E DL pipeline. (i.e. audio > features > transcript) ADV: • Less hand designing components required
26. THANK YOU