Upgrade to Pro — share decks privately, control downloads, hide ads and more …

WWC Talk: Five things I learned from turning research papers into industry prototypes

WWC Talk: Five things I learned from turning research papers into industry prototypes

B27e5bc114b24f86625025d4dae10184?s=128

ellenkoenig

April 17, 2018
Tweet

Transcript

  1. FIVE THINGS I LEARNED FROM TURNING RESEARCH PAPERS INTO INDUSTRY

    PROTOTYPES ELLEN KÖNIG / @ELLEN_KOENIG
  2. MOTIVATION

  3. WHEN SHOULD YOU LOOK FOR RESEARCH PAPERS? • General problem

    • No library exists
  4. A WORKFLOW FOR PROTOTYPING ML PAPERS 1. Search for research

    findings 2. Decide on your comparison criteria 3. Evaluate quality, relevance and reproducibility 4. Prioritize your chosen approaches 5. Prototype the best approaches
  5. STEP 1: SEARCH FOR RESEARCH FINDINGS • Goal: Get an

    overview of the field with Google Scholar • Start with survey papers, follow references • Compile your findings: • Common problems & approaches • Foundational and cutting edge papers
  6. STEP 2: DECIDE ON YOUR COMPARISON CRITERIA • Goal: Decide

    what will be a good paper • Summarize common metrics and baselines • Pick a few simple metrics and baselines • Decide which metric target is minimally required • Refresher on baselines: https://www.quora.com/ What-does-baseline-mean-in-machine-learning
  7. STEP 3: EVALUATE YOUR PAPERS • Goal: Identify three types

    of papers: ✓Groundbreaking ? Copycat XGarbage • Check journal / conference quality and team experience • Keep a log of your evaluation results
  8. STEP 3: EVALUATE YOUR PAPERS — A CHECKLIST 3. Results

    2. Methodology 1. Abstract & Introduction
  9. EVALUATING PAPERS CHECKLIST: ABSTRACT & INTRODUCTION ? Question: Is the

    approach relevant to your problem? ✓Problem: Do they address your problem? ✓Context:Is it similar to yours? ✓Approach: Is it groundbreaking or an improvement? ✓Results: Better than your targets & baseline? ✓Age: How old is the paper? 3. Results 2. Methodology ✔Abstract & Introduction
  10. EXAMPLE Problem Context Approach Results

  11. EVALUATING PAPERS CHECKLIST: METHODOLOGY SECTION ? Question: Will you be

    able to reproduce the approach? ✓Are the data set size and data types similar to yours? ✓Do they describe the entire process they used? ✓Are the pre-processing steps described completely? ✓Do they use standard methods? ✓If not, do they provide complete algorithmic descriptions of the methods? 3. Results ✔ Methodology ✔ Abstract & Introduction
  12. EXAMPLE ✓Is the data set size and content similar to

    yours? ✓22k black-and-white pages ✓German corpus ? Research documents rather than banking documents ✓Do they describe the entire process they used? ✓Seems to be complete ✓Are the pre-processing steps described completely? ✓Image conversion and scaling is described ? OCR tool / approach is not mentioned ✓Are the ML and statistical methods standard methods? ✓Standard methods (neural networks) with descriptions of the configuration
  13. EVALUATING PAPERS CHECKLIST: RESULTS SECTION ? Question: Are the results

    reliable? ✓Are the metrics relevant to your problem? ✓Are the metrics appropriate for the type of ML problem? ✓Are the metrics appropriate for the dataset (imbalanced classes, outliers, …)? ✔ Results ✔ Methodology ✔ Abstract & Introduction
  14. EVALUATING PAPERS CHECKLIST: RESULTS SECTION ? Question continued: Are the

    results reliable? ✓Are the results better than your baseline? ✓Are the results better than your metric targets? ✓Was any critique or review of the results published? ✓Are improvement over existing methods analyzed with proper statistical tests? ✔ Results ✔ Methodology ✔ Abstract & Introduction
  15. EVALUATING PAPERS CHECKLIST: RESULTS SECTION • For a refreshers on

    model evaluation see http:// www.oreilly.com/data/free/files/evaluating- machine-learning-models.pdf • For a summary of statistical tests, see: http:// www.pnrjournal.com/viewimage.asp? img=JPharmNegativeResults_2010_1_2_61_7570 8_f1.jpg ✔ Results ✔ Methodology ✔ Abstract & Introduction
  16. EXAMPLE ✓Are the metrics relevant to your problem? Accuracy ✓Are

    the metrics appropriate for the type of ML problem? Accuracy is a common metric for classification XAre the metrics appropriate for the dataset? Accuracy is not as suitable for imbalanced classes, and the labels are reported as „uneven“ ✓Are the results better than your baseline? Yes, by 0.25 over the baseline ? Are the results suitable for the business problem? They are close ? Was any critique or review of the results published? Not yet XAre improvement over existing methods analyzed with proper statistical tests? No statistical analysis, and reported measurements are not comparable
  17. STEP 4: PRIORITIZE YOUR CHOSEN APPROACHES High Effort High Impact

    Quick Wins Major Projects Thankless Tasks Fill-in Jobs
  18. STEP 5: PROTOTYPE YOUR CHOSEN APPROACHES A few recommendations •

    Compile a glossary of all unfamiliar terms and methods • Make sure you understand all equations & code • Prototype first in a higher level language (Python, R, Octave, Julia,…) • Reference papers and sections of papers in your code documentation
  19. STEP 5: PROTOTYPE YOUR CHOSEN APPROACHES A few more recommendations

    • Verify the results in the paper under the same conditions before adapting • Compile the performance of each approach in a table • More recommendations: http://codecapsule.com/ 2012/01/18/how-to-implement-a-paper/
  20. SHORT ADVERTISEMENT FOR PEOPLE BASED IN BERLIN

  21. BI-WEEKLY OPENTECHSCHOOL DATA SCIENCE CO-LEARNING • We are looking for

    volunteer coaches and learners! • Kick-off: Thursday, 19 at 7:30 PM in Kreuzberg • https://www.meetup.com/opentechschool-berlin/ events/249735100/ • OpenTechSchool is a non-profit, volunteer-run tech education community
  22. END OF ADVERTISEMENT BLOCK

  23. SUMMARY: A WORKFLOW FOR PROTOTYPING ML PAPERS 1. Search for

    research findings 2. Decide on your comparison criteria 3. Evaluate quality, relevance and reproducibility 4. Prioritize your chosen approaches 5. Prototype the best approaches Slides will be tweetet from @ellen_koenig
  24. IMAGE CREDITS 1. Title slide: https://www.flickr.com/photos/vblibrary/ 6671465981 2. Slide 3:

    pixabay.com 3. Slide 4 & 22: https://commons.wikimedia.org/wiki/ File:Pocketwatch_cutaway_drawing.jpg 4. Slide 5 -7: pixabay.com 5. Slide 16: Adapted from: http://www.sixsigmadaily.com/ impact-effort-matrix/