5 Things I learned from prototyping ML research papers (GOTO Berlin 2019)

B27e5bc114b24f86625025d4dae10184?s=47 ellenkoenig
October 25, 2019

5 Things I learned from prototyping ML research papers (GOTO Berlin 2019)

B27e5bc114b24f86625025d4dae10184?s=128

ellenkoenig

October 25, 2019
Tweet

Transcript

  1. FIVE THINGS I LEARNED WHILE PROTOTYPING ML PAPERS ELLEN KÖNIG

    / @ELLEN_KOENIG SENIOR DATA ENGINEER THOUGHTWORKS
  2. A LONG, LONG TIME AGO… IN AN OFFICE FAR AWAY…

  3. None
  4. AND ONE DAY MY TEAM FACED A CHALLENGE

  5. ? Bank statement Identity document Contract …

  6. ?

  7. None
  8. WHY DID WE CONSIDER ML RESEARCH PAPERS? • „Somebody must

    have solved this before!“ • No ready-to-use implementation
  9. HOW MANY OF YOU CAN RELATE TO OUR PROBLEM?

  10. None
  11. BUT WORK IS ALL ABOUT GROWTH, RIGHT??

  12. FORTUNATELY

  13. None
  14. KEY INSIGHT: BREADTH FIRST, NOT DEPTH FIRST

  15. GOAL: FIND AND REPRODUCE THE BEST APPROACHES 1. Search for

    research findings 2. Decide on comparison criteria 3. Evaluate your papers 4. Prioritize approaches 5. Prototype approaches
  16. STEP 1: SEARCH FOR RESEARCH FINDINGS Needed: An overview of

    the field
  17. COMPILING AN OVERVIEW OF THE FIELD: BREADTH FIRST! Compile Foundational

    and cutting edge papers Common problems and approaches Start with survey papers, follow references
  18. STEP 2: DECIDE ON YOUR COMPARISON CRITERIA

  19. WHICH PAPERS ARE RIGHT FOR YOU? Summarize common metrics and

    baselines Refresher on baselines: https://www.quora.com/What-does-baseline- mean-in-machine-learning Pick simple metrics and baselines Minimally required metric targets?
  20. STEP 3: EVALUATE YOUR PAPERS Groundbreaking? Copycat? Garbage? Journal /

    conference quality? Team experience?
  21. STEP 3: EVALUATE YOUR PAPERS — A CHECKLIST 3. Results

    2. Methodology 1. Abstract & Introduction
  22. ABSTRACT & INTRODUCTION Addresses your problem? Similar context? Approach: Groundbreaking

    or improvement? Results: Better than targets & baseline? Main question: Relevant to your problem? 3. Results 2. Methodology ✔Abstract & Introduction
  23. STEP 3: EVALUATE YOUR PAPERS 3. Results 2. Methodology 1.

    Abstract & Introduction
  24. 3. Results ✔ Methodology ✔ Abstract & Introduction METHODOLOGY SECTION

    Main question: Approach reproducible? Solves similar problem? Data set size and content similar? 1. Description complete? Entire process described? Pre-processing steps described completely? Well-known methods? Or completely described methods? 2.
  25. 3. Results ✔ Methodology ✔ Abstract & Introduction METHODOLOGY SECTION

    Data set size and content similar? ✓22k black-and-white pages ✓German corpus ? Research documents rather than banking documents
  26. METHODOLOGY SECTION Entire process described? ✓Seems to be complete Pre-processing

    steps described completely? ✓Image conversion and scaling is described ? OCR tool / approach is not mentioned Well-known methods? Or completely described methods? ✓Neural network with descriptions of the configuration 3. Results ✔ Methodology ✔ Abstract & Introduction
  27. STEP 3: EVALUATE YOUR PAPERS 3. Results 2. Methodology 1.

    Abstract & Introduction
  28. RESULTS SECTION Main question: Results reliable? Evaluated with suitable metrics?

    Relevant metrics for your use case? Metrics appropriate for the problem? Metrics appropriate for the dataset? ✔ Results ✔ Methodology ✔ Abstract & Introduction 1. Results good enough? Better than your baseline? Better than the metrics target? Any published review of the results? Improvement analyzed with suitable statistical tests? 2.
  29. RESULTS SECTION Relevant metrics for your use case? ✓Accuracy Metrics

    appropriate for the problem? ✓Common metric for classification Metrics appropriate for the dataset? XNot suitable for imbalanced classes ✔ Results ✔ Methodology ✔ Abstract & Introduction
  30. RESULTS SECTION Better than your baseline? ✓Yes, by 0.23 over

    the baseline Better than the metrics target? ? They are close Any published review of the results? ? Not yet Improvement analyzed with suitable statistical tests? X No statistical analysis, and reported measurements are not comparable ✔ Results ✔ Methodology ✔ Abstract & Introduction
  31. STEP 3: EVALUATE YOUR PAPERS 3. Results 2. Methodology 1.

    Abstract & Introduction
  32. STEP 4: PRIORITIZE YOUR CHOSEN APPROACHES

  33. PRIORIZATION MATRIX High Effort High Impact Quick Wins Major Projects

    Thankless Tasks Fill-in Jobs
  34. STEP 5: PROTOTYPE YOUR CHOSEN APPROACHES

  35. A FEW RECOMMENDATIONS Compile a glossary Understand all equations &

    code Higher level language Reference sections of papers
  36. MORE RECOMMENDATIONS http://codecapsule.com/2012/01/18/how-to- implement-a-paper/

  37. OUR FINAL RESULTS

  38. PRIORIZATION MATRIX High Effort High Impact Quick Wins Major Projects

    Thankless Tasks Fill-in Jobs
  39. SUMMARY: WHEN SHOULD YOU LOOK FOR RESEARCH PAPERS? • „Somebody

    must have solved this before!“ • No ready-to-use implementation
  40. SUMMARY: OUR MAIN LESSONS Pool your knowledge Follow a strategy

    Go „Breadth first“ Record your insights
  41. SUMMARY: A WORKFLOW FOR PROTOTYPING ML PAPERS 1. Search for

    research findings 2. Decide on your comparison criteria 3. Evaluate quality, relevance and reproducibility 4. Prioritize your chosen approaches 5. Prototype the best approaches
  42. HAVE (MORE ) FUN PROTOTYPING! Slides will be tweeted from

    @ellen_koenig
  43. IMAGE CREDITS • Title slide: https://www.flickr.com/photos/vblibrary/6671465981 • Slide 2: Google

    calendar & maps • Slide 10 & Slide 13: https://www.datasciencecentral.com/profiles/blogs/ 140-machine-learning-formulas • Slide 12 & 40: https://pixabay.com/de/bremer-stadtmusikanten- skulptur-2444326/ • Slide 14 & 40: https://commons.wikimedia.org/wiki/File:Breadth- first_tree.svg • Slide 14: https://commons.wikimedia.org/wiki/Depth_first_search#/ media/File:Depthfirst.png
  44. IMAGE CREDITS CONT. • Slide 16 https://en.wikipedia.org/wiki/Map#/media/ File:World_Map_1689.JPG • Slide

    29: https://commons.wikimedia.org/wiki/ File:Pocketwatch_cutaway_drawing.jpg • Slide 32: https://pxhere.com/en/photo/109282 • Slide 33: Adapted from: http://www.sixsigmadaily.com/impact-effort- matrix/ • Slide 34: https://pixnio.com/objects/computer/programming-code- programmer-coding-coffee-cup-computer-copy-hands-computer- keyboard