Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Pydata BLN 2018: Five things I learned while prototyping ML papers

Pydata BLN 2018: Five things I learned while prototyping ML papers

ellenkoenig

July 07, 2018
Tweet

More Decks by ellenkoenig

Other Decks in Technology

Transcript

  1. FIVE THINGS I LEARNED WHILE
    PROTOTYPING ML PAPERS
    ELLEN KÖNIG / @ELLEN_KOENIG
    SENIOR DATA SCIENTIST
    NATIVE INSTRUMENTS

    View Slide

  2. A LONG, LONG
    TIME AGO…
    IN AN OFFICE
    FAR AWAY…

    View Slide

  3. HOW MANY OF YOU CAN
    RELATE TO OUR PROBLEM?

    View Slide

  4. View Slide

  5. BUT WORK IS ALL ABOUT
    GROWTH, RIGHT??

    View Slide

  6. FORTUNATELY

    View Slide

  7. OUR USE
    CASE

    View Slide

  8. WHEN SHOULD YOU LOOK FOR
    RESEARCH PAPERS?
    • „Somebody must
    have solved this
    before!“
    • No ready-to-use
    implementation

    View Slide

  9. A WORKFLOW FOR
    PROTOTYPING ML PAPERS
    1. Search for
    research findings
    2. Decide on
    comparison criteria
    3. Evaluate your
    papers
    4. Prioritize
    approaches
    5. Prototype
    approaches

    View Slide

  10. STEP 1: SEARCH FOR RESEARCH
    FINDINGS
    Needed:
    An overview of the field

    View Slide

  11. COMPILING AN OVERVIEW OF
    THE FIELD
    Compile
    Foundational and
    cutting edge papers
    Problems and
    approaches
    Start with survey
    papers,
    follow references

    View Slide

  12. STEP 2: DECIDE ON YOUR
    COMPARISON CRITERIA

    View Slide

  13. WHICH PAPERS ARE RIGHT FOR
    YOU?
    Summarize
    common metrics
    and baselines
    Refresher on baselines: https://www.quora.com/What-does-
    baseline-mean-in-machine-learning
    Pick simple
    metrics and
    baselines
    Minimally required
    metric targets?

    View Slide

  14. STEP 3: EVALUATE YOUR
    PAPERS
    Groundbreaking? Copycat? Garbage?
    Journal / conference
    quality?
    Team experience?

    View Slide

  15. STEP 3: EVALUATE YOUR
    PAPERS — A CHECKLIST
    3. Results
    2. Methodology
    1. Abstract & Introduction

    View Slide

  16. ABSTRACT & INTRODUCTION
    Addresses your problem?
    Similar context?
    Approach: Groundbreaking or
    improvement?
    Results: Better than
    targets & baseline?
    Main question: Relevant to your problem?
    3. Results
    2. Methodology
    ✔Abstract & Introduction

    View Slide

  17. METHODOLOGY SECTION
    Main question: Approach reproducible?
    Data set size and content similar?
    ✓22k black-and-white pages
    ✓German corpus
    ? Research documents rather than
    banking documents
    3. Results
    ✔ Methodology
    ✔ Abstract & Introduction

    View Slide

  18. METHODOLOGY SECTION
    Entire process described?
    ✓Seems to be complete
    Pre-processing steps described completely?
    ✓Image conversion and scaling is described
    ? OCR tool / approach is not mentioned
    Well-known methods? Or completely described
    methods?
    ✓Neural network with descriptions of the
    configuration
    3. Results
    ✔ Methodology
    ✔ Abstract & Introduction

    View Slide

  19. RESULTS SECTION
    Main question: Results reliable?
    Relevant metrics for your use case?
    ✓Accuracy
    Metrics appropriate for the problem?
    ✓Common metric for classification
    Metrics appropriate for the dataset?
    XNot suitable for imbalanced classes
    ✔ Results
    ✔ Methodology
    ✔ Abstract & Introduction

    View Slide

  20. RESULTS SECTION
    Better than your baseline?
    ✓Yes, by 0.23 over the baseline
    Better than the metrics target?
    ? They are close
    Any published review of the results?
    ? Not yet
    Improvement analyzed with suitable statistical tests?
    X No statistical analysis, and reported
    measurements are not comparable
    ✔ Results
    ✔ Methodology
    ✔ Abstract & Introduction

    View Slide

  21. RESULTS SECTION
    • For a refreshers on model evaluation see http://
    www.oreilly.com/data/free/files/evaluating-
    machine-learning-models.pdf
    • For a summary of statistical tests, see: http://
    www.pnrjournal.com/viewimage.asp?
    img=JPharmNegativeResults_2010_1_2_61_7570
    8_f1.jpg ✔ Results
    ✔ Methodology
    ✔ Abstract & Introduction

    View Slide

  22. STEP 4: PRIORITIZE YOUR
    CHOSEN APPROACHES

    View Slide

  23. PRIORIZATION MATRIX
    High
    Effort
    High
    Impact
    Quick
    Wins
    Major
    Projects
    Thankless
    Tasks
    Fill-in
    Jobs

    View Slide

  24. STEP 5: PROTOTYPE YOUR
    CHOSEN APPROACHES

    View Slide

  25. A FEW RECOMMENDATIONS
    Compile a glossary
    Understand all
    equations & code
    Higher level
    language
    Reference
    sections of
    papers

    View Slide

  26. A FEW MORE
    RECOMMENDATIONS
    Verify under same conditions
    Compile the
    performance in a table

    View Slide

  27. MORE RECOMMENDATIONS
    http://codecapsule.com/2012/01/18/how-to-
    implement-a-paper/

    View Slide

  28. SUMMARY: WHEN SHOULD YOU
    LOOK FOR RESEARCH PAPERS?
    • „Somebody must
    have solved this
    before!“
    • No ready-to-use
    implementation

    View Slide

  29. SUMMARY: A WORKFLOW FOR
    PROTOTYPING ML PAPERS
    1. Search for research findings
    2. Decide on your comparison criteria
    3. Evaluate quality, relevance and reproducibility
    4. Prioritize your chosen approaches
    5. Prototype the best approaches

    View Slide

  30. HAVE (MORE ) FUN PROTOTYPING!
    Slides will be tweeted from @ellen_koenig

    View Slide

  31. IMAGE CREDITS
    • Title slide: https://www.flickr.com/photos/
    vblibrary/6671465981
    • Slide 2: Google calendar & maps
    • Slide 4: https://www.datasciencecentral.com/
    profiles/blogs/140-machine-learning-formulas
    • Slide 6: https://pixabay.com/de/bremer-
    stadtmusikanten-skulptur-2444326/

    View Slide

  32. IMAGE CREDITS
    • Slide 8 & 28: pixabay.com
    • Slide 9: thenounproject.com
    • Search icon by Luis Prado
    • Scales icon by Veronica Karenina
    • Evaluation icon by Dinosoft Labs
    • Priorities icon by Arthur Shlain
    • Prototype icon by asianson design

    View Slide

  33. IMAGE CREDITS
    • Slide 10
    • https://pixabay.com/en/book-address-book-learning-
    learn-1171564/
    • https://en.wikipedia.org/wiki/Map#/media/
    File:World_Map_1689.JPG
    • Slide 11: thenounproject.com
    • Network icon by Gregor Cresner
    • Problem solving icon and razor blade icon by Vector Market
    • Bank icon by Stock image photo

    View Slide

  34. IMAGE CREDITS
    • Slide 12: https://pxhere.com/en/photo/536212
    • Slide 13
    • Bar chart icon: pixabay.com
    • Touch icon by Jasfart for thenounproject.com
    • Target icon by Libby Ventura for
    thenounproject.com

    View Slide

  35. IMAGE CREDITS
    • Slide 14: thenounproject.com
    • Ground breaking icon by faisalovers
    • Trash icon by UNICORN
    • Newspaper icon by Aman
    • Slide 14: Cat icon: pixabay.com
    • Slide 22: https://pxhere.com/en/photo/109282
    • Slide 23: Adapted from: http://www.sixsigmadaily.com/impact-effort-matrix/
    • Slide 24: https://pixnio.com/objects/computer/programming-code-
    programmer-coding-coffee-cup-computer-copy-hands-computer-keyboard

    View Slide

  36. IMAGE CREDITS
    1. Slide 25: thenounproject.com
    • Table icon by Yu luck
    • Pi icon by Sumana Chamrumsorakist
    • Anaconda icon by parkjisun
    • Documents icon by Creative Stall
    2. Slide 26: thenounproject.com
    • Maginfiying glass icon by afredocreates.com/icons and flaticons.com
    • Table icon by Douglas Santos
    3. Slide 29: https://commons.wikimedia.org/wiki/
    File:Pocketwatch_cutaway_drawing.jpg

    View Slide