Visual Content's Journey to Agile Data Science

How to apply agile software engineering methods to a team of data scientists. A field study from trivago's (former) images team.

Transcript

1. Visual Content's Journey to Agile Data Science 2019-03-18, Düsseldorf, Germany Pascal Cremer
2. Pascal Cremer Visual Content, Hotel Search trivago Düsseldorf b00gizm b00gizm pcremer
3. “I didn’t know we have a dedicated team for images at trivago”
6. 30M Live Accommodation Images. 160M Total Accommodation Images. Source: trivago Hotel Search Controller 2019-03-18
7. Icons made by Pause08 from www.flaticon.com
8. Factual Data Existing information for value reflection Derived Data Extracted knowledge for personalization
9. Icons made by Pause08 from www.flaticon.com
10. Photo by Maksim Sansonov on Unsplash Project Flamingo
11. Photo by Jay Castor on Unsplash Project Spa Botox
12. Replace the main image on the item element on SEM Landing Pages for Spa & Wellness ("Spa Botox") and Pool & Beach ("Flamingo")
13. Before Spa Botox
14. After Spa Botox
15. Spa Botox C-test Rejected! Icons made by Freepik from www.flaticon.com
16. Icons made by Freepik from www.flaticon.com
17. Spa Botox 2.0: Validate the potential of contextual main images by leveraging our own custom tagging solution for images displayed on SEM Landing Pages for Spa & Wellness.
18. Input Conv + ReLu Pooling Flattening Fully Conn Prediction
19. So we gave it a shot. And we f**ked up!
20. How to not create value Soaking yourself in overly long research phases. Working backwards. Not documenting your results. And with all that: Not shipping anything.
21. “The intrinsically probabilistic and non- deterministic characteristics of Data Science do not make it an easy or a natural fit for organizations accustomed to predominantly linear and reasonably predictable development models.” Source: Applying Agile to Data Science - Medium.
22. So, is Data Science even a good fit for an agile (and cross-functional) team?
23. “Our Data Scientists are using Scrum, too. They work on their own 8 story point tasks each sprint." Source: An actual Product Owner!
24. Putting imaginary points to your JIRA tasks does not make you agile. That is not how that works!
25. Agile is about empowering teams to deliver value with an emphasis on close collaboration between team members and business stakeholders using short feedback cycles.
26. Agile Data Science In A Nutshell Embrace the MVP. Follow the Data Value Pyramid. Document everything! ⚗ Experiments, not tasks.
27. Embrace the MVP Start with something crappy!
28. Embrace the MVP The MVP in a Data Science initiative is whatever deliverable addresses some narrowly scoped business requirements using a minimal a set resources and tasks. "[An] MVP can simply be a predictive model that’s more accurate than random guessing." Source: Agile Development in Team Data Science - Wikibon Research.
29. Follow the Data Value Pyramid Data Acquisition Data Exploration Modelling Deployment Action Value Generation
30. Document Everything Assets like tables, charts, reports, and predictions emerge as artifacts while iteratively climbing the Data Value Pyramid. And while they might not be "shippable" in software sense, they provide a strong basis for the dialog with business stakeholders.
31. Experiments, Not Tasks While iterating, we want to achieve insights based on data, which can be best described as experiments. Experiments should be reproducible with all their artifacts documented.
32. Spa Botox 2.0, the second take!
33. Embrace the MVP SEM Landing Page powered by Clairfai (3rd party) tags. MVP model with >80% accuracy on average for all classes.
 Training & validation set of 1000 images per class. Model with >80% overall accuracy for each class.
 Training & validation set of 2000 images per class.
34. Experiments & Variants Experiments as counterpart to User Stories. They come with their own DoD like being reproducible (Jupyter Notebook checked into Github) and all artifacts documented inside a Dropbox Paper. Variants are related to "parent" experiments and can be run in parallel. Variations can differ from each other by their choice of model architecture, learning rate, batch size, loss function and other hyper parameters.
35. Documentation of Artifacts
36. One Agile Board to rule them all! Backlog Defined In Progress In Review Blocked Done!
37. Thank You. @b00gizm | github.com/b00gizm I'd love to get your feedback.