Stanford ACM Tech Talk - Agile Data and Machine Learning

Talk I gave at a Stanford ACM tech talk in May 2013.

Abstract of the talk was:
How Sharethrough uses the power of Scala, Hadoop, and agile development to built data driven ad products. Young engineers and data scientists often get overwhelmed and caught up with the latest algorithm. We'll discuss how the tools and concepts of software engineering are just as important as picking the right algorithm when building data products. In a startup environment the ability to rapidly iterate on underlying algorithms while shipping product is an exciting challenge, especially given the size of modern data sets.

Transcript

1. @rweald Agile Data and Machine Learning Ryan Weald http://rweald.github.io Sunday, June 2, 13
2. @rweald Who is this guy? Sunday, June 2, 13
3. @rweald * Courtesy of Big Data Borat Me Sunday, June 2, 13
4. @rweald Data Scientist @Sharethrough Native advertising platform Sunday, June 2, 13
5. @rweald Outline 1) The problem 2) Understanding the business requirements 3) 3 keys to moving fast when your data is big 4) Things that make you slower actually make you faster 5) Architecture we used to stay lean 6) Q & A Sunday, June 2, 13
6. @rweald What this Talk is Not • What algorithms you should use • Bleeding edge machine learning • Something that is going to be on your final Sunday, June 2, 13
7. @rweald The Problem Sunday, June 2, 13
8. @rweald How to ensure you don’t throw away 3 months of work Sunday, June 2, 13
9. @rweald Sunday, June 2, 13
10. @rweald Understand the Business Requirements Sunday, June 2, 13
11. @rweald How Good Does your algorithm need to be? Sunday, June 2, 13
12. @rweald Move Fast!! Sunday, June 2, 13
13. @rweald 3 keys to moving fast with a large data set Sunday, June 2, 13
14. @rweald 1) Create a smaller sample of your data Sunday, June 2, 13
15. @rweald Sunday, June 2, 13
16. @rweald 2) Normalize and compress Sunday, June 2, 13
17. @rweald 3) Utilize powerful open source tools Sunday, June 2, 13
18. @rweald Sunday, June 2, 13
19. @rweald Most Importantly Sunday, June 2, 13
20. @rweald Running in production is hard! Sunday, June 2, 13
21. @rweald Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. Brian W. Kernighan and P. J. Plauger in The Elements of Programming Style. Sunday, June 2, 13
22. @rweald Who Understands your Algorithm? Sunday, June 2, 13
23. @rweald Monitoring is Key Sunday, June 2, 13
24. @rweald Monitoring is Key WTF? Sunday, June 2, 13
25. @rweald What you think makes you slower actually makes you faster Sunday, June 2, 13
26. @rweald Write Tests!!!! Sunday, June 2, 13
27. @rweald When People Don’t Write Tests (›°□°)›ớ ᵲ━ᵲ Sunday, June 2, 13
28. @rweald Abstractions are key Sunday, June 2, 13
29. @rweald Architectural Abstractions Ad Server Hive For Ad Hoc Reporting Raw Input Data Raw Input Data Normalized Session Data Domain Data Ad Serving Models & Reporting Ad Serving Models & Reporting Ad Serving Models & Reporting Aggregated Reporting Data Content Models User Based Models Customer Facing App Sunday, June 2, 13
30. @rweald Functional Programming Abstractions Sunday, June 2, 13
31. @rweald Functional Programming Abstractions Sunday, June 2, 13
32. @rweald We’re Hiring http://bit.ly/str-engineering Sunday, June 2, 13
33. @rweald Thanks! Twitter: @rweald Email: ryan@sharethrough.com Sunday, June 2, 13