Exploring the Unknown with Scala

Whatever your experience level, I'm willing to bet that you want to become a better engineer and problem-solver. You have probably already noticed that exploring unfamiliar technologies, patterns, algorithms and approaches helps you become better at your job, and sometimes even identify what job you want to be in. At this point you probably have a voice in your head saying that exploring, while fun and exciting, takes time. Perhaps you often tell yourself that you're too busy getting things done to indulge your curiosity.

I'm going to tell you the story of my personal confrontation with a dilemma facing every human being on earth: exploration vs. exploitation. I'll share with you the lessons I've learned about becoming a better explorer. I'll discuss how my usage of Scala and engineering mindset have changed as I have gone from developing large-scale production applications to conducting a series of experiments that test hypotheses. I hope you walk out of this talk with a renewed enthusiasm for exploration and a framework that you can use to decide when to explore and when to exploit.

Transcript

1. Exploring the unknown with Scala Julie Pitt Order of Magnitude Labs @yakticus
2. Chapter I The Founding
3. Julie Pitt Greg Orzell Christian Kaiser Order of Magnitude Labs founding team
4. intelligent machines the frontier
5. https://flickr.com/photo/13547802@N05/8907414305/ 1. emerging approaches 2. scale 3. profit the business hypothesis
6. why prototype in Scala? • functional • immutable • type system • industrial scale libraries
7. prototype #1 • distributed, async neural net • actor = “neuron” • message passing between neurons
8. Just because you can do it doesn’t mean you should.
9. prototype #2 • embarrassingly simple • case classes • pure functions http://github.com/oomagnitude/scala-spatialpooler
10. https://flickr.com/photo/69793885@N00/7874464932/ current approaches intelligent machines the reality
11. Chapter II Crossroads
13. exploitation exploration value http://www.fil.ion.ucl.ac.uk/~karl/Active%20inference%20and%20epistemic%20value.pdf
14. exploitation exploration it works why exploration is hard
15. Chapter III (Re)learning to explore
16. but the spec says... unused code is dead code we need a single peer-reviewed codebase we need a project plan
17. • building to spec • low-risk releases • continuous deployment • minimal tech debt I was optimizing for...
18. • formulating hypotheses • ...testing them • ...rejecting bad ones • repeat! I should have optimized for...
19. lessons learned about hypothesis testing 1. contradict yourself 2. prepare your laboratory 3. merge ideas, fork code
20. 1. contradict yourself
21. When you encounter contrary evidence, form new hypotheses.
22. experimental insight app server data collection framework https://www.youtube.com/watch?v=PQuDD_EHM9I experiments
23. low-tech compromise experiments
24. The more you’ve built, the harder it is to admit you’re wrong.
25. When being right means adding complexity, apply Occam’s razor.
26. 2. prepare your laboratory
27. When all you have is a hammer, everything looks like a nail.
28. tools hypotheses experiments
29. • index of what you know ◦ be greedy • be lazy ◦ ease of use, versatility • don’t over-engineer ◦ think println toolbox considerations
30. • primitives -> pure functions ◦ analogous to numpy • what is possible -> types • how to do it -> methods ◦ builds on primitives • types ensure tools are used correctly our toolbox
31. • make tools accessible • minimal setup • distraction-free workspace requirements
32. • easily import tools • Amm REPL for gut checks • Scala scripts for repeatable tests • SBT for involved tests Ammonite to the rescue http://www.lihaoyi.com/Ammonite/
33. 3. merge ideas, fork code
34. Ambiguity means several hypotheses seem equally plausible.
35. Requiring a single codebase downplays ambiguity.
36. With a single codebase, discussions center on “how.”
37. • hypotheses ◦ divide & conquer • tools • requirements* ◦ *as they become clear what to share
38. optimize for hypothesis testing 1. contradict yourself 2. prepare your laboratory 3. merge ideas, fork code
39. Chapter IV Mode Switching
40. https://flickr.com/photo/9136641@N07/2177935073/ execution creativity
41. mode switching algorithm 1. which mode am I in? 2. am I ready to switch? 3. how much time to I need to switch?
42. https://flickr.com/photo/69793885@N00/7874464932/ which mountain should I climb? creative mode
43. https://flickr.com/photo/69793885@N00/7874464932/ climb the mountain. execution mode
44. activate creative mode when... • your assumptions are wrong • you’re tempted to add unnatural assumptions • complexity is mounting
45. activate execution mode when... • you can describe the goal • you can’t find evidence against your hypotheses • achieving the goal is probable
46. give yourself time to switch The farther in or out you need to zoom, the more time it will take to switch modes
47. Chapter V Conclusion
48. prototyping • build the embarrassingly simple version first • just because you can do it doesn’t mean you should
49. optimize for hypothesis testing • contradict yourself • prepare your laboratory • merge ideas, fork code
50. mode switching • creativity and execution are modal • know which mode you’re in and when to switch • give yourself time to switch
51. https://flickr.com/photo/69793885@N00/7874464932/