Data Science at Scale @ Barricade.io

This talk describes the challenges with data science and how we run data analysis at scale at https://Barricade.io

1. Data Science @ Scale
2. @davidcoallier Part of an amazing team at Barricade.io
3. Data Science is Hard
4. Data Hacking is “Easy”
5. Data Analysis is “Easy”
6. Data Expertise is “Easy”
7. Got all? Having the three is real hard!
9. Is that it? Well don’t forget your purpose.
10. You are not an economist. ɪˈkɒnəmɪst/: Someone with all the answers, and none of the questions.
11. The Data Scientific Method
12. Find a question.
13. Use the data you have
14. Features & Tests
15. Analyse Results You will be sad.
16. Conversate Talk about your findings.
17. Good Chats Imply egoless and collaborative data scientists.
18. Recap.
19. 1. Hacking 2. Maths & Stats 3. Expertise
20. And
21. 1. Question 2. Be Pragmatic 3. Features 4. Analyse 5. Share.
22. A team! Rarely a single-person effort.
23. An Example Fraud Prevention — Business Prevention
24. I knew better. Obviously… duh
25. We didn’t share. Science has historically been shared.
26. Not with p-values
27. Empathise. Use human language, not lingo.
28. For us at Barricade
30. Doing this at scale is hard.
31. We’re still small About a billion data points a day.
32. Humble Beginnings Typically… an Queue and an API.
33. This had issues. Hard to scale, hard to decouple, etc.
34. Enter the Lambda Architecture.
39. Speed Layer
41. Batch Layer
43. Speed Layer: U new behaviour from new data Batch Layer: All classified behaviour since T
44. Serving Layer
46. Speed Layer: U new behaviour from new data Batch Layer: All classified behaviour since T Serve Layer: Batch layer U Speed Layer
47. Cache Layer
49. On Amazon AWS
50. Identifying an Attack.
52. Ahh! What’s that?
53. Kafka Queue. Distributed messaging system Append-only log Consumers have offsets Partition for parallelism Replicate for redundancy Message order guaranteed, per-partition
59. Barricade Customer
61. Questions?
62. @davidcoallier @barricadeio