Streetlife Analytics

Snowplow London meetup talk about how we do analytics at streetlife using snowplow and how realtime is helping us to deliver value to our users.

Transcript

1. ANALYTICS
2. • Local social network • Web, iOS and Android • 1.3M users in uk. • 1.5B events
8. analytics are core for our business
9. we track millions of events a day
14. Subject: Here's what your neighbours are talking about... To: user@domain.com
15. Subject: Here's what your neighbours are talking about... To: user@domain.com
16. Subject: Here's what your neighbours are talking about... To: user@domain.com
17. Susan P.
18. Susan P.
19. Susan P.
20. we currently use the batch pipeline Redshift
21. we gather insights once a day Redshift user_stats message_stats SQL
22. last_visit last_email_open 2016/01/15 12:33:54 2016/07/12 22:13:10 2016/07/10 04:52:01 2016/03/11 21:01:03 2016/07/11 05:34:45 2016/07/12 14:30:21
23. day impressions 2016/01/15 engagments id 1 u_impressions u_engagments 345 122 299 99 2016/01/15 2 123 99 94 84 2016/01/15 3 934 845 843 789 2016/01/14 1 899 777 744 645 2016/01/14 2 754 543 634 433 2016/01/14 3 103 99 91 77 2016/01/13 1 499 301 382 235 2016/01/13 2 1893 1400 991 1099
24. With this data we give businesses insights Your message got 890 impressions and 350 engagements.
25. We tune each message's reach dynamically 2000 target 1200 current >
26. It works great, but the data has several hours of lag event ingest Load SQL View S3 Load
27. We are going to start using the realtime pipeline realtime analytics realtime content router
28. realtime pipeline Redshift Kinesis λ
29. Kinesis λ λ Dynamodb Lambda Lambda
30. The latency from events happening to us acting on them would be nearly zero event ingest Kinesis Lambda DynamoDB Load
31. AWS Lambdas: 20,000 feet view .py .js .java DEPENDENCIES ZIP AWS S3 λ
32. λ • Zero-administration compute platform • Connect Lambdas with AWS services • Kinesis, DynamoDB, APIGateway, S3, CloudWatch, Cron… • Pricing based on usage: • ~$0.0000002 per/run • ~$0.000000208 per/100ms λ λ AWS Lambdas: 20,000 feet view
33. There is a ‘but’ Deploying/Managing them is a pita λ
35. • Tool to create, wire and deploy AWS Lambdas using CloudFormation • Python/Javascript/Java/Golang/Scala runtimes… • Supported integrations • APIGateway, CloudWatch, Dynamodb, Kinesis, S3
36. + snowplow events simulator λ DynamoDB Kinesis
37. We are hiring! Thank you! Software Engineer (data team) Software Engineer (core team) Infrastructure Engineer Data Scientist