Florian Pfeiffer - User Behavior Tracking

In this talk I'll outline the different ideas we had, when we built our system to track users&events on our websites. Beside the infrastructure we have built I'll also present an actual usecase for that system: feeding data into our recommendation engine for newly registered users to avoid the coldstart problem.

Transcript

1. User Behaviour Tracking
   Track - Store - Process ! //Florian Pfeiffer - Head of Data&Infrastructure - gutefrage.net
   !
3. Vision „Let’s build our own Google Analytics“
4. Why analytics does sampling we want the (raw) data
5. Ideas,Thoughts&Goals fast / minimal impact on page loading time high availability track user over multiple platforms storage engine? -> hbase
6. Infrastructure
7. Numbers! 10-20ms Response Time per pixel record for now: ~2500 concurrent reqs 1,5 billion entries in Hbase 10 Nodes in Hadoop Cluster
8. Serving Infrastructure Loadbalancers & RR DNS nginx with empty_gif module (~2ms) data is written to logfile
9. Storing Infrastructure every nginx node has flume-ng flume ingests logfile AsyncHBaseSink with custom Serializer direct writes to HBase
10. why flume? we had it already in production ;) Storm might be an interesting alternative
11. HBase rowkey design
12. Why? You can scan through all data and use filters for selecting specific data But scanning with start & stop row speeds things up (a lot)
13. HBase rowkey design Do I need a fast user or a fast timespan lookup? User - clientid,ts<,connectionId> Timespan - ts,clientid<,connectionId>
14. Inverse Timestamps Data in HBase is stored lexicographicaly sorted Normal TS - scan would yield oldest results first Inverse TS - newer entries come first (and you can cancel the scan if you have enough data)
15. Cross Domain Tracking (Flash)Cookies Fingerprinting Etag HTML5 Storage
16. The olden times… or Cookies Easy to drop a 3rd party cookie with userId on different websites Gets more and more blocked (Safari, FF..)
17. Fingerprinting Yields interesting results on desktop, difficult on e.g. iPhone invisible to user Last resort if everything else fails?
18. Etag Quite new, based on browser cache sounds interesting
19. HTML5 Storage Store data in local HTML5 storage Retrieve data with Cross Domain Messaging
20. Store data e.g. UserId, SessionId, GeoIP, URL, action, data
21. Batch Processing Calculate how many users are active on platform A and also on B Get Traffic of all Questions belonging to Channel X sorted by Country
22. Now to something completely different…
23. demo
24. with Myrrix Recommendations
25. Myrrix Evolution: taste -> mahout -> myrrix (-> oryx) Recommender based on ALS
26. Recommendations @ GF.net User emit signals on questions view, like, gives answer, answer is voted best Application sends signals through RabbitMQ to recommendation servers
27. but what happens, when a new user signs up? YEAH
28. ?
29. and feed it into myrrix Fetch data from tracking
30. using & processing is another thing ;) Collecting&Storing data works great