Jan Stępień - Tracking those who Track

Talk by Jan Stępień at the firsta Munich DataGeeks Meetup
Data: 02.07.2013

Transcript

1. Tracking those who track us Jan Stępień
2. My name is Jan Stępień and I come from Warsaw
3. Data analysis is not just big data
4. Data analysis is fun
5. It all started with ads tracking “like” buttons other irrelevant things
6. 1. Use an adblock plugin 2. Block all network communication to unwelcome domains
7. My machine website.com ads.website.com
8. My machine website.com ads.website.com
9. Let’s capture all those requests!
10. 03.2012 – 06.2013 106 414 requests 322 distinct days approx. 330 requests per day
11. SQLite3 + Incanter + R + Weka
12. http_if_none_match http_referer http_accept_encoding http_accept http_cookie http_connection http_host http_user_agent http_version path_info http_accept_charset http_accept_language http_cache_control http_if_modified_since request_method request_path request_uri query_string remote_host remote_addr script_name server_name server_port server_protocol http_dnt timestamp
13. timestamp
14. 03 04 05 06 07 08 09 10 11 12 01 02 03 04 05 06 15k 10k 5k 0
15. 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 100 0 200 300 400 500
16. 8k 6k 4k 2k 0 Mo Tu We Th Fr Sa Su
17. http_host
18. www.google-analytics.com 36197 static.adzerk.net 13983 edge.quantserve.com 11659 www.facebook.com 9641 ad.doubleclick.net 3822 pagead2.googlesyndication.com 3764 s.youtube.com 2173 b.scorecardresearch.com 1974 pubads.g.doubleclick.net 1465 googleads.g.doubleclick.net 1231
19. 48.9% of requests sent to domains owned by Google
20. http_referer
21. 22902 distinct referrers 4692 distinct domains
22. Let’s try to combine this dataset with something else
23. Weather influence?
24. ogimet.com Humidity, min/max/avg temperature, cloud coverage, visibility, rain/snow, wind speed/direction, etc.
25. No correlations!
26. Tags at stackoverflow.com
27. http://stackoverflow.com/questions/123/title
28. data.stackexchange.com
29. Thanks, wordle.net!
30. Can be my WWW traffic grouped into clusters?
31. 1. Group requests into 15 minute intervals 2. Count domains per interval
32. 5008 intervals Each described by over 4500 values
33. 1. Select request from popular domains 2. Group requests into 15 minute intervals 3. Count domains per interval
34. 5008 intervals Each described by 95 values Only 2% of cells with non-zero values
35. Principal Component Analysis 95 domains → 16 descriptors
36. X-means K-means based clustering algorithm
37. cluster 0 1268 cluster 1 702 cluster 2 651 cluster 3 2387 What is the meaning behind these clusters?
38. 3 stackoverflow.com
39. 2 reddit.com redditmedia.com bbc.co.uk
40. 1 linkedin.com dictionary.reference.com meetup.com
41. 0 rubyonrails.pl developer.android.com tex.stackexchange.com amazon.com youtube.com
42. How accurate is this clustering? Let’s build a classifier on the original data
43. 0 1 2 3 ← classified as 1188 29 11 40 cluster 0 47 654 1 0 cluster 1 10 1 622 18 cluster 2 50 0 18 2319 cluster 3 cluster 0: rubyonrails.pl developer.android.com amazon.com youtube.com cluster 1: linkedin.com dictionary.reference.com meetup.com cluster 2: reddit.com redditmedia.com bbc.co.uk cluster 3: stackoverflow.com
44. Let’s wrap up
45. Data analysis is not just big data
46. Data analysis is fun
47. Thank you very much The picture of Warsaw is © Dennis Jarvis 2009