Tweetlyze

A slide deck on my MS project which I built to understand trends on twitter.

Transcript

1. trend analysis
2. the motivation twitter has a lot of activity abstract trends on twitter trends are location specific
3. Why understand trends? It’s the society telling you what they want and what is currently on their mind. Cater to those needs and you have the power to grow your business.
4. Why understand trends? It’s the society telling you what they want and what is currently on their mind. Cater to those needs and you have the power to grow your business.
5. Example? Let us assume Manchester United are trending today in London and you are a t-shirt seller. Would it not be awesome if you know about this and you could give a bigger discount on Manchester United Shirts?
6. The Problems how to get data? what to analyze?
7. How to get data? REST Streaming twitter exposes
8. How to get data? REST Streaming Pros Cons can be handled with small infrastructure does not require a cleansing framework { { rate limits
9. How to get data? REST Streaming Pros Cons access to real time data need a big infrastructure to handle all the incoming data { { no rate limits lots of unwanted data
10. How to get data? REST Streaming i chose REST 200 GB { 25 days
11. how i overcame rate limits twitter app has a rate limit.
12. how i overcame rate limits twitter app has a rate limit create multiple twitter apps
13. how i overcame rate limits twitter app has a rate limit push oauth objects into a queue create multiple twitter apps
14. how i overcame rate limits twitter app has a rate limit create multiple twitter apps pushed oauth objects into queue distribute calls throughout the day
15. how i overcame rate limits twitter app has a rate limit create multiple twitter apps pushed oauth objects into queue distribute calls throughout the day all my apps stayed within rate limits
16. how i overcame rate limits twitter app has a rate limit create multiple twitter apps pushed oauth objects into queue distribute calls throughout the day all apps stayed within rate limits 45 days = 200 GB
17. how i overcame rate limits twitter app has a rate limit create multiple twitter apps pushed oauth objects into queue distribute calls throughout the day all apps stayed within rate limits [problem 1 solved] 45 Days = 200 GB
18. technologies used JAVA postgresql
19. What to analyze? trends
20. What to analyze? trends have tweets
21. What to analyze? trends have tweets entities
22. What to analyze? trends understand the relation tweets entities
23. understand the relation entities significance picking relevant tweets reducing tweet set
24. Final outcome? temporal and location based trends
25. Final outcome? temporal and location based trends entity significance per trend
26. Final outcome? temporal and location based trends entity significance per trend understanding similarity of trends
27. technologies used python html5 css3 js d3 long sql queries
28. Questions?
29. Thank you.