Extracting insights with the DataSift platform

Transcript

1. DataSift Social Data Mining Lorenzo Alberton, CTA [email protected] @lorenzoalberton Monday,

   11 June 2012

2. Listen and Act at scale Monday, 11 June 2012

3. What’s in a Tweet http://www.scribd.com/doc/30146338/map-of-a-tweet Monday, 11 June 2012

4. What’s in a Tweet http://www.scribd.com/doc/30146338/map-of-a-tweet actual tweet Monday, 11 June

   2012

5. What’s in a Tweet http://www.scribd.com/doc/30146338/map-of-a-tweet actual tweet metadata Monday, 11

   June 2012

6. Architecture Overview Monday, 11 June 2012

7. 1. Ingestion of input streams Monday, 11 June 2012

8. 1. Ingestion of input streams Monday, 11 June 2012

9. 1. Ingestion of input streams Monday, 11 June 2012

10. 2. Filtering Monday, 11 June 2012

11. 2. Filtering Filters (not just keywords!) Input Messages Monday, 11

    June 2012

12. 2. Filtering Filters (not just keywords!) Input Messages Custom Output

    Streams Monday, 11 June 2012

13. 3. Delivery + API + Frontend Monday, 11 June 2012

14. 3. Delivery + API + Frontend Monday, 11 June 2012

15. 3. Delivery + API + Frontend Monday, 11 June 2012

16. 4. Monitoring/Storage/Analytics Monday, 11 June 2012

17. 4. Monitoring/Storage/Analytics Monday, 11 June 2012

18. Works Historically Too Monday, 11 June 2012

19. Works Historically Too Monday, 11 June 2012

20. Works Historically Too Monday, 11 June 2012

21. Works Historically Too Monday, 11 June 2012

22. 13 http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 8 7 2 2 5 2 60+ 7

    7 7 7 7 7 8 2 5 2 Monday, 11 June 2012

23. 350+ Million input messages/day * ~340 Million from the Twitter

    Firehose alone Monday, 11 June 2012

24. 2 Terabyte messages processed in real time and stored every

    day ~1 Petabyte of storage available Monday, 11 June 2012

25. Thousands of concurrent, custom output streams all crafted with tender

    love and surgical precision Monday, 11 June 2012

26. Lexalytics Usage Stats 7 servers 16x 2.40GHz cores/server ~400% CPU

    per instance ~20ms per interaction 4 languages Sentiment, Topics, Entities Monday, 11 June 2012

27. Messages Volume by Language 0 2K 3K 1K Portuguese Spanish

    English (long) English (short) French Monday, 11 June 2012

28. Average CPU Load by Language 0 7 12 5 Portuguese

    + French + Spanish Portuguese + Spanish English (short + long) Monday, 11 June 2012

29. Sifting Gold - Case Studies http://www.the-exponent.com/wp-content/uploads/2010/09/Sifting-Gold1.jpg Monday, 11 June 2012

30. Combine Different Signals Buzz Volume Sentiment Topics Entities Authority/Influence Hard-Data

    Content being shared Monday, 11 June 2012

31. RIM CEOs resignation Monday, 11 June 2012

32. RIM CEOs resignation Monday, 11 June 2012

33. RIM CEOs resignation Monday, 11 June 2012

34. RIM CEOs resignation Monday, 11 June 2012

35. RIM CEOs resignation RIM stock opens and falls Monday, 11

    June 2012

36. RIM CEOs resignation RIM stock opens and falls sentiment switches

    to positive Monday, 11 June 2012

37. RIM CEOs resignation RIM stock opens and falls sentiment switches

    to positive 5 mins later the stock price reaches support level Monday, 11 June 2012

38. Crowd reaction for the news • Significance of the event

    proportional to spike of traffic generated • Switch from negative to positive sentiment when the market decided the price had been discounted enough Monday, 11 June 2012

39. Facebook IPO Monday, 11 June 2012

40. Sentiment + Demographic Distribution http://www.forbes.com/sites/jeffbercovici/2012/05/17/on-twitter-theyre-bullish-about-facebook/ Monday, 11 June 2012

41. Sentiment/Share price Correlation http://mashable.com/2012/05/19/facebook-ipo-twitter-prediction/ Monday, 11 June 2012

42. The Most Popular Topics 1% random sample Monday, 11 June

    2012

43. Boris Johnson vs Ken Livingstone London Mayoral Election 2012 Monday,

    11 June 2012

44. Boris Johnson vs Ken Livingstone London Mayoral Election 2012 very

    positive sentiment spike for Boris at the end of the campaign Monday, 11 June 2012

45. Social Interest Analysis What are fashion lovers in London talking

    about? Monday, 11 June 2012

46. Bring Feeds Into Your Application Monday, 11 June 2012

47. Discover Social Interest of Customers Monday, 11 June 2012

48. Q&A [email protected] @lorenzoalberton Monday, 11 June 2012