Visualize Twitter Hashtags in Real Time

This is a talk I gave at SDRuby (sdruby.org) about the process of monitoring the Twitter live stream API for certain keywords and then visualizing retweets in real time using Arbor.js.

Transcript

1. Visualize Twitter #tags in Real Time Etienne de Bruin @etdebruin
2. But first...
3. This changed my life
6. Instead...
18. MonkDev, Inc. • Established 2006 • Old Town, San Diego • 20 Employees • We host Ruby Hack Nights! • NEXT ONE IS NEXT WEEK (NONW)!
19. I Devs
21. Awesome new project • Visualize Retweets in real time • 4 stakeholders • 31% equity • 2 months to do it • GO! We demo in May.
33. Streaming API REST API
34. Streaming API • Monitor different streams • Streaming • User Streams • Site Streams (beta)
35. Streaming API
36. Streaming API • Rate Limited • One persistent connection per account • intridea/tweetstream • OAuth
37. REST API • Rate Limited • Unauthenticated 150/hr (~2/min) • Authenticated 350/hr (~5/min) • jnunemaker/twitter
38. REST API
39. Monitor Stream by #tag enqueue user_id
40. Monitor Stream by #tag enqueue user_id Resque Worker
41. Monitor Stream by #tag enqueue user_id Resque Worker
42. Redis follower_id:follows = user_id
43. @mokolabs tweets @etdebruin:follows @mokolabs @orangewolf:follows @mokolabs @rubiety:follows @mokolabs
44. @bensie retweets (via @mokolabs) @rafamvc:follows @bensie @rweald:follows @bensie @merbist:follows @bensie
45. Redis user_id_follows = follower_id:follows
46. Redis @rubiety:follows => @mokolabs
47. @merbist retweets (via @bensie) @camba:follows @merbist @mekdigital:follows @merbist @tenderlove:follows @merbist @evanphx:follows @merbist @drnic:follows @merbist
48. Redis @merbist:follows => @bensie
49. Dude
50. Dude • GET statuses/:id/retweeted_by
51. Dude • GET statuses/:id/retweeted_by • GET statuses/retweets/:id
52. Dude • GET statuses/:id/retweeted_by • GET statuses/retweets/:id • Returns first 100 objects
53. Dude • GET statuses/:id/retweeted_by • GET statuses/retweets/:id • Returns first 100 objects • No relationships
54. But, Dude?! • retweet_count • in_reply_to_user_id • retweeted
55. @mokolabs @bensie RT @merbist RT
56. @mokolabs @bensie RT @merbist RT
57. Two types of RT • Retweet • retweeted_status: true • original status_id (and user_id) • Quote Tweet • no mention of retweet • no original status_id • no guarantees on status text
58. Dealing with Quote RT’s • Introduce additional dimension • Unique • bit.ly/asdf
59. incr Increment tweet counter set/get Store as string type rpush Append tweet to list zadd Add user to sorted set lrange Return list of new tweets expire Expire api counter key
60. a graph visualization library using web workers and jQuery
61. treedata controller
62. treedata controller 5 second poll /:id/:offset
63. treedata controller 5 second poll /:id/:offset lrange :offset + n
64. treedata controller 5 second poll /:id/:offset lrange :offset + n list
65. treedata controller 5 second poll /:id/:offset json lrange :offset + n list
66. from @bensie to @merbist {"from":"272162572", "to":"283758257", "pic":"http://a1.tmg.com/imgs/1315295649/m.jpg", "followers":"6", "friends":"1", "location":"", "klout":"10", "date":"Fri Dec 23 08:27:49 +0000 2011"},
67. Heroku Procfile web: bundle exec thin start -e $RACK_ENV -p $PORT worker: QUEUE=* bundle exec rake environment resque:work stream: bundle exec lib/tasks/stream_campaign.rb
71. Thankyou! Rails Basics @rafamvc Ruby Basics @merbist Heroku Basics @orangewolf
73. Goto user @sdruby @stevenbono