Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Oh, the APIs You'll Call!

Jerry Chen
September 26, 2017

Oh, the APIs You'll Call!

An overview of APIs, Twitter APIs, and how to build things that call them.

Jerry Chen

September 26, 2017
Tweet

Other Decks in Programming

Transcript

  1. The Rundown • What is an API? • What are

    some of Twitter's APIs? • Putting it all together • Useful tools of the trade
  2. What is an API? • Early civilizations had no concept

    of zero, or APIs • In our case: well-defined interfaces for external data sources
  3. What are some Twitter APIs? • (Classic™) REST API: largely

    user-centric, everything you need to build your own client • Search API: search through much of Twitter history • Full-Archive Search: search through ALL of Twitter history. Every. Single. Tweet. • Engagement API: every interaction with owned tweets
  4. Other Twitter API Types ☎ • Streaming: No, you hang

    up • Webhooks: Don't call us, we'll call you
  5. More on Streaming Twitter APIs • Examples: PowerTrack & Decahose

    • Connect through long-lived HTTPS connection • Streams events in either Original Format or Activity Stream formats • Bring your own consumer • Enforce your own compliance
  6. More on Streaming Twitter APIs • Examples: PowerTrack & Decahose

    & Compliance • Connect through long-lived HTTPS connection • Streams events in either Original Format or Activity Stream formats • Bring your own consumer • Enforce your own compliance
  7. Let's bout Building a Report • Let's make it fast

    • Let's make it resilient • Let's iterate quickly using various Twitter REST APIs
  8. Let's bout Building a Report • Let's make it fast

    • Let's make it resilient • Let's iterate quickly using various Twitter REST APIs
  9. Make it fast • Spend most of the report time

    waiting on the network • Spend the least amount of time waiting on the network
  10. Serial Access Example: Find your hashtag buddy • Get User

    Timeline (@jcsalterego) • Determine Most Used Hashtag
  11. Serial Access Example: Find your hashtag buddy • Get User

    Timeline (@jcsalterego) • Determine Most Used Hashtag • Search for Tweets with Hashtag (#wendys)
  12. Serial Access • Get User Timeline (@jcsalterego) • Determine Most

    Used Hashtag • Search for Tweets with Hashtag (#wendys) • Get Timeline of Top Hashtag User Example: Find your hashtag buddy
  13. Serial Access Example: Find your hashtag buddy • Get User

    Timeline (@jcsalterego) • Determine Most Used Hashtag • Search for Tweets with Hashtag (#wendys) • Get Timeline of Top Hashtag User
  14. Concurrent Access • Search for #tacos has:links • Get search

    counts for top three URLs Example: Find related hashtags by link affinity
  15. Concurrent Access • Search for #tacos has:links • Get search

    counts for top three URLs • Search for tacodeli.com has:hashtags Example: Find related hashtags by link affinity
  16. Concurrent Access • Search for #tacos has:links • Get search

    counts for top three URLs • Search for tacodeli.com has:hashtags • Get search counts for top three hashtags Example: Find related hashtags by link affinity
  17. Concurrent Access • Search for #tacos has:links • Get search

    counts for top three URLs • Search for tacodeli.com has:hashtags • Get search counts for top three hashtags • Choose the top hashtag Example: Find related hashtags by link affinity
  18. • Search for #tacos has:links • Get search counts for

    top three URLs • Search for tacodeli.com has:hashtags • Get search counts for top three hashtags • Choose the top hashtag Concurrent Access #doña Example: Find related hashtags by link affinity
  19. • Search for #tacos has:links • Get search counts for

    top three URLs • Search for tacodeli.com has:hashtags • Get search counts for top three hashtags • Choose the top hashtag Concurrent Access #doña Example: Find related hashtags by link affinity
  20. 1 2 3 4 5 6 7 8 9 •

    Search for #tacos has:links • Get search counts for top three URLs • Search for tacodeli.com has:hashtags • Get search counts for top three hashtags • Choose the top hashtag Concurrent Access #doña Example: Find related hashtags by link affinity
  21. • Search for #tacos has:links • Get search counts for

    top three URLs • Search for tacodeli.com has:hashtags • Get search counts for top three hashtags • Choose the top hashtag Example: Find related hashtags by link affinity As Serial Access #doña 1 2 3 4 5 6 7 8 9
  22. • Search for #tacos has:links • Get search counts for

    top three URLs • Search for tacodeli.com has:hashtags • Get search counts for top three hashtags • Choose the top hashtag Example: Find related hashtags by link affinity As Serial Access #doña 1 2 3 4 5 6 7 8 9
  23. In Practice: Java Futures CompletableFuture<List<Tweet>> ownedTweetsFuture = getPublicTweetsFuture(userId); CompletableFuture<List<TweetEngagement>> engagementFuture

    = ownedTweetsFuture.thenApply( (List<Tweet> tweets) -> { return getEngagementData(tweets); }); CompletableFuture<String> topHashtagFuture = ownedTweetsFuture.thenApply( (List<Tweet> tweets) -> { return getTopHashtagByCount(tweets, 3); });
  24. In Practice: Java Futures CompletableFuture<List<Tweet>> ownedTweetsFuture = getPublicTweetsFuture(userId); CompletableFuture<List<TweetEngagement>> engagementFuture

    = ownedTweetsFuture.thenApply( (List<Tweet> tweets) -> { return getEngagementData(tweets); }); CompletableFuture<String> topHashtagFuture = ownedTweetsFuture.thenApply( (List<Tweet> tweets) -> { return getTopHashtagByCount(tweets, 3); }); CompletableFuture.allOf( engagementFuture, topHashtagFuture).get();
  25. Let's bout Building a Report • Let's make it fast

    • Let's make it resilient • Let's iterate quickly using various Twitter REST APIs
  26. Make it resilient • Expect failures • API: Rate Limited,

    Permissions Revoked • Network: Timeouts, Partitions, DNS • Other: Solar Flare
  27. Rate Limits • Know them (per-minute, per-second windows) • If

    retrying, use reasonable backoff • Look at the headers • Cache aggressively
  28. Let's bout Building a Report • Let's make it fast

    • Let's make it resilient • Let's iterate quickly using various Twitter REST APIs
  29. Iterate quickly Cache responses to memory Save responses to disk:

    fixtures! Use fixtures for testing Make self-referential Expanding Brain Meme
  30. Tools of the trade • curl • Twitter for Mac

    Developer Console • jq • pbpaste
  31. Toolkit: jq $ cat tweet.json | jq .
 {
 "id":

    912340256745230300,
 "id_str": "912340256745230337",
 "in_reply_to_screen_name": "cowboyd",
 "in_reply_to_status_id": 912340150503510000,
 "in_reply_to_status_id_str": "912340150503510017",
 "user": {
 "contributors_enabled": false,
 "created_at": "Wed Apr 09 17:28:47 +0000 2008",
 "default_profile": false,
 "default_profile_image": false,
  32. Toolkit: jq $ cat tweet.json | jq .
 {
 "id":

    912340256745230300,
 "id_str": "912340256745230337",
 "in_reply_to_screen_name": "cowboyd",
 "in_reply_to_status_id": 912340150503510000,
 "in_reply_to_status_id_str": "912340150503510017",
 "user": {
 "contributors_enabled": false,
 "created_at": "Wed Apr 09 17:28:47 +0000 2008",
 "default_profile": false,
 "default_profile_image": false,
  33. $ cat tweet.json | jq .
 {
 "id": 912340256745230300,
 "id_str":

    "912340256745230337",
 "in_reply_to_screen_name": "cowboyd",
 "in_reply_to_status_id": 912340150503510000,
 "in_reply_to_status_id_str": "912340150503510017",
 "user": {
 "contributors_enabled": false,
 "created_at": "Wed Apr 09 17:28:47 +0000 2008",
 "default_profile": false,
 "default_profile_image": false, 64-bit IEEE 754 = 53 bits of precision Toolkit: jq
  34. $ cat tweet.json | jq .
 {
 "id": 912340256745230300,
 "id_str":

    "912340256745230337",
 "in_reply_to_screen_name": "cowboyd",
 "in_reply_to_status_id": 912340150503510000,
 "in_reply_to_status_id_str": "912340150503510017",
 "user": {
 "contributors_enabled": false,
 "created_at": "Wed Apr 09 17:28:47 +0000 2008",
 "default_profile": false,
 "default_profile_image": false, 64-bit IEEE 754 = 53 bits of precision Toolkit: jq ¯\_(ツ)_/¯
  35. Toolkit: pbpaste & jq $ pbpaste | jq .id_str
 "912340256745230337"


    $ pbpaste | jq .user.screen_name
 "jcsalterego"
 $ pbpaste | jq .
 {
 "id": 912340256745230300,
 "id_str": "912340256745230337",
 "in_reply_to_screen_name": "cowboyd",
 "in_reply_to_status_id": 912340150503510000,
 "in_reply_to_status_id_str": "912340150503510017",
 "user": {
 "contributors_enabled": false,
 "created_at": "Wed Apr 09 17:28:47 +0000 2008",

  36. Toolkit: jqpaste $ cat > jqpaste
 #!/bin/bash
 pbpaste | jq

    -S $@
 ^D
 $ chmod +x jqpaste
 $ jqpaste .
 {
 "id": 912340256745230300,
 "id_str": "912340256745230337",
 "in_reply_to_screen_name": "cowboyd",
 "in_reply_to_status_id": 912340150503510000,
 "in_reply_to_status_id_str": "912340150503510017",
 "user": {
 "contributors_enabled": false,
  37. Toolkit: cURL Logging INFO 2017-09-26 03:44:25.050 [gnip-query-63 ] c.u.common.http.AuthorizedHttpClient -

    curl -H 'Authorization: Basic nourbasic==' 'https://data- api.twitter.com/search/fullarchive/prod/counts.json? query=wendys&fromDate=201708270344&bucket=minute'