Lessons from integrating 1,500 APIs

C056051a062fb1054cb45d98177384aa?s=47 Fokke Zandbergen
September 24, 2019
Technology
0
81

Lessons from integrating 1,500 APIs

Talk at WebCore Amsterdam (https://www.meetup.com/WebCore/) about what Zapier is, what we've run into integrating thousands of APIs and some technical details.

C056051a062fb1054cb45d98177384aa?s=128

Fokke Zandbergen

September 24, 2019
Tweet

More Decks by Fokke Zandbergen

See All by Fokke Zandbergen
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
52
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
52
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
29
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
150
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
280
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
2
640
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
200
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
3
380
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
2
590

Other Decks in Technology

See All in Technology
B49933741d74e122bc1314b2975e9fc9?s=48 mrtc0
1
6.4k
34fcd5f1deeb25b1138e9845137feb6e?s=48 kakutani
2
580
Ad22fcf5773b906c08330f4d57242212?s=48 inductor
0
110
61a68b2172503ecdf7a7f7757df56071?s=48 toyship
0
110
0f993881fa8709dbe986bcade6c1fbad?s=48 satonaoki
0
110
229b1596ce57cd0935a2bacd410d87a0?s=48 aaaddress1
0
300
7a0bc9670ff654b04d4361a9afd1bba8?s=48 hcrane
17
1.2k
9df0566fad9c9ca3bf669917848878fe?s=48 i35_267
2
120
Cd931bc05e7cee46b9a950a63e47ba4c?s=48 you
0
120
9bf153d8c0ce36ba6d9d20c2914b70f4?s=48 shimastripe
9
440
55c9804c185f493816433678393cbe92?s=48 sumihiro3
0
150
53850955f15249a1a9dc49df6113e400?s=48 line_developers
0
470

Featured

See All Featured
C86443373fbfe92996aa882d0d7a59f8?s=48 imathis
478
150k
79acae287842acf29084005533a7fb3b?s=48 hursman
103
9.1k
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
65
2.9k
9128d500301ae51524e887bb680f471d?s=48 caitiem20
304
16k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
1
320
7cbd3f5f922d798cc312eaf596de7ae6?s=48 iamctodd
10
1.4k
64827b31aef81498662e8af4bd6d5157?s=48 jonrohan
1020
360k
B3ace07412a5178ea778e7eec161fc0a?s=48 jcasabona
7
420
7c8469ee8c9e594c65c59b919626c08d?s=48 scottboms
250
11k
24fc194843a71f10949be18d5a692682?s=48 gr2m
82
11k
95d9f37115fbb0d13135d0f77132f856?s=48 morganepeng
87
12k
5ce91fa49a613cbc3e20d5f96856473f?s=48 edds
55
9.2k

Transcript

  1. None

  2. Date Lessons from integrating 1,500 APIs WebCore - September 24,

    2019

  3. Fokke Zandbergen Full Stack Engineer fokkezb.nl ❋ twitter.com/fokkezb ❋ linkedin.com/in/fokkezb

    ❋ github.com/fokkezb Fokke Zandbergen

  4. About Zapier

  5. None

  6. Milestones • • September 2011: Idea • May 2012: Y

    Combinator Graduation • September 2012: Raised $1.3M • May 2014: Profitable • June 2016: At 1M Users, $10M ARR • August 2017: At 100 Zapiens (or Zorps) • January 2018: At 1K Apps, $35M ARR • January 2019: At 1,5K Apps, $50M ARR

  7. But what does it do?!

  8. … or in developer speak

  9. User Experience

  10. Developer Experience

  11. … or using the SDK

  12. What makes an API easy to integrate with Zapier?

  13. Don’t reinvent the wheel • • REST JSON • OAuth

    2.0 (w/refresh) • REST Hooks * • OpenAPI *

  14. Don’t save on docs to be... • • There •

    Complete • Versioned • Public * • Standard (OpenAPI / Swagger)

  15. • They should never expire… • ...unless we have a

    refresh token • (which should never expire) • ...unless the user revokes it. We encrypt and censor them, please do so to. Secret tokens are Sacred •

  16. • Limiting: number | timestamp • Pagination: (number | token)

    & meta • Filtering & Searching • Field Masks: fields GraphQL via the SDK: yarn add apollo-boost cross-fetch Work that body!

  17. • Field: created_at • Direction: DESC Assuming the above, what

    will go wrong here? ?limit=10 &is_published=true Orderrrr!

  18. • Field: id • Composite Fields: obj.object_id = obj.id; obj.id

    = hash([id,updated_at]); What else is new? •

  19. • Please call us. We’ll only call you once. POST

    you.com/subscribe?url=zapier.com/1d2x • Better UX than (static) webhooks (+24%) • Reduces server load & traffic (-98.5%) • Triggers Zaps instantly Required for public apps resthooks.org Get Hooked!

  20. HTTP/1.0 200 OK Content-Type: application/json { "status": "error", "code": 404,

    "message": "Object is not found." } HTTP Status Codes have a purpose

  21. … as do HTTP Methods GET /api/book?title=APIs+For+Dummies HTTP/1.0 201 Created

    Content-Type text/plain Created “APIs For Dummies”

  22. … and Content-Type HTTP/1.0 400 Bad Request Content-Type: application/json Parameter

    'id' is missing. GET /books Accept: application/json HTTP/1.0 401 Unauthorized Content-Type: text/html <html> <title>Login</title> ...

  23. How to have millions of conversations with 1,500 apps… daily?

  24. Polling Triggers • Shedding: Skip % of regular polling request

    • Deduplication (simplified pseudo) hash = md5(obj.id) ZADD(zapId, hash) ZSCAN(zapId, hash) • Held Tasks: This… can’t be right… right?

  25. 1. Response always written to S3 for (manual) replay. 2.

    First pass by trigger integration to parse response. After that, same for polling triggers: 3. For each object, write state to ES. 4. Queue first/next step for worker boxes. 5. Worker passes state to step trigger/action. 6. Write updated state and GOTO 4 Webhook Triggers

  26. • Integrations are NodeJS apps. • Generated AWS Lambda function

    for each app version. • Lambda called to run trigger or action. output = createOrder(z, { auth, input }); • Bound to AWS limits for size of the app (50MB), payload (6MB) and time to complete (30 seconds). Running Integration Code

  27. • Each integration sandboxed. • Only access to data mapped

    to the action’s input fields. • No binary data, just public (!) URLs to download data. publicUrl = z.stash(z.request(privateUrl)) • Credentials stored, encrypted, on a separate MS. • Credentials and token-like string censored in logs. Security

  28. Thank you!

  29. Is This You? Python React Scaling for 10x RabbitMQ SQL

    Django k8s empathy Redis default to action GraphQL Next.js Flow Typescript Node

  30. Join us. From anywhere. zapier.com/jobs