Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lessons from integrating 1,500 APIs

C056051a062fb1054cb45d98177384aa?s=47 Fokke Zandbergen
September 24, 2019
Technology
0
87

Lessons from integrating 1,500 APIs

Talk at WebCore Amsterdam (https://www.meetup.com/WebCore/) about what Zapier is, what we've run into integrating thousands of APIs and some technical details.

C056051a062fb1054cb45d98177384aa?s=128

Fokke Zandbergen

September 24, 2019
Tweet

More Decks by Fokke Zandbergen

See All by Fokke Zandbergen
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
58
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
53
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
32
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
160
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
320
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
2
640
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
220
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
3
390
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
2
600

Other Decks in Technology

See All in Technology
6b8e2101579190ad96e747e01c279898?s=48 pauloxnet
2
110
F89f6a80305746253d4bedb128b2f445?s=48 mats16
0
170
49c19cca70c23c71d0f86da45be2eb61?s=48 puttyo__
0
110
De266761b955b2636e454a1bc7a99ed4?s=48 masayaaoyama
0
890
88964b936e864ca7d326272eaa70fa9a?s=48 hgsgtk
2
1.6k
646a01801bac1886ddf86aee2de913ed?s=48 miyukki
0
1k
Dc20c893cac0daddd607dfc9a5855d27?s=48 amarelo_n24
0
180
Ad5eddc28171c05461bf198681c2b828?s=48 kazuki88
0
210
69b93af68320a590f607c296e8edff73?s=48 k1low
8
2.1k
966e29b84ae7dbde170f66b0bc75b7a6?s=48 ishiayaya
PRO
0
100
7a54d6143169419fc6d4d9a0ff3056ea?s=48 maekawa123
0
630
9a38c60104a629648564527582d071c6?s=48 ta28masa
0
130

Featured

See All Featured
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
101
11k
39488f9d172ab92fd352f2cd7b73258d?s=48 mza
81
4.2k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
56
5.5k
433acaea1012b25d97ae66da9b998dad?s=48 malarkey
193
8.8k
0bce06bd06ee7a2c4f5500f25dcf7f18?s=48 paulrobertlloyd
73
1.5k
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
33
3.6k
F383c6a4dc55e331bbe2987b622cee6b?s=48 stephaniewalter
262
11k
C620790ae5bf5b50c245b2e0ef95f338?s=48 deanohume
293
28k
027d1ebf66cc039a0bd3b55eeadbe75d?s=48 sachag
446
36k
0bce06bd06ee7a2c4f5500f25dcf7f18?s=48 paulrobertlloyd
71
3.7k
B83d5b590577969ee79a5ad845411a7d?s=48 ammeep
657
54k
7653c7c449918ff6542880a8c284f5e7?s=48 jponch
103
5.2k

Transcript

  1. None

  2. Date Lessons from integrating 1,500 APIs WebCore - September 24,

    2019

  3. Fokke Zandbergen Full Stack Engineer fokkezb.nl ❋ twitter.com/fokkezb ❋ linkedin.com/in/fokkezb

    ❋ github.com/fokkezb Fokke Zandbergen

  4. About Zapier

  5. None

  6. Milestones • • September 2011: Idea • May 2012: Y

    Combinator Graduation • September 2012: Raised $1.3M • May 2014: Profitable • June 2016: At 1M Users, $10M ARR • August 2017: At 100 Zapiens (or Zorps) • January 2018: At 1K Apps, $35M ARR • January 2019: At 1,5K Apps, $50M ARR

  7. But what does it do?!

  8. … or in developer speak

  9. User Experience

  10. Developer Experience

  11. … or using the SDK

  12. What makes an API easy to integrate with Zapier?

  13. Don’t reinvent the wheel • • REST JSON • OAuth

    2.0 (w/refresh) • REST Hooks * • OpenAPI *

  14. Don’t save on docs to be... • • There •

    Complete • Versioned • Public * • Standard (OpenAPI / Swagger)

  15. • They should never expire… • ...unless we have a

    refresh token • (which should never expire) • ...unless the user revokes it. We encrypt and censor them, please do so to. Secret tokens are Sacred •

  16. • Limiting: number | timestamp • Pagination: (number | token)

    & meta • Filtering & Searching • Field Masks: fields GraphQL via the SDK: yarn add apollo-boost cross-fetch Work that body!

  17. • Field: created_at • Direction: DESC Assuming the above, what

    will go wrong here? ?limit=10 &is_published=true Orderrrr!

  18. • Field: id • Composite Fields: obj.object_id = obj.id; obj.id

    = hash([id,updated_at]); What else is new? •

  19. • Please call us. We’ll only call you once. POST

    you.com/subscribe?url=zapier.com/1d2x • Better UX than (static) webhooks (+24%) • Reduces server load & traffic (-98.5%) • Triggers Zaps instantly Required for public apps resthooks.org Get Hooked!

  20. HTTP/1.0 200 OK Content-Type: application/json { "status": "error", "code": 404,

    "message": "Object is not found." } HTTP Status Codes have a purpose

  21. … as do HTTP Methods GET /api/book?title=APIs+For+Dummies HTTP/1.0 201 Created

    Content-Type text/plain Created “APIs For Dummies”

  22. … and Content-Type HTTP/1.0 400 Bad Request Content-Type: application/json Parameter

    'id' is missing. GET /books Accept: application/json HTTP/1.0 401 Unauthorized Content-Type: text/html <html> <title>Login</title> ...

  23. How to have millions of conversations with 1,500 apps… daily?

  24. Polling Triggers • Shedding: Skip % of regular polling request

    • Deduplication (simplified pseudo) hash = md5(obj.id) ZADD(zapId, hash) ZSCAN(zapId, hash) • Held Tasks: This… can’t be right… right?

  25. 1. Response always written to S3 for (manual) replay. 2.

    First pass by trigger integration to parse response. After that, same for polling triggers: 3. For each object, write state to ES. 4. Queue first/next step for worker boxes. 5. Worker passes state to step trigger/action. 6. Write updated state and GOTO 4 Webhook Triggers

  26. • Integrations are NodeJS apps. • Generated AWS Lambda function

    for each app version. • Lambda called to run trigger or action. output = createOrder(z, { auth, input }); • Bound to AWS limits for size of the app (50MB), payload (6MB) and time to complete (30 seconds). Running Integration Code

  27. • Each integration sandboxed. • Only access to data mapped

    to the action’s input fields. • No binary data, just public (!) URLs to download data. publicUrl = z.stash(z.request(privateUrl)) • Credentials stored, encrypted, on a separate MS. • Credentials and token-like string censored in logs. Security

  28. Thank you!

  29. Is This You? Python React Scaling for 10x RabbitMQ SQL

    Django k8s empathy Redis default to action GraphQL Next.js Flow Typescript Node

  30. Join us. From anywhere. zapier.com/jobs