Save 37% off PRO during our Black Friday Sale! »

Lessons from integrating 1,500 APIs

C056051a062fb1054cb45d98177384aa?s=47 Fokke Zandbergen
September 24, 2019
Technology
0
91

Lessons from integrating 1,500 APIs

Talk at WebCore Amsterdam (https://www.meetup.com/WebCore/) about what Zapier is, what we've run into integrating thousands of APIs and some technical details.

C056051a062fb1054cb45d98177384aa?s=128

Fokke Zandbergen

September 24, 2019
Tweet

More Decks by Fokke Zandbergen

See All by Fokke Zandbergen
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
62
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
55
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
34
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
170
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
360
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
2
650
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
1
230
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
3
390
C056051a062fb1054cb45d98177384aa?s=48 fokkezb
2
620

Other Decks in Technology

See All in Technology
3dadebc0f79d74e50d8bfe5208cb7662?s=48 nadare881
0
920
140494d272a4d89883a94fdfdb29dea2?s=48 oracle4engineer
PRO
0
11k
942bb606679caf4c57b38927f83178e1?s=48 qsona
16
6k
3282faa17a370dd254382f4bf2c7ba3a?s=48 yukiiiiikuma
6
2k
8f84b7d8869ef6005d89b378e8661f7c?s=48 masuda220
PRO
4
330
140494d272a4d89883a94fdfdb29dea2?s=48 oracle4engineer
PRO
2
3.5k
8b27cf0be02118c4fb97c59b19f0f871?s=48 ykimi
0
2.7k
79c7cde17310f3a8ee640411baed289d?s=48 hayatow
0
150
Cc9e31933e199606c510ba9b0b0258eb?s=48 takashi_toyosaki
0
220
Ef2327689a92d784d7d63803140e833e?s=48 yulstat
0
390
66487f93d23ed4f64752cffbafdb938a?s=48 convto
0
160
211c8e4e5b96c116670c00cd77ebe2c1?s=48 rigolon
0
110

Featured

See All Featured
F57e2136eb9895eb95ff5dc8a1c02677?s=48 zakiwarfel
90
3.7k
78b475797a14c84799063c7cd073962f?s=48 holman
451
140k
282d599b414022eba5096510f675b6e2?s=48 chrislema
232
16k
016edace78ffe826f2348d1e0c504afd?s=48 maggiecrowley
16
960
Be9caeb9d4ef9944d151af909063ed6e?s=48 samlambert
238
10k
24fc194843a71f10949be18d5a692682?s=48 gr2m
86
11k
5f83ef806155b403c3393160ce51f955?s=48 jlugia
218
16k
3d6ace9554821d552146413bcdf874f6?s=48 trishagee
37
5.4k
4262b9cfacfb6b2c77f23e1d0310cd3e?s=48 myddelton
110
11k
A0517b08532aec8ab2aae3f65d40ad86?s=48 hannesfritz
32
1.2k
465724d73fe3a92c0879fdfb43a3a6f3?s=48 philhawksworth
196
17k
465724d73fe3a92c0879fdfb43a3a6f3?s=48 philhawksworth
195
9.1k

Transcript

  1. None

  2. Date Lessons from integrating 1,500 APIs WebCore - September 24,

    2019

  3. Fokke Zandbergen Full Stack Engineer fokkezb.nl ❋ twitter.com/fokkezb ❋ linkedin.com/in/fokkezb

    ❋ github.com/fokkezb Fokke Zandbergen

  4. About Zapier

  5. None

  6. Milestones • • September 2011: Idea • May 2012: Y

    Combinator Graduation • September 2012: Raised $1.3M • May 2014: Profitable • June 2016: At 1M Users, $10M ARR • August 2017: At 100 Zapiens (or Zorps) • January 2018: At 1K Apps, $35M ARR • January 2019: At 1,5K Apps, $50M ARR

  7. But what does it do?!

  8. … or in developer speak

  9. User Experience

  10. Developer Experience

  11. … or using the SDK

  12. What makes an API easy to integrate with Zapier?

  13. Don’t reinvent the wheel • • REST JSON • OAuth

    2.0 (w/refresh) • REST Hooks * • OpenAPI *

  14. Don’t save on docs to be... • • There •

    Complete • Versioned • Public * • Standard (OpenAPI / Swagger)

  15. • They should never expire… • ...unless we have a

    refresh token • (which should never expire) • ...unless the user revokes it. We encrypt and censor them, please do so to. Secret tokens are Sacred •

  16. • Limiting: number | timestamp • Pagination: (number | token)

    & meta • Filtering & Searching • Field Masks: fields GraphQL via the SDK: yarn add apollo-boost cross-fetch Work that body!

  17. • Field: created_at • Direction: DESC Assuming the above, what

    will go wrong here? ?limit=10 &is_published=true Orderrrr!

  18. • Field: id • Composite Fields: obj.object_id = obj.id; obj.id

    = hash([id,updated_at]); What else is new? •

  19. • Please call us. We’ll only call you once. POST

    you.com/subscribe?url=zapier.com/1d2x • Better UX than (static) webhooks (+24%) • Reduces server load & traffic (-98.5%) • Triggers Zaps instantly Required for public apps resthooks.org Get Hooked!

  20. HTTP/1.0 200 OK Content-Type: application/json { "status": "error", "code": 404,

    "message": "Object is not found." } HTTP Status Codes have a purpose

  21. … as do HTTP Methods GET /api/book?title=APIs+For+Dummies HTTP/1.0 201 Created

    Content-Type text/plain Created “APIs For Dummies”

  22. … and Content-Type HTTP/1.0 400 Bad Request Content-Type: application/json Parameter

    'id' is missing. GET /books Accept: application/json HTTP/1.0 401 Unauthorized Content-Type: text/html <html> <title>Login</title> ...

  23. How to have millions of conversations with 1,500 apps… daily?

  24. Polling Triggers • Shedding: Skip % of regular polling request

    • Deduplication (simplified pseudo) hash = md5(obj.id) ZADD(zapId, hash) ZSCAN(zapId, hash) • Held Tasks: This… can’t be right… right?

  25. 1. Response always written to S3 for (manual) replay. 2.

    First pass by trigger integration to parse response. After that, same for polling triggers: 3. For each object, write state to ES. 4. Queue first/next step for worker boxes. 5. Worker passes state to step trigger/action. 6. Write updated state and GOTO 4 Webhook Triggers

  26. • Integrations are NodeJS apps. • Generated AWS Lambda function

    for each app version. • Lambda called to run trigger or action. output = createOrder(z, { auth, input }); • Bound to AWS limits for size of the app (50MB), payload (6MB) and time to complete (30 seconds). Running Integration Code

  27. • Each integration sandboxed. • Only access to data mapped

    to the action’s input fields. • No binary data, just public (!) URLs to download data. publicUrl = z.stash(z.request(privateUrl)) • Credentials stored, encrypted, on a separate MS. • Credentials and token-like string censored in logs. Security

  28. Thank you!

  29. Is This You? Python React Scaling for 10x RabbitMQ SQL

    Django k8s empathy Redis default to action GraphQL Next.js Flow Typescript Node

  30. Join us. From anywhere. zapier.com/jobs