Lessons from integrating 1,500 APIs

Date Lessons from integrating 1,500 APIs WebCore - September 24,
2019

Fokke Zandbergen Full Stack Engineer fokkezb.nl ❋ twitter.com/fokkezb ❋ linkedin.com/in/fokkezb
❋ github.com/fokkezb Fokke Zandbergen

About Zapier

Milestones • • September 2011: Idea • May 2012: Y
Combinator Graduation • September 2012: Raised $1.3M • May 2014: Proﬁtable • June 2016: At 1M Users, $10M ARR • August 2017: At 100 Zapiens (or Zorps) • January 2018: At 1K Apps, $35M ARR • January 2019: At 1,5K Apps, $50M ARR

But what does it do?!

… or in developer speak

User Experience

Developer Experience

… or using the SDK

What makes an API easy to integrate with Zapier?

Don’t reinvent the wheel • • REST JSON • OAuth
2.0 (w/refresh) • REST Hooks * • OpenAPI *

Don’t save on docs to be... • • There •
Complete • Versioned • Public * • Standard (OpenAPI / Swagger)

• They should never expire… • ...unless we have a
refresh token • (which should never expire) • ...unless the user revokes it. We encrypt and censor them, please do so to. Secret tokens are Sacred •

• Limiting: number | timestamp • Pagination: (number | token)
& meta • Filtering & Searching • Field Masks: fields GraphQL via the SDK: yarn add apollo-boost cross-fetch Work that body!

• Field: created_at • Direction: DESC Assuming the above, what
will go wrong here? ?limit=10 &is_published=true Orderrrr!

• Field: id • Composite Fields: obj.object_id = obj.id; obj.id
= hash([id,updated_at]); What else is new? •

• Please call us. We’ll only call you once. POST
you.com/subscribe?url=zapier.com/1d2x • Better UX than (static) webhooks (+24%) • Reduces server load & traﬃc (-98.5%) • Triggers Zaps instantly Required for public apps resthooks.org Get Hooked!

HTTP/1.0 200 OK Content-Type: application/json { "status": "error", "code": 404,
"message": "Object is not found." } HTTP Status Codes have a purpose

… as do HTTP Methods GET /api/book?title=APIs+For+Dummies HTTP/1.0 201 Created
Content-Type text/plain Created “APIs For Dummies”

… and Content-Type HTTP/1.0 400 Bad Request Content-Type: application/json Parameter
'id' is missing. GET /books Accept: application/json HTTP/1.0 401 Unauthorized Content-Type: text/html <html> <title>Login</title> ...

How to have millions of conversations with 1,500 apps… daily?

Polling Triggers • Shedding: Skip % of regular polling request
• Deduplication (simpliﬁed pseudo) hash = md5(obj.id) ZADD(zapId, hash) ZSCAN(zapId, hash) • Held Tasks: This… can’t be right… right?

1. Response always written to S3 for (manual) replay. 2.
First pass by trigger integration to parse response. After that, same for polling triggers: 3. For each object, write state to ES. 4. Queue ﬁrst/next step for worker boxes. 5. Worker passes state to step trigger/action. 6. Write updated state and GOTO 4 Webhook Triggers

• Integrations are NodeJS apps. • Generated AWS Lambda function
for each app version. • Lambda called to run trigger or action. output = createOrder(z, { auth, input }); • Bound to AWS limits for size of the app (50MB), payload (6MB) and time to complete (30 seconds). Running Integration Code

• Each integration sandboxed. • Only access to data mapped
to the action’s input ﬁelds. • No binary data, just public (!) URLs to download data. publicUrl = z.stash(z.request(privateUrl)) • Credentials stored, encrypted, on a separate MS. • Credentials and token-like string censored in logs. Security

Thank you!

Is This You? Python React Scaling for 10x RabbitMQ SQL
Django k8s empathy Redis default to action GraphQL Next.js Flow Typescript Node

Join us. From anywhere. zapier.com/jobs

Lessons from integrating 1,500 APIs

Lessons from integrating 1,500 APIs

WebCore Meetup

More Decks by WebCore Meetup

Other Decks in Programming

Featured

Transcript

Date Lessons from integrating 1,500 APIs WebCore - September 24,

Fokke Zandbergen Full Stack Engineer fokkezb.nl ❋ twitter.com/fokkezb ❋ linkedin.com/in/fokkezb

About Zapier

Milestones • • September 2011: Idea • May 2012: Y

But what does it do?!

… or in developer speak

User Experience

Developer Experience

… or using the SDK

What makes an API easy to integrate with Zapier?

Don’t reinvent the wheel • • REST JSON • OAuth

Don’t save on docs to be... • • There •

• They should never expire… • ...unless we have a

• Limiting: number | timestamp • Pagination: (number | token)

• Field: created_at • Direction: DESC Assuming the above, what

• Field: id • Composite Fields: obj.object_id = obj.id; obj.id

• Please call us. We’ll only call you once. POST

HTTP/1.0 200 OK Content-Type: application/json { "status": "error", "code": 404,

… as do HTTP Methods GET /api/book?title=APIs+For+Dummies HTTP/1.0 201 Created

… and Content-Type HTTP/1.0 400 Bad Request Content-Type: application/json Parameter

How to have millions of conversations with 1,500 apps… daily?

Polling Triggers • Shedding: Skip % of regular polling request

1. Response always written to S3 for (manual) replay. 2.

• Integrations are NodeJS apps. • Generated AWS Lambda function

• Each integration sandboxed. • Only access to data mapped

Thank you!

Is This You? Python React Scaling for 10x RabbitMQ SQL

Join us. From anywhere. zapier.com/jobs