Lessons from integrating 1,500 APIs

Talk at WebCore Amsterdam (https://www.meetup.com/WebCore/) about what Zapier is, what we've run into integrating thousands of APIs and some technical details.

Transcript

1. View Slide

2. Date
   Lessons from integrating
   1,500 APIs
   WebCore - September 24, 2019
   

   View Slide

3. Fokke Zandbergen
   Full Stack Engineer
   fokkezb.nl ❋ twitter.com/fokkezb ❋ linkedin.com/in/fokkezb ❋ github.com/fokkezb
   Fokke Zandbergen
   

   View Slide

4. About Zapier
   

   View Slide

5. View Slide

6. Milestones
   •
   • September 2011: Idea
   • May 2012: Y Combinator Graduation
   • September 2012: Raised $1.3M
   • May 2014: Profitable
   • June 2016: At 1M Users, $10M ARR
   • August 2017: At 100 Zapiens (or Zorps)
   • January 2018: At 1K Apps, $35M ARR
   • January 2019: At 1,5K Apps, $50M ARR
   

   View Slide

7. But what does it do?!
   

   View Slide

8. … or in developer speak
   

   View Slide

9. User Experience
   

   View Slide

10. Developer Experience
    

    View Slide

11. … or using the SDK
    

    View Slide

12. What makes an API easy to
    integrate with Zapier?
    

    View Slide

13. Don’t reinvent the wheel
    •
    • REST JSON
    • OAuth 2.0 (w/refresh)
    • REST Hooks *
    • OpenAPI *
    

    View Slide

14. Don’t save on docs to be...
    •
    • There
    • Complete
    • Versioned
    • Public *
    • Standard (OpenAPI / Swagger)
    

    View Slide

15. • They should never expire…
    • ...unless we have a refresh token
    • (which should never expire)
    • ...unless the user revokes it.
    We encrypt and censor them, please do so to.
    Secret tokens are Sacred
    •
    

    View Slide

16. • Limiting: number | timestamp
    • Pagination: (number | token) & meta
    • Filtering & Searching
    • Field Masks: fields
    GraphQL via the SDK:
    yarn add apollo-boost cross-fetch
    Work that body!
    

    View Slide

17. • Field: created_at
    • Direction: DESC
    Assuming the above,
    what will go wrong here?
    ?limit=10
    &is_published=true
    Orderrrr!
    

    View Slide

18. • Field: id
    • Composite Fields:
    obj.object_id = obj.id;
    obj.id = hash([id,updated_at]);
    What else is new?
    •
    

    View Slide

19. • Please call us. We’ll only call you once.
    POST you.com/subscribe?url=zapier.com/1d2x
    • Better UX than (static) webhooks (+24%)
    • Reduces server load & traffic (-98.5%)
    • Triggers Zaps instantly
    Required for public apps resthooks.org
    Get Hooked!
    

    View Slide

20. HTTP/1.0 200 OK
    Content-Type: application/json
    {
    "status": "error",
    "code": 404,
    "message": "Object is not found."
    }
    HTTP Status Codes have a purpose
    

    View Slide

21. … as do HTTP Methods
    GET /api/book?title=APIs+For+Dummies
    HTTP/1.0 201 Created
    Content-Type text/plain
    Created “APIs For Dummies”
    

    View Slide

22. … and Content-Type
    HTTP/1.0 400 Bad Request
    Content-Type: application/json
    Parameter 'id' is missing.
    GET /books
    Accept: application/json
    HTTP/1.0 401 Unauthorized
    Content-Type: text/html
    
    Login
    ...
    

    View Slide

23. How to have millions of
    conversations with 1,500
    apps… daily?
    

    View Slide

24. Polling Triggers
    • Shedding: Skip % of regular polling request
    • Deduplication (simplified pseudo)
    hash = md5(obj.id)
    ZADD(zapId, hash)
    ZSCAN(zapId, hash)
    • Held Tasks: This… can’t be right… right?
    

    View Slide

25. 1. Response always written to S3 for (manual) replay.
    2. First pass by trigger integration to parse response.
    After that, same for polling triggers:
    3. For each object, write state to ES.
    4. Queue first/next step for worker boxes.
    5. Worker passes state to step trigger/action.
    6. Write updated state and GOTO 4
    Webhook Triggers
    

    View Slide

26. • Integrations are NodeJS apps.
    • Generated AWS Lambda function for each app version.
    • Lambda called to run trigger or action.
    output = createOrder(z, { auth, input });
    • Bound to AWS limits for size of the app (50MB),
    payload (6MB) and time to complete (30 seconds).
    Running Integration Code
    

    View Slide

27. • Each integration sandboxed.
    • Only access to data mapped to the action’s input fields.
    • No binary data, just public (!) URLs to download data.
    publicUrl = z.stash(z.request(privateUrl))
    • Credentials stored, encrypted, on a separate MS.
    • Credentials and token-like string censored in logs.
    Security
    

    View Slide

28. Thank you!
    

    View Slide

29. Is This You?
    Python React
    Scaling for 10x
    RabbitMQ
    SQL
    Django
    k8s
    empathy
    Redis
    default to action
    GraphQL
    Next.js
    Flow
    Typescript
    Node
    

    View Slide

30. Join us. From anywhere.
    zapier.com/jobs
    

    View Slide