Implementing Webhooks: not so trivial!

Implementing Webhooks: not so trivial!

Webhooks, we know what they are, right? You’ve probably already used some to be notified of Github commits, to react to text messages received via Twilio, or created a fulfillment for a Dialogflow chatbot to answer users. From the Webhook consumer side, it doesn’t seem complicated to code… On the server-side, is it really just a web API to implement? Hmm… maybe not! Actually, we’ll see that it might be not as simple as it first sounds ! After an introduction on the concept of Webhooks, we’ll create our own callbacks to be notified of events. Then, we’ll go on the other side of the mirror, by creating our own Webhook backend. We’ll study how to deal with client subscription queues, manage all kinds of errors, debug the webhook, handle retries to avoid flooding subscribers, or how to secure those hooks. There’s lots to cover!

137d3908243acfc30e126615d59d4e6d?s=128

Guillaume Laforge

September 17, 2019
Tweet

Transcript

  1. @glaforge WEB HOOKS WEB HOOKS Not as trivial as it

    may seem Not as trivial as it may seem @glaforge
  2. @glaforge Long time no see...

  3. @glaforge Introduction

  4. @glaforge Ask a service to notify you via an HTTP

    callback to a URL you specify when an event occur Ask a service to notify you via an HTTP callback to a URL of your choice when an event occur
  5. @glaforge Server to server realtime notification Simple way to connect

    apps together
  6. @glaforge

  7. @glaforge Emails • SendGrid • MailChimp Chat messages • Dialogflow

    • Intercom Payments • Stripe • BrainTree Build results • TravisCI • CircleCI Who else is using webhooks?
  8. @glaforge Advantages

  9. @glaforge Realtime

  10. @glaforge No polling

  11. @glaforge No broadcast

  12. @glaforge Inconvenients

  13. @glaforge Not in control

  14. @glaforge Github Webhooks DEMO

  15. @glaforge Check runs Check suites Commit comments Branch / tag

    creation Branch / tag deletion Deploy keys Deployments Deployment statuses Forks Wiki Issue comments Issues Labels Collaborator Milestones Page builds Projects Project cards Project columns Visibility changes Pull requests PR reviews PR review comments Pushes Registry packages Releases Repositories Repository imports Repository vuln. alerts Stars Statuses Team adds Watches
  16. @glaforge Implementing Webhooks

  17. @glaforge Develop & deploy a handler to receive POST requests

    Register the handler’s URL to the service provider Service provider sends a requests to your handler when an event occur 1 2 3 Add a webhook mechanism to a service provider 0
  18. @glaforge Status codes: retry if not 2xx

  19. @glaforge Exponential back off

  20. @glaforge Handler bombed by too many event notifications or retries

  21. @glaforge Batch multiple events together

  22. @glaforge Missed calls

  23. @glaforge Dead letter queue

  24. @glaforge Google Cloud Pub/Sub

  25. @glaforge Google Cloud Pub/Sub

  26. @glaforge Idempotent

  27. @glaforge Security

  28. @glaforge Use HTTPS

  29. @glaforge Whitelist IP addresses

  30. @glaforge Sign requests with user-provided secret

  31. @glaforge Ngrok and RequestBin DEMO

  32. @glaforge Demo: ngrok, requestbin

  33. @glaforge Apply good security principles: Authentication Authorization

  34. @glaforge Some best practices

  35. @glaforge Data, or not data

  36. @glaforge Webhook handlers should answer rapidly

  37. @glaforge Handlers should do nothing

  38. @glaforge Enqueue calls and handling

  39. @glaforge Google Cloud Task

  40. @glaforge Zzz… what if nobody calls...

  41. @glaforge Webhooks benefit from serverless solutions

  42. @glaforge Cloud Functions DEMO

  43. @glaforge Cloud Functions, Cloud Run, App Engine

  44. @glaforge Google Cloud Functions

  45. @glaforge Summary

  46. @glaforge Implementing Webhooks CLIENT • Reply with 200 • Reply

    fast • Ack reception and defer work to a worker queue • Calls should be idempotent • IP whitelisting • Check request signature • Take advantage of serverless solutions SERVER • Send small data payloads (re-fetch) • Timeout if client too slow • Retry with exponential backoff • Keep track of delivery with a worker queue • Batch events when too frequent • Use a dead letter queue for auditing • Use HTTPS for secured connections • Sign requests with a secret • Use proper authentication / authorization solutions
  47. @glaforge Resources

  48. @glaforge Resources Crafting a great webhooks experience (John Sheehan) https://speakerdeck.com/apistrat/crafting-a-great-webhooks-experience-by-john-sheehan

    WebHooks: the definitive guide https://requestbin.com/blog/working-with-webhooks/ WebHooks: The API Strikes Back (InfoQ) https://www.infoq.com/presentations/webhooks-api/ Webhooks vs APIs https://hackernoon.com/webhook-vs-api-whats-the-difference-8d41e6661652 What is a Webhooks push-style API & how does it work (ProgrammableWeb) https://www.programmableweb.com/news/what-webhooks-push-styled-api-and-how-does-it-work/analysis/The2017/03/28 Webhooks do’s & dont’s: what we learned after integration 100+ APIs https://restful.io/webhooks-dos-and-dont-s-what-we-learned-after-integrating-100-apis-d567405a3671#.s0qgt1i9p
  49. @glaforge Resources How & why Pusher adopted Webhooks https://www.programmableweb.com/news/what-are-webhooks-and-how-do-they-enable-real-time-web/2012/01/30 Webhooks

    vs WebSub: Which Is Better For Real-Time Event Streaming? https://nordicapis.com/webhooks-vs-websub-which-one-is-better-to-stream-your-events-in-real-time/ Webhooks, the devil is in the details https://techblog.commercetools.com/webhooks-the-devil-in-the-details-ca7f7982c24f#.u49yswnm7 How to design a webhook for my API https://phalt.github.io/webhooks-in-apis/ Serverless webhooks to revolutionize the SaaS https://tomasz.janczuk.org/2018/03/serverless-webhooks-to-revolutionize-the-saas.html