Implementing Webhooks, not as trivial as it may seem

Implementing Webhooks, not as trivial as it may seem

Webhooks, we know what they are, right? You’ve probably already used some to be notified of Github commits, to react to text messages received via Twilio, or created a fulfillment for a Dialogflow chatbot to answer users. From the Webhook consumer side, it doesn’t seem complicated to code… On the server-side, is it really just a web API to implement? Hmm… maybe not! Actually, we’ll see that it might be not as simple as it first sounds ! After an introduction on the concept of Webhooks, we’ll create our own callbacks to be notified of events. Then, we’ll go on the other side of the mirror, by creating our own Webhook backend. We’ll study how to deal with client subscription queues, manage all kinds of errors, debug the webhook, handle retries to avoid flooding subscribers, or how to secure those hooks. There’s lots to cover!

137d3908243acfc30e126615d59d4e6d?s=128

Guillaume Laforge

November 15, 2019
Tweet

Transcript

  1. @glaforge WEB HOOKS WEB HOOKS Not as trivial as it

    may seem Not as trivial as it may seem @glaforge
  2. @glaforge Introduction @glaforge

  3. @glaforge Ask a service to notify you via an HTTP

    callback to a URL you specify when an event occur Ask a service to notify you via an HTTP callback to a URL of your choice when an event occurs @glaforge
  4. @glaforge Server to server realtime notification Simple way to connect

    apps together @glaforge
  5. @glaforge @glaforge

  6. @glaforge Emails • SendGrid • MailChimp Chat messages • Dialogflow

    • Intercom Payments • Stripe • BrainTree Build results • TravisCI • CircleCI Who else is using webhooks? @glaforge
  7. @glaforge Advantages @glaforge

  8. @glaforge Realtime @glaforge

  9. @glaforge No polling @glaforge

  10. @glaforge No broadcast @glaforge

  11. @glaforge Inconvenients @glaforge

  12. @glaforge Not in control @glaforge

  13. @glaforge Check runs Check suites Commit comments Branch / tag

    creation Branch / tag deletion Deploy keys Deployments Deployment statuses Forks Wiki Issue comments Issues Labels Collaborator Milestones Page builds Projects Project cards Project columns Visibility changes Pull requests PR reviews PR review comments Pushes Registry packages Releases Repositories Repository imports Repository vuln. alerts Stars Statuses Team adds Watches @glaforge
  14. @glaforge Github Webhooks DEMO @glaforge

  15. @glaforge Implementing Webhooks @glaforge

  16. @glaforge Develop & deploy a handler to receive POST requests

    Register the handler’s URL to the service provider Service provider sends a requests to your handler when an event occurs 1 2 3 Add a webhook mechanism to a service provider 0 @glaforge
  17. @glaforge Status codes: retry if not 2xx @glaforge

  18. @glaforge Exponential back off PROVIDER @glaforge

  19. @glaforge Handler bombed by too many event notifications or retries

    @glaforge
  20. @glaforge Batch multiple events together PROVIDER @glaforge

  21. @glaforge Missed calls @glaforge

  22. @glaforge Dead letter queue PROVIDER @glaforge

  23. @glaforge Google Cloud Pub/Sub @glaforge

  24. @glaforge Google Cloud Pub/Sub @glaforge

  25. @glaforge Idempotent HANDLER @glaforge

  26. @glaforge Security @glaforge

  27. @glaforge Use HTTPS @glaforge

  28. @glaforge Whitelist IP addresses HANDLER @glaforge

  29. @glaforge Sign requests with user-provided secret PROVIDER @glaforge

  30. @glaforge Ngrok and RequestBin DEMO @glaforge

  31. @glaforge Apply good security principles: Authentication Authorization @glaforge

  32. @glaforge Some best practices @glaforge

  33. @glaforge Data, or not data @glaforge

  34. @glaforge Webhook handlers should answer rapidly HANDLER @glaforge

  35. @glaforge Handlers should do nothing HANDLER @glaforge

  36. @glaforge Enqueue calls and handling PROVIDER HANDLER @glaforge

  37. @glaforge Google Cloud Task @glaforge

  38. @glaforge Keep track of non-responding handlers PROVIDER @glaforge

  39. @glaforge Zzz… what if nobody calls... @glaforge

  40. @glaforge Webhooks benefit from serverless solutions HANDLER @glaforge

  41. @glaforge Cloud Functions DEMO @glaforge

  42. @glaforge Cloud Functions, Cloud Run, App Engine @glaforge

  43. @glaforge Google Cloud Functions @glaforge

  44. @glaforge Summary @glaforge

  45. @glaforge Implementing Webhooks CLIENT • Reply with 200 • Reply

    fast • Ack reception and defer work to a worker queue • Calls should be idempotent • IP whitelisting • Check request signature • Take advantage of serverless solutions SERVER • Send small data payloads (re-fetch) • Timeout if client too slow • Retry with exponential backoff • Keep track of non-responding handlers and delivery with a worker queue • Batch events when too frequent • Use a dead letter queue for auditing • Use HTTPS for secured connections • Sign requests with a secret • Use proper authentication / authorization solutions @glaforge
  46. @glaforge Thank you! @glaforge

  47. @glaforge Resources @glaforge

  48. @glaforge Resources Crafting a great webhooks experience (John Sheehan) https://speakerdeck.com/apistrat/crafting-a-great-webhooks-experience-by-john-sheehan

    WebHooks: the definitive guide https://requestbin.com/blog/working-with-webhooks/ WebHooks: The API Strikes Back (InfoQ) https://www.infoq.com/presentations/webhooks-api/ Webhooks vs APIs https://hackernoon.com/webhook-vs-api-whats-the-difference-8d41e6661652 What is a Webhooks push-style API & how does it work (ProgrammableWeb) https://www.programmableweb.com/news/what-webhooks-push-styled-api-and-how-does-it-work/analysis/The2017/03/28 Webhooks do’s & dont’s: what we learned after integration 100+ APIs https://restful.io/webhooks-dos-and-dont-s-what-we-learned-after-integrating-100-apis-d567405a3671#.s0qgt1i9p @glaforge
  49. @glaforge Resources How & why Pusher adopted Webhooks https://www.programmableweb.com/news/what-are-webhooks-and-how-do-they-enable-real-time-web/2012/01/30 Webhooks

    vs WebSub: Which Is Better For Real-Time Event Streaming? https://nordicapis.com/webhooks-vs-websub-which-one-is-better-to-stream-your-events-in-real-time/ Webhooks, the devil is in the details https://techblog.commercetools.com/webhooks-the-devil-in-the-details-ca7f7982c24f#.u49yswnm7 How to design a webhook for my API https://phalt.github.io/webhooks-in-apis/ Serverless webhooks to revolutionize the SaaS https://tomasz.janczuk.org/2018/03/serverless-webhooks-to-revolutionize-the-saas.html @glaforge