Webhooks: Lessons (Un)learned

Webhooks are a pillar of modern application development. They notify us of that new commit, an incoming text message, our email was delivered, and a payment was processed. Our systems can’t function without webhooks sending data seamlessly and securely across the internet.

But what happens if they’re not secure? What happens if your webhooks are intercepted, manipulated, or even replayed against your systems? What are the best ways - as both a provider and consumer - to protect our systems?

In this session, we’ll delve into the 100+ implementations we explored to build webhooks.fyi to identify the best and worst patterns to protect our systems now and in the future.

Transcript

1. Webhooks Lessons (Un)Learned Danger Casey API Problem Solver, GTM Guy, General Nuisance danger@caseysoftware.com July 2023
3. © ngrok. All rights reserved. Confidential Information of ngrok • Get out your phone • Open Postman, Insomnia, or curl (for bonus points) • Tweet at @ngrokHQ if I say something useful Requests for you
4. © ngrok. All rights reserved. Confidential Information of ngrok Agenda 01 Intro 02 Webhooks 101 03 The Fatal Flaw 04 Our Risks 05 What should Providers do? 06 What should Consumers do? 07 Closing / Q&A
5. 01 Who am I?
6. © ngrok. All rights reserved. Confidential Information of ngrok Who am I?
7. © ngrok. All rights reserved. Confidential Information of ngrok Who am I? https://www.youtube.com/@geekamongthetrees
8. © ngrok. All rights reserved. Confidential Information of ngrok Why are you talking about this? Ref: https://webhooks.fyi/
9. 02 Webhooks 101
10. © ngrok. All rights reserved. Confidential Information of ngrok Webhooks: In the Beginning Ref: https://progrium.github.io/blog/2007/05/03/web-hooks-to-revolutionize-the-web/
11. © ngrok. All rights reserved. Confidential Information of ngrok Sidebar on Jeff Lindsay • Coined the concept of “web hooks” • Early employee at Twilio • Created localtunnel (inspiration for ngrok) • Architect at dotCloud (which created Docker)
12. © ngrok. All rights reserved. Confidential Information of ngrok Back to webhooks..
13. © ngrok. All rights reserved. Confidential Information of ngrok • Simple protocol: HTTP • Simple payload: JSON or XML • Tech stack agnostic • Share state between systems • Building everything yourself sucks • Super easy to spoof & compromise 😈 • Er.. I mean super easy to test and mock But why webhooks?
14. 03 The Fatal Flaw
15. © ngrok. All rights reserved. Confidential Information of ngrok How do we secure this?
16. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok Let’s play with it
17. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok (go to demo)
18. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok (606) 531-1770
19. 04 Our Risks
20. © ngrok. All rights reserved. Confidential Information of ngrok What we’re doing
21. © ngrok. All rights reserved. Confidential Information of ngrok • Interception • Impersonation • Modification/Manipulation • Replay attacks • Downtime (missing requests) • Forward Compatibility • Multicast Destinations Risks & Considerations
22. © ngrok. All rights reserved. Confidential Information of ngrok • Interception • Impersonation • Modification/Manipulation • Replay attacks • Downtime (missing requests) • Forward Compatibility • Multicast Destinations Risks & Considerations
23. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok Let’s play with it abuse it
24. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok (back to demo)
25. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok (606) 531-1770
26. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok Use Postman to POST to https://thatrocks.ngrok.io
27. 05 Best Practices for Providers
28. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok https
29. © ngrok. All rights reserved. Confidential Information of ngrok • Show the end to end process ◦ Including parameters, urls, and options • Show a payload ◦ Yes, all of them • Demonstrate verification • Embed verification in your libraries Docs
30. © ngrok. All rights reserved. Confidential Information of ngrok Secure your payload
31. © ngrok. All rights reserved. Confidential Information of ngrok • Mitigates replay ◦ Requires a (somewhat) sync’d clock ◦ Choose one format (unix timestamp ftw) ◦ Include it in the signature Use the timestamp (or request id) Ref: https://webhooks.fyi/security/replay-prevention
32. © ngrok. All rights reserved. Confidential Information of ngrok Multi-version support Ref: https://webhooks.fyi/ops-experience/versioning • X-PagerDuty-Signature: v1=f03de6f61df6e454f3620c4d6aca17ad072d3f8bbb2760eac3b2ad 391b5e8073, v2=130dcacb53a94d983a37cf2acba98e805a1c37185309ba56fdcccb cf00d6dd8b • X-HubSpot-Signature-v2: <signed value> X-HubSpot-Signature-v3: <signed value>
33. © ngrok. All rights reserved. Confidential Information of ngrok • X-PagerDuty-Signature: v1=f03de6f61df6e454f3620c4d6aca17ad072d3f8bbb2760eac3b2ad 391b5e8073, v1=130dcacb53a94d983a37cf2acba98e805a1c37185309ba56fdcccb cf00d6dd8b Zero Downtime (key rotation) Ref: https://webhooks.fyi/ops-experience/versioning
34. © ngrok. All rights reserved. Confidential Information of ngrok State of webhooks
35. © ngrok. All rights reserved. Confidential Information of ngrok State of webhooks (cont)
36. 06 Best Practices for Consumers
37. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok https
38. © ngrok. All rights reserved. Confidential Information of ngrok What we’re doing
39. © ngrok. All rights reserved. Confidential Information of ngrok What we should actually do
40. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok (back to demo) * with verification
41. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok (606) 531-1770
42. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok Use Postman to POST to https://tekrocks.ngrok.io
43. Closing Thoughts 07
44. © ngrok. All rights reserved. Confidential Information of ngrok //todo for providers • https or death • Doc all of it - patterns, payloads, & code • Strong authN methods • Mitigate replay attacks (timestamps) • Bonus points: Testing mechanism • Bonus points: Multi-version support Ref: https://webhooks.fyi/best-practices/webhook-providers
45. © ngrok. All rights reserved. Confidential Information of ngrok • Transact over https • Verify senders • Validate payloads • Check (on critical events) • Bonus points: IP restrictions //todo for consumers Ref: https://webhooks.fyi/best-practices/webhook-consumers
46. © ngrok. All rights reserved. Confidential Information of ngrok
47. Thank you
48. © ngrok. All rights reserved. Confidential Information of ngrok © ngrok. All rights reserved. Confidential Information of ngrok About those bonus points for curl If you show me that you used curl, let me know today and I’ll give you a free year of ngrok Pro.
49. https://that.land/43z1eGt
51. Webhooks Lessons (Un)Learned Danger Casey API Problem Solver, GTM Guy, General Nuisance danger@caseysoftware.com July 2023