Don’t reduce dispatcher testing to an afterthought

Frankly speaking, dispatcher usually does not receive much attention when it comes to testing. In most cases configuration updates get inspected only on the go without any regression tests being triggered alongside.

Neglected dispatcher may lead to application issues that often slip through unnoticed across lower level environments and surface right after production deployment.

Overlooked dispatcher testing doesn’t come out of nowhere - it’s neither straightforward nor easy to implement a reliable test suite for it. Moreover, dispatcher is often not verbose enough and all its surroundings would have to be mocked out to test it in isolation.

This talk will describe various techniques (man-in-the-middle proxying, mocking strategies, configuration tricks, CI/CD pipeline examples, etc) to let you test and promote your dispatcher config with confidence.

Transcript

1. EUROPE'S LEADING AEM DEVELOPER CONFERENCE 25th – 27th SEPTEMBER 2023 Don’t reduce dispatcher testing to an afterthought Jakub Wądołowski, diva-e (@jwadolowski)
2. Is dispatcher testing worth it? ▪ “Dispatcher is a one-off setup” ▪ “It doesn’t do much” ▪ “An over-engineering?” https://flic.kr/p/anyLkn
3. Project recap ▪ e-commerce platform ▪ AEMaaCS as a headless content API (JSON) ▪ Stateless API ▪ 100% cacheable ▪ Numerous HTTP clients https://flic.kr/p/djkkPw
6. What could possibly go wrong? ▪ Request/response modifications ▪ Caching (dispatcher/CDN) ▪ Cache bypassing ▪ Request filtering / security ▪ Error handling ▪ … https://flic.kr/p/jdZ178
7. The right tool for the job ▪ Simple ▪ Fast (AEM mocks) ▪ Easy to maintain & extend ▪ CI/CD friendly https://flic.kr/p/9Z1Q9Q
9. The tools ▪ Hurl ▪ command-line HTTP client ▪ HTTP test tool ▪ Rust + libcurl ▪ Dispatcher ▪ official SDK Docker image ▪ mitmproxy ▪ powerful man-in-the-middle proxy (Python) https://flic.kr/p/jPY82T
10. Basic HTTP flow ▪ 1st request expectations ▪ 200 OK ▪ valid JSON ▪ cached by dispatcher ▪ 2nd request expectations ▪ conditional GET (304 Not Modified) ▪ empty body ▪ served from cache (no AEM request) https://flic.kr/p/9v71eS
25. Hurl reporting ▪ Verbose flag(s) ▪ JUnit ▪ HTML https://flic.kr/p/dgge8U
27. What about compression? https://flic.kr/p/E3DufC
33. Broken conditional GETs ▪ GET response ▪ Read response body ▪ Calculate ETag (mod_core; “FileETag Digest”) ▪ Compress (mod_deflate; “-gzip” suffix in ETag) ▪ Conditional GET request ▪ Read request ▪ Compare If-None-Match and ETag (mod_core) ▪ “abcde-gzip” != “abcde” https://flic.kr/p/8rihKK
35. Cache hit or miss? https://flic.kr/p/FEzhCB
36. Let dispatcher talk! https://flic.kr/p/2jyA3fL
40. ETag - the hard way https://flic.kr/p/F3JzM
50. What went wrong? ▪ No browser cache - no problem ▪ CDN supports multiple variations per URL ▪ Browser stores only one variation ▪ Not all headers are mandatory in 304 response ▪ ETag’s fault again? https://flic.kr/p/6VBNjW
51. ETag rules ▪ ETag and Vary must come hand in hand ▪ Response body checksum is not enough ▪ Cover all variations ▪ CORS headers ▪ Encoding ▪ Language ▪ … https://flic.kr/p/qwVMGS
56. AEMaaCS domains & layers ▪ Domain landscape ▪ adobeaemcloud.com (CDN) ▪ custom ones (CDN) ▪ adobeaemcloud.net (k8s ingress) ▪ localhost (k8s pod) ▪ and a few more :) ▪ Layers: CDN / k8s ingress / Dispatcher / AEM https://flic.kr/p/bNVzTz
57. Dispatcher cache poisoning ▪ Absolute URLs in response body ▪ External domain sources ▪ Host header ▪ OSGi config (CM variable) ▪ API crawling functional test https://flic.kr/p/kDBhHa
64. Cache poisoning learnings ▪ Functional tests must bypass dispatcher cache ▪ Normalise Host header https://flic.kr/p/6kH5V8
65. Links ▪ https://hurl.dev/ ▪ https://mitmproxy.org/ https://flic.kr/p/83eFJY
66. Thank you!