Testing before release 5 • System behave differently depends on environment and traffic pattern Production Staging • Integration tests • Load tests • Smoke test Pass & Release V 1.1 SQA Real users V 1.2 Rollback V 1.1 V 1.2
Dark traffic (so called shadowing or mirroring) 6 • Sampling real traffic is the only way to reliably capture the request path Staging V 1.2 V 1.1 Sampling HTTP/HTTPS Replaying the requests to prod and staging Real users Production Compare the response, record and verify Pass & Release V 1.2
The solutions of dark traffic 7 • The best solution depends on the project Router-based Package-based • Facebook’s McRouter: • Sample entire prod traffics • GoReplay : • Sniff the HTTP packages and replay
For non-Idempotent or stateful web service? 8 • Or you need to keep sync the database Staging V 1.2 V 1.1 Sampling HTTP/HTTPS Replaying the requests to prod and staging Real users Production Compare the response, record and verify Log out Failed!
Summary • Choose the proper solution for your team and products 9 You need to consider that… • Shadowing may be suitable for: 1. Idempotent services (冪等) 2. Stateless services (無狀態) 3. Or you have to keep in-sync the data from the prod database to the staging database
GoReplay is a tool to test the system with real traffics 11 • It’s a Golang project which is based on WinPcap (Windows) • Ability to capture and replay(or accelerate) the traffics • Ability to plugin the middleware
GoReplay usage scenario 12 Real users HTTP Gor replay //Replay traffics to the B site from files gor --input-file data/request_0.gor –output-http=http://staging -output-http="http://production" Gor listen Staging service //Capture traffics from the specific port and output the traffic files gor --input-raw :3000 -output-file= "data/request.gor" Prod service Listen port 3000 //Replay traffics to the B site in the real time gor –input-raw:3000 –output-http=http://staging -output-http="http://production" Traffic files
Architecture of the dark traffic system 15 Staging V 1.2 V 1.1 Sample HTTP/HTTPS Compare the response Record and verify Real users Production Replay the requests to prod and staging Build a control panel SQA
1. How GoRelpay samples HTTPS ? 16 //Create a file server gor --file-server :8800 //Capture the packages gor --input-raw :8800 -output-file=data/request.gor • Duplicate HTTPS packages + SSL termination + Gor file server Real users Nginx Gor file server Prod service Gor listen HTTPS Traffic files HTTPS HTTP