Design A Dark Traffic System With GoReplay

A slides for Golang Taipei Gathering #38 on Jan, 31th, 2019

Transcript

1. Design A Dark Traffic System With GoReplay Golang Taipei Gathering #38, Thu, Jan 31th 2019 1 Sean Hsieh 謝任軒 <of_alpha@hotmail.com> Software engineer, Garmin International
2. 2 • Software engineer • Garmin International (Jul 2017 ~ Present) • MOXA.inc (May 2014 ~ April 2017) • Information Sean Hsieh 謝任軒 Linkedin Medium
3. 3 Introduction to the dark traffic A traffic listener and player: GoReplay Design a dark traffic system Design A Dark Traffic System With GoReplay
4. Design A Dark Traffic System With GoReplay 4 Introduction to the dark traffic A traffic listener and player: GoReplay Design a dark traffic system
5. Testing before release 5 • System behave differently depends on environment and traffic pattern Production Staging • Integration tests • Load tests • Smoke test Pass & Release V 1.1 SQA Real users V 1.2 Rollback V 1.1 V 1.2
6. Dark traffic (so called shadowing or mirroring) 6 • Sampling real traffic is the only way to reliably capture the request path Staging V 1.2 V 1.1 Sampling HTTP/HTTPS Replaying the requests to prod and staging Real users Production Compare the response, record and verify Pass & Release V 1.2
7. The solutions of dark traffic 7 • The best solution depends on the project Router-based Package-based • Facebook’s McRouter: • Sample entire prod traffics • GoReplay : • Sniff the HTTP packages and replay
8. For non-Idempotent or stateful web service? 8 • Or you need to keep sync the database Staging V 1.2 V 1.1 Sampling HTTP/HTTPS Replaying the requests to prod and staging Real users Production Compare the response, record and verify Log out Failed!
9. Summary • Choose the proper solution for your team and products 9 You need to consider that… • Shadowing may be suitable for: 1. Idempotent services (冪等) 2. Stateless services (無狀態) 3. Or you have to keep in-sync the data from the prod database to the staging database
10. Design A Dark Traffic System With GoReplay 10 Introduction to the dark traffic A traffic listener and player: GoReplay Design a dark traffic system
11. GoReplay is a tool to test the system with real traffics 11 • It’s a Golang project which is based on WinPcap (Windows) • Ability to capture and replay(or accelerate) the traffics • Ability to plugin the middleware
12. GoReplay usage scenario 12 Real users HTTP Gor replay //Replay traffics to the B site from files gor --input-file data/request_0.gor –output-http=http://staging -output-http="http://production" Gor listen Staging service //Capture traffics from the specific port and output the traffic files gor --input-raw :3000 -output-file= "data/request.gor" Prod service Listen port 3000 //Replay traffics to the B site in the real time gor –input-raw:3000 –output-http=http://staging -output-http="http://production" Traffic files
13. Summary • GoReplay can’t support HTTPS traffics • According to the author: 13 You need to consider that… What???
14. Design A Dark Traffic System With GoReplay 14 Introduction to the dark traffic A traffic listener and player: GoReplay Design a dark traffic system
15. Architecture of the dark traffic system 15 Staging V 1.2 V 1.1 Sample HTTP/HTTPS Compare the response Record and verify Real users Production Replay the requests to prod and staging Build a control panel SQA
16. 1. How GoRelpay samples HTTPS ? 16 //Create a file server gor --file-server :8800 //Capture the packages gor --input-raw :8800 -output-file=data/request.gor • Duplicate HTTPS packages + SSL termination + Gor file server Real users Nginx Gor file server Prod service Gor listen HTTPS Traffic files HTTPS HTTP
17. Nginx: Duplicate HTTPS packages + SSL termination 17 http { server { listen 443; server_name IP/domain name; ssl on; ssl_certificate C:/nginx-1.14.1/ssl/gor_proxy.crt; ssl_certificate_key C:/nginx-1.14.1/ssl/gor_proxy.key.org; #charset koi8-r; #access_log logs/host.access.log main; location / { proxy_pass https://a.com; post_action @afterdownload; } location @afterdownload { proxy_pass http://b.com$uri?$args; } } }
18. 18 2 How GoReplay compares responses? • Author suggest that using the middleware
19. Node.js middleware framework 19 var gor = require("goreplay_middleware"); var map = {}; gor.init(); gor.on("message", function(msg) { if ((msg.ID in map) === false) { map[msg.ID] = [] } if (msg.type === '1') { map[msg.ID].push({ 'Method': gor.httpMethod(msg.http), 'Status': gor.httpStatus(msg.http), 'Body': gor.httpBody(msg.http).toString() }) } else { map[msg.ID].push({ 'Body': gor.httpBody(msg.http).toString(), 'Status': gor.httpStatus(msg.http) }) } if (map[msg.ID].length == 3) { console.error(`Replay 1's body: ${map[msg.ID][1]['Body']}`) console.error(`Replay 2's body: '${map[msg.ID][2]['Body']}`) console.error(`Replay 1's status: ${map[msg.ID][1]['Status']}`) console.error(`Replay 2's status: ${map[msg.ID][2]['Status']}`) } return msg; }) //Replay traffics to the B site from files gor --input-file data/request_0.gor –output-http=http://localhost:8800 -output-http=http://B --output-http-track-response --input-raw-track- response --middleware "node ~\app.js" (2) (3)
20. 20 3. How GoReplay records compared results? • Middleware • ElasticSearch • Kibana
21. Node.js middleware framework 21 var elasticsearch = require('elasticsearch'); const client = new elasticsearch.Client({ hosts: [ 'http://localhost:9200'] }); client.ping({ requestTimeout: 30000, }, function(error) { if (error) { console.error('Elasticsearch cluster is down!'); } else { console.error('Everything is ok'); } }); client.indices.create({ index: 'gor' }, function(error, response, status) { if (error) { console.error(error); } else { console.error("created a new index", response); } }); //Replay traffics to the B site from files gor --input-file data/request_0.gor –output-http=http://localhost:8800 -output-http=http://B --output-http-track-response --input-raw-track- response --middleware "node ~\app.js"
22. 22 Thanks! Any questions? You can find me at: Linkedin Medium