Save 37% off PRO during our Black Friday Sale! »

Design A Dark Traffic System With GoReplay

C88188fbe1bb33ec383a47c2c767825e?s=47 Sean Hsieh
January 31, 2019

Design A Dark Traffic System With GoReplay

A slides for Golang Taipei Gathering #38 on Jan, 31th, 2019

C88188fbe1bb33ec383a47c2c767825e?s=128

Sean Hsieh

January 31, 2019
Tweet

Transcript

  1. Design A Dark Traffic System With GoReplay Golang Taipei Gathering

    #38, Thu, Jan 31th 2019 1 Sean Hsieh 謝任軒 <of_alpha@hotmail.com> Software engineer, Garmin International
  2. 2 • Software engineer • Garmin International (Jul 2017 ~

    Present) • MOXA.inc (May 2014 ~ April 2017) • Information Sean Hsieh 謝任軒 Linkedin Medium
  3. 3 Introduction to the dark traffic A traffic listener and

    player: GoReplay Design a dark traffic system Design A Dark Traffic System With GoReplay
  4. Design A Dark Traffic System With GoReplay 4 Introduction to

    the dark traffic A traffic listener and player: GoReplay Design a dark traffic system
  5. Testing before release 5 • System behave differently depends on

    environment and traffic pattern Production Staging • Integration tests • Load tests • Smoke test Pass & Release V 1.1 SQA Real users V 1.2 Rollback V 1.1 V 1.2
  6. Dark traffic (so called shadowing or mirroring) 6 • Sampling

    real traffic is the only way to reliably capture the request path Staging V 1.2 V 1.1 Sampling HTTP/HTTPS Replaying the requests to prod and staging Real users Production Compare the response, record and verify Pass & Release V 1.2
  7. The solutions of dark traffic 7 • The best solution

    depends on the project Router-based Package-based • Facebook’s McRouter: • Sample entire prod traffics • GoReplay : • Sniff the HTTP packages and replay
  8. For non-Idempotent or stateful web service? 8 • Or you

    need to keep sync the database Staging V 1.2 V 1.1 Sampling HTTP/HTTPS Replaying the requests to prod and staging Real users Production Compare the response, record and verify Log out Failed!
  9. Summary • Choose the proper solution for your team and

    products 9 You need to consider that… • Shadowing may be suitable for: 1. Idempotent services (冪等) 2. Stateless services (無狀態) 3. Or you have to keep in-sync the data from the prod database to the staging database
  10. Design A Dark Traffic System With GoReplay 10 Introduction to

    the dark traffic A traffic listener and player: GoReplay Design a dark traffic system
  11. GoReplay is a tool to test the system with real

    traffics 11 • It’s a Golang project which is based on WinPcap (Windows) • Ability to capture and replay(or accelerate) the traffics • Ability to plugin the middleware
  12. GoReplay usage scenario 12 Real users HTTP Gor replay //Replay

    traffics to the B site from files gor --input-file data/request_0.gor –output-http=http://staging -output-http="http://production" Gor listen Staging service //Capture traffics from the specific port and output the traffic files gor --input-raw :3000 -output-file= "data/request.gor" Prod service Listen port 3000 //Replay traffics to the B site in the real time gor –input-raw:3000 –output-http=http://staging -output-http="http://production" Traffic files
  13. Summary • GoReplay can’t support HTTPS traffics • According to

    the author: 13 You need to consider that… What???
  14. Design A Dark Traffic System With GoReplay 14 Introduction to

    the dark traffic A traffic listener and player: GoReplay Design a dark traffic system
  15. Architecture of the dark traffic system 15 Staging V 1.2

    V 1.1 Sample HTTP/HTTPS Compare the response Record and verify Real users Production Replay the requests to prod and staging Build a control panel SQA
  16. 1. How GoRelpay samples HTTPS ? 16 //Create a file

    server gor --file-server :8800 //Capture the packages gor --input-raw :8800 -output-file=data/request.gor • Duplicate HTTPS packages + SSL termination + Gor file server Real users Nginx Gor file server Prod service Gor listen HTTPS Traffic files HTTPS HTTP
  17. Nginx: Duplicate HTTPS packages + SSL termination 17 http {

    server { listen 443; server_name IP/domain name; ssl on; ssl_certificate C:/nginx-1.14.1/ssl/gor_proxy.crt; ssl_certificate_key C:/nginx-1.14.1/ssl/gor_proxy.key.org; #charset koi8-r; #access_log logs/host.access.log main; location / { proxy_pass https://a.com; post_action @afterdownload; } location @afterdownload { proxy_pass http://b.com$uri?$args; } } }
  18. 18 2 How GoReplay compares responses? • Author suggest that

    using the middleware
  19. Node.js middleware framework 19 var gor = require("goreplay_middleware"); var map

    = {}; gor.init(); gor.on("message", function(msg) { if ((msg.ID in map) === false) { map[msg.ID] = [] } if (msg.type === '1') { map[msg.ID].push({ 'Method': gor.httpMethod(msg.http), 'Status': gor.httpStatus(msg.http), 'Body': gor.httpBody(msg.http).toString() }) } else { map[msg.ID].push({ 'Body': gor.httpBody(msg.http).toString(), 'Status': gor.httpStatus(msg.http) }) } if (map[msg.ID].length == 3) { console.error(`Replay 1's body: ${map[msg.ID][1]['Body']}`) console.error(`Replay 2's body: '${map[msg.ID][2]['Body']}`) console.error(`Replay 1's status: ${map[msg.ID][1]['Status']}`) console.error(`Replay 2's status: ${map[msg.ID][2]['Status']}`) } return msg; }) //Replay traffics to the B site from files gor --input-file data/request_0.gor –output-http=http://localhost:8800 -output-http=http://B --output-http-track-response --input-raw-track- response --middleware "node ~\app.js" (2) (3)
  20. 20 3. How GoReplay records compared results? • Middleware •

    ElasticSearch • Kibana
  21. Node.js middleware framework 21 var elasticsearch = require('elasticsearch'); const client

    = new elasticsearch.Client({ hosts: [ 'http://localhost:9200'] }); client.ping({ requestTimeout: 30000, }, function(error) { if (error) { console.error('Elasticsearch cluster is down!'); } else { console.error('Everything is ok'); } }); client.indices.create({ index: 'gor' }, function(error, response, status) { if (error) { console.error(error); } else { console.error("created a new index", response); } }); //Replay traffics to the B site from files gor --input-file data/request_0.gor –output-http=http://localhost:8800 -output-http=http://B --output-http-track-response --input-raw-track- response --middleware "node ~\app.js"
  22. 22 Thanks! Any questions? You can find me at: Linkedin

    Medium