Design A Dark Traffic System With GoReplay

A slides for Golang Taipei Gathering #38 on Jan, 31th, 2019

Transcript

1. Design A Dark Traffic
   System With GoReplay
   Golang Taipei Gathering #38, Thu, Jan 31th 2019
   1
   Sean Hsieh 謝任軒
   Software engineer, Garmin International
   

   View Slide

2. 2
   • Software engineer
   • Garmin International (Jul 2017 ~ Present)
   • MOXA.inc (May 2014 ~ April 2017)
   • Information
   Sean Hsieh 謝任軒
   Linkedin
   Medium
   

   View Slide

3. 3
   Introduction to
   the dark traffic
   A traffic listener
   and player:
   GoReplay
   Design a dark
   traffic system
   Design A Dark Traffic
   System With GoReplay
   

   View Slide

4. Design A Dark Traffic
   System With GoReplay
   4
   Introduction to
   the dark traffic
   A traffic listener
   and player:
   GoReplay
   Design a dark
   traffic system
   

   View Slide

5. Testing before release
   5
   • System behave differently depends on environment and traffic pattern
   Production
   Staging
   • Integration tests
   • Load tests
   • Smoke test
   Pass & Release
   V 1.1
   SQA
   Real users
   V 1.2
   Rollback
   V 1.1
   V 1.2
   

   View Slide

6. Dark traffic (so called shadowing or mirroring)
   6
   • Sampling real traffic is the only way to reliably capture the request path
   Staging
   V 1.2
   V 1.1
   Sampling
   HTTP/HTTPS
   Replaying the requests
   to prod and staging
   Real users
   Production
   Compare the response,
   record and verify
   Pass & Release
   V 1.2
   

   View Slide

7. The solutions of dark traffic
   7
   • The best solution depends on the project
   Router-based Package-based
   • Facebook’s McRouter:
   • Sample entire prod traffics
   • GoReplay :
   • Sniff the HTTP packages and replay
   

   View Slide

8. For non-Idempotent or stateful web service?
   8
   • Or you need to keep sync the database
   Staging
   V 1.2
   V 1.1
   Sampling
   HTTP/HTTPS
   Replaying the requests
   to prod and staging
   Real users
   Production
   Compare the response,
   record and verify
   Log out
   Failed!
   

   View Slide

9. Summary
   • Choose the proper solution for your
   team and products
   9
   You need to consider that…
   • Shadowing may be suitable for:
   1. Idempotent services (冪等)
   2. Stateless services (無狀態)
   3. Or you have to keep in-sync the
   data from the prod database to
   the staging database
   

   View Slide

10. Design A Dark Traffic
    System With GoReplay
    10
    Introduction to
    the dark traffic
    A traffic listener
    and player:
    GoReplay
    Design a dark
    traffic system
    

    View Slide

11. GoReplay is a tool to test the system with real
    traffics
    11
    • It’s a Golang project which is based on WinPcap (Windows)
    • Ability to capture and replay(or accelerate) the traffics
    • Ability to plugin the middleware
    

    View Slide

12. GoReplay usage scenario
    12
    Real users
    HTTP Gor replay
    //Replay traffics to the B site from files
    gor --input-file data/request_0.gor –output-http=http://staging -output-http="http://production"
    Gor listen
    Staging service
    //Capture traffics from the specific port and output the traffic files
    gor --input-raw :3000 -output-file= "data/request.gor"
    Prod service
    Listen port 3000
    //Replay traffics to the B site in the real time
    gor –input-raw:3000 –output-http=http://staging -output-http="http://production"
    Traffic files
    

    View Slide

13. Summary
    • GoReplay can’t support HTTPS traffics
    • According to the author:
    13
    You need to consider that…
    What???
    

    View Slide

14. Design A Dark Traffic
    System With GoReplay
    14
    Introduction to
    the dark traffic
    A traffic listener
    and player:
    GoReplay
    Design a dark
    traffic system
    

    View Slide

15. Architecture of the dark traffic system
    15
    Staging
    V 1.2
    V 1.1
    Sample
    HTTP/HTTPS
    Compare the response
    Record and verify
    Real users
    Production
    Replay the requests to prod and
    staging
    Build a control panel
    SQA
    

    View Slide

16. 1. How GoRelpay
    samples HTTPS ?
    16
    //Create a file server
    gor --file-server :8800
    //Capture the packages
    gor --input-raw :8800 -output-file=data/request.gor
    • Duplicate HTTPS packages + SSL termination
    + Gor file server
    Real users Nginx
    Gor file server
    Prod service
    Gor listen
    HTTPS
    Traffic files
    HTTPS
    HTTP
    

    View Slide

17. Nginx:
    Duplicate HTTPS packages + SSL termination
    17
    http {
    server {
    listen 443;
    server_name IP/domain name;
    ssl on;
    ssl_certificate C:/nginx-1.14.1/ssl/gor_proxy.crt;
    ssl_certificate_key C:/nginx-1.14.1/ssl/gor_proxy.key.org;
    #charset koi8-r;
    #access_log logs/host.access.log main;
    location / {
    proxy_pass https://a.com;
    post_action @afterdownload;
    }
    location @afterdownload {
    proxy_pass http://b.com$uri?$args;
    }
    }
    }
    

    View Slide

18. 18
    2 How GoReplay
    compares responses?
    • Author suggest that using the middleware
    

    View Slide

19. Node.js middleware framework
    19
    var gor = require("goreplay_middleware");
    var map = {};
    gor.init();
    gor.on("message", function(msg) {
    if ((msg.ID in map) === false) {
    map[msg.ID] = []
    }
    if (msg.type === '1') {
    map[msg.ID].push({
    'Method': gor.httpMethod(msg.http),
    'Status': gor.httpStatus(msg.http),
    'Body': gor.httpBody(msg.http).toString()
    })
    } else {
    map[msg.ID].push({
    'Body': gor.httpBody(msg.http).toString(),
    'Status': gor.httpStatus(msg.http)
    })
    }
    if (map[msg.ID].length == 3) {
    console.error(`Replay 1's body: ${map[msg.ID][1]['Body']}`)
    console.error(`Replay 2's body: '${map[msg.ID][2]['Body']}`)
    console.error(`Replay 1's status: ${map[msg.ID][1]['Status']}`)
    console.error(`Replay 2's status: ${map[msg.ID][2]['Status']}`)
    }
    return msg;
    })
    //Replay traffics to the B site from files
    gor --input-file data/request_0.gor –output-http=http://localhost:8800
    -output-http=http://B --output-http-track-response --input-raw-track-
    response --middleware "node ~\app.js"
    (2)
    (3)
    

    View Slide

20. 20
    3. How GoReplay
    records compared
    results?
    • Middleware
    • ElasticSearch
    • Kibana
    

    View Slide

21. Node.js middleware framework
    21
    var elasticsearch = require('elasticsearch');
    const client = new elasticsearch.Client({
    hosts: [ 'http://localhost:9200']
    });
    client.ping({
    requestTimeout: 30000,
    }, function(error) {
    if (error) {
    console.error('Elasticsearch cluster is down!');
    } else {
    console.error('Everything is ok');
    }
    });
    client.indices.create({
    index: 'gor'
    }, function(error, response, status) {
    if (error) {
    console.error(error);
    } else {
    console.error("created a new index", response);
    }
    });
    //Replay traffics to the B site from files
    gor --input-file data/request_0.gor –output-http=http://localhost:8800
    -output-http=http://B --output-http-track-response --input-raw-track-
    response --middleware "node ~\app.js"
    

    View Slide

22. 22
    Thanks!
    Any questions?
    You can find me at:
    Linkedin
    Medium
    

    View Slide