Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Design A Dark Traffic System With GoReplay

Sean Hsieh
January 31, 2019

Design A Dark Traffic System With GoReplay

A slides for Golang Taipei Gathering #38 on Jan, 31th, 2019

Sean Hsieh

January 31, 2019
Tweet

More Decks by Sean Hsieh

Other Decks in Technology

Transcript

  1. Design A Dark Traffic
    System With GoReplay
    Golang Taipei Gathering #38, Thu, Jan 31th 2019
    1
    Sean Hsieh 謝任軒
    Software engineer, Garmin International

    View Slide

  2. 2
    • Software engineer
    • Garmin International (Jul 2017 ~ Present)
    • MOXA.inc (May 2014 ~ April 2017)
    • Information
    Sean Hsieh 謝任軒
    Linkedin
    Medium

    View Slide

  3. 3
    Introduction to
    the dark traffic
    A traffic listener
    and player:
    GoReplay
    Design a dark
    traffic system
    Design A Dark Traffic
    System With GoReplay

    View Slide

  4. Design A Dark Traffic
    System With GoReplay
    4
    Introduction to
    the dark traffic
    A traffic listener
    and player:
    GoReplay
    Design a dark
    traffic system

    View Slide

  5. Testing before release
    5
    • System behave differently depends on environment and traffic pattern
    Production
    Staging
    • Integration tests
    • Load tests
    • Smoke test
    Pass & Release
    V 1.1
    SQA
    Real users
    V 1.2
    Rollback
    V 1.1
    V 1.2

    View Slide

  6. Dark traffic (so called shadowing or mirroring)
    6
    • Sampling real traffic is the only way to reliably capture the request path
    Staging
    V 1.2
    V 1.1
    Sampling
    HTTP/HTTPS
    Replaying the requests
    to prod and staging
    Real users
    Production
    Compare the response,
    record and verify
    Pass & Release
    V 1.2

    View Slide

  7. The solutions of dark traffic
    7
    • The best solution depends on the project
    Router-based Package-based
    • Facebook’s McRouter:
    • Sample entire prod traffics
    • GoReplay :
    • Sniff the HTTP packages and replay

    View Slide

  8. For non-Idempotent or stateful web service?
    8
    • Or you need to keep sync the database
    Staging
    V 1.2
    V 1.1
    Sampling
    HTTP/HTTPS
    Replaying the requests
    to prod and staging
    Real users
    Production
    Compare the response,
    record and verify
    Log out
    Failed!

    View Slide

  9. Summary
    • Choose the proper solution for your
    team and products
    9
    You need to consider that…
    • Shadowing may be suitable for:
    1. Idempotent services (冪等)
    2. Stateless services (無狀態)
    3. Or you have to keep in-sync the
    data from the prod database to
    the staging database

    View Slide

  10. Design A Dark Traffic
    System With GoReplay
    10
    Introduction to
    the dark traffic
    A traffic listener
    and player:
    GoReplay
    Design a dark
    traffic system

    View Slide

  11. GoReplay is a tool to test the system with real
    traffics
    11
    • It’s a Golang project which is based on WinPcap (Windows)
    • Ability to capture and replay(or accelerate) the traffics
    • Ability to plugin the middleware

    View Slide

  12. GoReplay usage scenario
    12
    Real users
    HTTP Gor replay
    //Replay traffics to the B site from files
    gor --input-file data/request_0.gor –output-http=http://staging -output-http="http://production"
    Gor listen
    Staging service
    //Capture traffics from the specific port and output the traffic files
    gor --input-raw :3000 -output-file= "data/request.gor"
    Prod service
    Listen port 3000
    //Replay traffics to the B site in the real time
    gor –input-raw:3000 –output-http=http://staging -output-http="http://production"
    Traffic files

    View Slide

  13. Summary
    • GoReplay can’t support HTTPS traffics
    • According to the author:
    13
    You need to consider that…
    What???

    View Slide

  14. Design A Dark Traffic
    System With GoReplay
    14
    Introduction to
    the dark traffic
    A traffic listener
    and player:
    GoReplay
    Design a dark
    traffic system

    View Slide

  15. Architecture of the dark traffic system
    15
    Staging
    V 1.2
    V 1.1
    Sample
    HTTP/HTTPS
    Compare the response
    Record and verify
    Real users
    Production
    Replay the requests to prod and
    staging
    Build a control panel
    SQA

    View Slide

  16. 1. How GoRelpay
    samples HTTPS ?
    16
    //Create a file server
    gor --file-server :8800
    //Capture the packages
    gor --input-raw :8800 -output-file=data/request.gor
    • Duplicate HTTPS packages + SSL termination
    + Gor file server
    Real users Nginx
    Gor file server
    Prod service
    Gor listen
    HTTPS
    Traffic files
    HTTPS
    HTTP

    View Slide

  17. Nginx:
    Duplicate HTTPS packages + SSL termination
    17
    http {
    server {
    listen 443;
    server_name IP/domain name;
    ssl on;
    ssl_certificate C:/nginx-1.14.1/ssl/gor_proxy.crt;
    ssl_certificate_key C:/nginx-1.14.1/ssl/gor_proxy.key.org;
    #charset koi8-r;
    #access_log logs/host.access.log main;
    location / {
    proxy_pass https://a.com;
    post_action @afterdownload;
    }
    location @afterdownload {
    proxy_pass http://b.com$uri?$args;
    }
    }
    }

    View Slide

  18. 18
    2 How GoReplay
    compares responses?
    • Author suggest that using the middleware

    View Slide

  19. Node.js middleware framework
    19
    var gor = require("goreplay_middleware");
    var map = {};
    gor.init();
    gor.on("message", function(msg) {
    if ((msg.ID in map) === false) {
    map[msg.ID] = []
    }
    if (msg.type === '1') {
    map[msg.ID].push({
    'Method': gor.httpMethod(msg.http),
    'Status': gor.httpStatus(msg.http),
    'Body': gor.httpBody(msg.http).toString()
    })
    } else {
    map[msg.ID].push({
    'Body': gor.httpBody(msg.http).toString(),
    'Status': gor.httpStatus(msg.http)
    })
    }
    if (map[msg.ID].length == 3) {
    console.error(`Replay 1's body: ${map[msg.ID][1]['Body']}`)
    console.error(`Replay 2's body: '${map[msg.ID][2]['Body']}`)
    console.error(`Replay 1's status: ${map[msg.ID][1]['Status']}`)
    console.error(`Replay 2's status: ${map[msg.ID][2]['Status']}`)
    }
    return msg;
    })
    //Replay traffics to the B site from files
    gor --input-file data/request_0.gor –output-http=http://localhost:8800
    -output-http=http://B --output-http-track-response --input-raw-track-
    response --middleware "node ~\app.js"
    (2)
    (3)

    View Slide

  20. 20
    3. How GoReplay
    records compared
    results?
    • Middleware
    • ElasticSearch
    • Kibana

    View Slide

  21. Node.js middleware framework
    21
    var elasticsearch = require('elasticsearch');
    const client = new elasticsearch.Client({
    hosts: [ 'http://localhost:9200']
    });
    client.ping({
    requestTimeout: 30000,
    }, function(error) {
    if (error) {
    console.error('Elasticsearch cluster is down!');
    } else {
    console.error('Everything is ok');
    }
    });
    client.indices.create({
    index: 'gor'
    }, function(error, response, status) {
    if (error) {
    console.error(error);
    } else {
    console.error("created a new index", response);
    }
    });
    //Replay traffics to the B site from files
    gor --input-file data/request_0.gor –output-http=http://localhost:8800
    -output-http=http://B --output-http-track-response --input-raw-track-
    response --middleware "node ~\app.js"

    View Slide

  22. 22
    Thanks!
    Any questions?
    You can find me at:
    Linkedin
    Medium

    View Slide