Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Scalable Infrastructure at Mercari

Building Scalable Infrastructure at Mercari

Tatsuhiko Kubo

June 24, 2016
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Building scalable infrastructure
    with nginx and Go @
    Tatsuhiko Kubo@cubicdaiya
    dots. 2016/06/23

    View full-size slide

  2. @cubicdaiya / Tatsuhiko Kubo
    Principal Engineer, SRE @ Mercari, Inc.
    ngx_small_light, ngx_dynamic_upstream,
    nginx-build, slackboard, cachectl, gaurun, widebullet, etc…

    View full-size slide

  3. Architecture
    nginx nginx nginx
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    Client Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    IAM Add-on Example:
    IAM Add-on
    ence
    )
    Assignment/
    Task
    Requester
    Workers
    DNS-RR
    App App App
    App App App
    MySQL MySQL
    memcached
    memcached
    JP US
    nginx nginx nginx
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    User Users Client Multimedia C
    d
    Mobile Client
    Internet AWS Management
    Console
    IAM Add-on Example:
    IAM Add-on
    Human Intelligence
    Tasks (HIT)
    Assignment/
    Task
    Requester
    Workers
    Amazon
    Mechanical Turk
    Non-Service Specific
    DNS-RR
    App App App
    App App App
    MySQL MySQL
    memcached
    memcached
    EC2
    EC2 EC2 EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    RDS
    SPDY/HTTP2
    SPDY/HTTP2
    EC2

    View full-size slide

  4. Scale @
    • API requests (HTTPS)
    • 1,300,000 req / min
    • CDN Traffic
    • 10Gbps
    • at peak…

    View full-size slide

  5. Architecture
    nginx nginx nginx
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    Client Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    IAM Add-on Example:
    IAM Add-on
    ence
    )
    Assignment/
    Task
    Requester
    Workers
    DNS-RR
    App App App
    App App App
    MySQL MySQL
    memcached
    memcached
    JP US
    nginx nginx nginx
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    User Users Client Multimedia C
    d
    Mobile Client
    Internet AWS Management
    Console
    IAM Add-on Example:
    IAM Add-on
    Human Intelligence
    Tasks (HIT)
    Assignment/
    Task
    Requester
    Workers
    Amazon
    Mechanical Turk
    Non-Service Specific
    DNS-RR
    App App App
    App App App
    MySQL MySQL
    memcached
    memcached
    EC2
    EC2 EC2 EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    RDS
    SPDY/HTTP2
    SPDY/HTTP2
    EC2

    View full-size slide

  6. nginx
    • HTTP server for general purpose
    • event-driven, non-blocking architecture
    • 100+ nginx works @
    • HTTP/2, SPDY Termination
    • Reverse Proxy, Load Balancer, …

    View full-size slide

  7. Why we use nginx?
    • Scalable
    • serve over 10K connections always per 1 instance
    • Flexible
    • TLS Termination, Reverse Proxy, Load Balancing(L4 / L7),
    Content Caching, and so on…
    • Extensible
    • developer can extend nginx with C, Lua, JavaScript, mruby, …
    • selects C and Lua

    View full-size slide

  8. Programming Language @
    • API, Web
    • PHP
    • Backend
    • Go, Lua
    • sometimes PHP

    View full-size slide

  9. Programming Language @
    • API, Web
    • PHP
    • Backend
    • Go, Lua
    • sometimes PHP

    View full-size slide

  10. Why we use Go?
    • Simple
    • build and deployment
    • Enough
    • scalability, robustness, productivity
    • Powerful
    • concurrency support with goroutine and channel

    View full-size slide

  11. mercari.go
    • nginx-build - seamless nginx builder
    (github.com/cubicdaiya/nginx-build)
    • Gaurun - push notification server
    (github.com/mercari/gaurun)
    • Widebullet - API gateway
    (github.com/mercari/widebullet)
    • etc… ( e.g. CLI in-house operation tools)

    View full-size slide

  12. Building scalable infrastructure with nginx and Go
    • has been building scalable infrastructure with nginx and Go
    • Gaurun - Push infrastructure
    • Pascal - Log analysis infrastructure
    • Widebullet - API gateway
    • etc… (Some infrastructure is powered by gRPC)

    View full-size slide

  13. Gaurun - Push infrastructure
    ($.
    "1/T
    (BVSVO
    (BVSVO
    (BVSVO
    /(*/9
    "1*4FSWFS
    "1*4FSWFS
    "1*4FSWFS
    #BUDI4FSWFS
    QVTI
    XJUI
    LFFQBMJWF
    )551
    1045QVTI

    View full-size slide

  14. Gaurun - Push infrastructure
    • The events in are notified to users.
    • Purchase, Like, Comment, Shipment, …
    • Push notification to all users at once?
    • Over a few tens of millions pushʂ
    • High performance and scalability are
    required for pushing quickly

    View full-size slide

  15. Gaurun
    • Push notification server in Go
    • github.com/mercari/gaurun
    • Provides HTTP APIs for push and metrics
    • Supports GCM and APNs ( HTTP/2, too)
    • Scalable
    • In , a few Gaurun instances process all pushes.

    View full-size slide

  16. 0QFO3FTUZ
    0QFO3FTUZ
    0QFO3FTUZ
    (PPHMF#JH2VFSZ
    Developer Data Scientist
    Analyze
    by SQL
    send events
    send events
    send events
    Powered by
    cookpad/puree-(ios|android)
    utilize events
    utilize events
    utilize events
    hydra(※)
    hydra(※)
    hydra(※)
    (※) fluent-agent-hydra
    Pascal - Log analysis infrastructure
    in_tail & out_forward

    View full-size slide

  17. PascalʙLog analysis infrastructureʙ
    • Built with the software blocks below
    • Puree, OpenResty (ngx_lua), Fluentd,
    fluent-agent-hydra, Google BigQuery
    • Aggregate various logs to Google BigQuery
    • Event in app
    • A/B Testing
    • etc…

    View full-size slide

  18. • Low resource usage is required
    • Pascal indicates modestly high workload
    • OpenResty processes a lot of JSONs and
    outputs various logs
    • Requests are come from not only device but
    internal subsystems (e.g. API).
    • OpenResty and fluent-agent-hydra are very fast
    and efficient
    PascalʙLog analysis infrastructureʙ

    View full-size slide

  19. Widebullet - API gateway
    • In , subsystems have been increased day by day.
    • some subsystem must call multiple APIs at once
    • Call APIs concurrently for high performance
    • Widebullet proxies multiple API calls at once

    View full-size slide

  20. Widebullet
    • API gateway in Go
    • github.com/mercari/widebullet
    • Protocol
    • based JSON-RPC ( a little extended)

    View full-size slide

  21. Widebullet - JSON-RPC gateway

    View full-size slide

  22. Widebullet - API gateway

    View full-size slide

  23. Summary
    • has been building scalable infrastructure with nginx and Go
    • Why we use nginx and Go?
    • nginx
    • Scalable, Flexible, Extensible
    • Go
    • Simple, Enough, Powerful
    • These infrastructures are constructed by SREs and
    Backend System Engineers

    View full-size slide

  24. We are hiring!
    • SRE
    • https://www.mercari.com/jp/jobs/sre/
    • Backend System engineer
    • https://www.mercari.com/jp/jobs/backend/

    View full-size slide

  25. Summer Internship 2016
    https://www.wantedly.com/projects/58138

    View full-size slide