Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Scalable Infrastructure at Mercari

Building Scalable Infrastructure at Mercari

Tatsuhiko Kubo

June 24, 2016
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Building scalable infrastructure
    with nginx and Go @
    Tatsuhiko [email protected]
    dots. 2016/06/23

    View Slide

  2. @cubicdaiya / Tatsuhiko Kubo
    Principal Engineer, SRE @ Mercari, Inc.
    ngx_small_light, ngx_dynamic_upstream,
    nginx-build, slackboard, cachectl, gaurun, widebullet, etc…

    View Slide

  3. View Slide

  4. Architecture
    nginx nginx nginx
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    Client Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    IAM Add-on Example:
    IAM Add-on
    ence
    )
    Assignment/
    Task
    Requester
    Workers
    DNS-RR
    App App App
    App App App
    MySQL MySQL
    memcached
    memcached
    JP US
    nginx nginx nginx
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    User Users Client Multimedia C
    d
    Mobile Client
    Internet AWS Management
    Console
    IAM Add-on Example:
    IAM Add-on
    Human Intelligence
    Tasks (HIT)
    Assignment/
    Task
    Requester
    Workers
    Amazon
    Mechanical Turk
    Non-Service Specific
    DNS-RR
    App App App
    App App App
    MySQL MySQL
    memcached
    memcached
    EC2
    EC2 EC2 EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    RDS
    SPDY/HTTP2
    SPDY/HTTP2
    EC2

    View Slide

  5. Scale @
    • API requests (HTTPS)
    • 1,300,000 req / min
    • CDN Traffic
    • 10Gbps
    • at peak…

    View Slide

  6. Architecture
    nginx nginx nginx
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    Client Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    IAM Add-on Example:
    IAM Add-on
    ence
    )
    Assignment/
    Task
    Requester
    Workers
    DNS-RR
    App App App
    App App App
    MySQL MySQL
    memcached
    memcached
    JP US
    nginx nginx nginx
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    User Users Client Multimedia C
    d
    Mobile Client
    Internet AWS Management
    Console
    IAM Add-on Example:
    IAM Add-on
    Human Intelligence
    Tasks (HIT)
    Assignment/
    Task
    Requester
    Workers
    Amazon
    Mechanical Turk
    Non-Service Specific
    DNS-RR
    App App App
    App App App
    MySQL MySQL
    memcached
    memcached
    EC2
    EC2 EC2 EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    EC2
    RDS
    SPDY/HTTP2
    SPDY/HTTP2
    EC2

    View Slide

  7. nginx
    • HTTP server for general purpose
    • event-driven, non-blocking architecture
    • 100+ nginx works @
    • HTTP/2, SPDY Termination
    • Reverse Proxy, Load Balancer, …

    View Slide

  8. Why we use nginx?
    • Scalable
    • serve over 10K connections always per 1 instance
    • Flexible
    • TLS Termination, Reverse Proxy, Load Balancing(L4 / L7),
    Content Caching, and so on…
    • Extensible
    • developer can extend nginx with C, Lua, JavaScript, mruby, …
    • selects C and Lua

    View Slide

  9. Programming Language @
    • API, Web
    • PHP
    • Backend
    • Go, Lua
    • sometimes PHP

    View Slide

  10. Programming Language @
    • API, Web
    • PHP
    • Backend
    • Go, Lua
    • sometimes PHP

    View Slide

  11. Why we use Go?
    • Simple
    • build and deployment
    • Enough
    • scalability, robustness, productivity
    • Powerful
    • concurrency support with goroutine and channel

    View Slide

  12. mercari.go
    • nginx-build - seamless nginx builder
    (github.com/cubicdaiya/nginx-build)
    • Gaurun - push notification server
    (github.com/mercari/gaurun)
    • Widebullet - API gateway
    (github.com/mercari/widebullet)
    • etc… ( e.g. CLI in-house operation tools)

    View Slide

  13. Building scalable infrastructure with nginx and Go
    • has been building scalable infrastructure with nginx and Go
    • Gaurun - Push infrastructure
    • Pascal - Log analysis infrastructure
    • Widebullet - API gateway
    • etc… (Some infrastructure is powered by gRPC)

    View Slide

  14. Gaurun - Push infrastructure
    ($.
    "1/T
    (BVSVO
    (BVSVO
    (BVSVO
    /(*/9
    "1*4FSWFS
    "1*4FSWFS
    "1*4FSWFS
    #BUDI4FSWFS
    QVTI
    XJUI
    LFFQBMJWF
    )551
    1045QVTI

    View Slide

  15. Gaurun - Push infrastructure
    • The events in are notified to users.
    • Purchase, Like, Comment, Shipment, …
    • Push notification to all users at once?
    • Over a few tens of millions pushʂ
    • High performance and scalability are
    required for pushing quickly

    View Slide

  16. Gaurun
    • Push notification server in Go
    • github.com/mercari/gaurun
    • Provides HTTP APIs for push and metrics
    • Supports GCM and APNs ( HTTP/2, too)
    • Scalable
    • In , a few Gaurun instances process all pushes.

    View Slide

  17. 0QFO3FTUZ
    0QFO3FTUZ
    0QFO3FTUZ
    (PPHMF#JH2VFSZ
    Developer Data Scientist
    Analyze
    by SQL
    send events
    send events
    send events
    Powered by
    cookpad/puree-(ios|android)
    utilize events
    utilize events
    utilize events
    hydra(※)
    hydra(※)
    hydra(※)
    (※) fluent-agent-hydra
    Pascal - Log analysis infrastructure
    in_tail & out_forward

    View Slide

  18. PascalʙLog analysis infrastructureʙ
    • Built with the software blocks below
    • Puree, OpenResty (ngx_lua), Fluentd,
    fluent-agent-hydra, Google BigQuery
    • Aggregate various logs to Google BigQuery
    • Event in app
    • A/B Testing
    • etc…

    View Slide

  19. • Low resource usage is required
    • Pascal indicates modestly high workload
    • OpenResty processes a lot of JSONs and
    outputs various logs
    • Requests are come from not only device but
    internal subsystems (e.g. API).
    • OpenResty and fluent-agent-hydra are very fast
    and efficient
    PascalʙLog analysis infrastructureʙ

    View Slide

  20. Widebullet - API gateway
    • In , subsystems have been increased day by day.
    • some subsystem must call multiple APIs at once
    • Call APIs concurrently for high performance
    • Widebullet proxies multiple API calls at once

    View Slide

  21. Widebullet
    • API gateway in Go
    • github.com/mercari/widebullet
    • Protocol
    • based JSON-RPC ( a little extended)

    View Slide

  22. Widebullet - JSON-RPC gateway

    View Slide

  23. View Slide

  24. Widebullet - API gateway

    View Slide

  25. Summary
    • has been building scalable infrastructure with nginx and Go
    • Why we use nginx and Go?
    • nginx
    • Scalable, Flexible, Extensible
    • Go
    • Simple, Enough, Powerful
    • These infrastructures are constructed by SREs and
    Backend System Engineers

    View Slide

  26. We are hiring!
    • SRE
    • https://www.mercari.com/jp/jobs/sre/
    • Backend System engineer
    • https://www.mercari.com/jp/jobs/backend/

    View Slide

  27. Summer Internship 2016
    https://www.wantedly.com/projects/58138

    View Slide