Building Scalable Infrastructure at Mercari

Transcript

1. Building scalable infrastructure
   with nginx and Go @
   Tatsuhiko Kubo@cubicdaiya
   dots. 2016/06/23
   

   View Slide

2. @cubicdaiya / Tatsuhiko Kubo
   Principal Engineer, SRE @ Mercari, Inc.
   ngx_small_light, ngx_dynamic_upstream,
   nginx-build, slackboard, cachectl, gaurun, widebullet, etc…
   

   View Slide

3. View Slide

4. Architecture
   nginx nginx nginx
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   IAM Add-on Example:
   IAM Add-on
   ence
   )
   Assignment/
   Task
   Requester
   Workers
   DNS-RR
   App App App
   App App App
   MySQL MySQL
   memcached
   memcached
   JP US
   nginx nginx nginx
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia C
   d
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   DNS-RR
   App App App
   App App App
   MySQL MySQL
   memcached
   memcached
   EC2
   EC2 EC2 EC2
   EC2
   EC2
   EC2
   EC2
   EC2
   EC2
   EC2
   RDS
   SPDY/HTTP2
   SPDY/HTTP2
   EC2
   

   View Slide

5. Scale @
   • API requests (HTTPS)
   • 1,300,000 req / min
   • CDN Traffic
   • 10Gbps
   • at peak…
   

   View Slide

6. Architecture
   nginx nginx nginx
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   IAM Add-on Example:
   IAM Add-on
   ence
   )
   Assignment/
   Task
   Requester
   Workers
   DNS-RR
   App App App
   App App App
   MySQL MySQL
   memcached
   memcached
   JP US
   nginx nginx nginx
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia C
   d
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   DNS-RR
   App App App
   App App App
   MySQL MySQL
   memcached
   memcached
   EC2
   EC2 EC2 EC2
   EC2
   EC2
   EC2
   EC2
   EC2
   EC2
   EC2
   RDS
   SPDY/HTTP2
   SPDY/HTTP2
   EC2
   

   View Slide

7. nginx
   • HTTP server for general purpose
   • event-driven, non-blocking architecture
   • 100+ nginx works @
   • HTTP/2, SPDY Termination
   • Reverse Proxy, Load Balancer, …
   

   View Slide

8. Why we use nginx?
   • Scalable
   • serve over 10K connections always per 1 instance
   • Flexible
   • TLS Termination, Reverse Proxy, Load Balancing(L4 / L7),
   Content Caching, and so on…
   • Extensible
   • developer can extend nginx with C, Lua, JavaScript, mruby, …
   • selects C and Lua
   

   View Slide

9. Programming Language @
   • API, Web
   • PHP
   • Backend
   • Go, Lua
   • sometimes PHP
   

   View Slide

10. Programming Language @
    • API, Web
    • PHP
    • Backend
    • Go, Lua
    • sometimes PHP
    

    View Slide

11. Why we use Go?
    • Simple
    • build and deployment
    • Enough
    • scalability, robustness, productivity
    • Powerful
    • concurrency support with goroutine and channel
    

    View Slide

12. mercari.go
    • nginx-build - seamless nginx builder
    (github.com/cubicdaiya/nginx-build)
    • Gaurun - push notification server
    (github.com/mercari/gaurun)
    • Widebullet - API gateway
    (github.com/mercari/widebullet)
    • etc… ( e.g. CLI in-house operation tools)
    

    View Slide

13. Building scalable infrastructure with nginx and Go
    • has been building scalable infrastructure with nginx and Go
    • Gaurun - Push infrastructure
    • Pascal - Log analysis infrastructure
    • Widebullet - API gateway
    • etc… (Some infrastructure is powered by gRPC)
    

    View Slide

14. Gaurun - Push infrastructure
    ($.
    "1/T
    (BVSVO
    (BVSVO
    (BVSVO
    /(*/9
    "1*4FSWFS
    "1*4FSWFS
    "1*4FSWFS
    #BUDI4FSWFS
    QVTI
    XJUI
    LFFQBMJWF
    )551
    1045QVTI
    

    View Slide

15. Gaurun - Push infrastructure
    • The events in are notified to users.
    • Purchase, Like, Comment, Shipment, …
    • Push notification to all users at once?
    • Over a few tens of millions pushʂ
    • High performance and scalability are
    required for pushing quickly
    

    View Slide

16. Gaurun
    • Push notification server in Go
    • github.com/mercari/gaurun
    • Provides HTTP APIs for push and metrics
    • Supports GCM and APNs ( HTTP/2, too)
    • Scalable
    • In , a few Gaurun instances process all pushes.
    

    View Slide

17. 0QFO3FTUZ
    0QFO3FTUZ
    0QFO3FTUZ
    (PPHMF#JH2VFSZ
    Developer Data Scientist
    Analyze
    by SQL
    send events
    send events
    send events
    Powered by
    cookpad/puree-(ios|android)
    utilize events
    utilize events
    utilize events
    hydra(※)
    hydra(※)
    hydra(※)
    (※) fluent-agent-hydra
    Pascal - Log analysis infrastructure
    in_tail & out_forward
    

    View Slide

18. PascalʙLog analysis infrastructureʙ
    • Built with the software blocks below
    • Puree, OpenResty (ngx_lua), Fluentd,
    fluent-agent-hydra, Google BigQuery
    • Aggregate various logs to Google BigQuery
    • Event in app
    • A/B Testing
    • etc…
    

    View Slide

19. • Low resource usage is required
    • Pascal indicates modestly high workload
    • OpenResty processes a lot of JSONs and
    outputs various logs
    • Requests are come from not only device but
    internal subsystems (e.g. API).
    • OpenResty and fluent-agent-hydra are very fast
    and efficient
    PascalʙLog analysis infrastructureʙ
    

    View Slide

20. Widebullet - API gateway
    • In , subsystems have been increased day by day.
    • some subsystem must call multiple APIs at once
    • Call APIs concurrently for high performance
    • Widebullet proxies multiple API calls at once
    

    View Slide

21. Widebullet
    • API gateway in Go
    • github.com/mercari/widebullet
    • Protocol
    • based JSON-RPC ( a little extended)
    

    View Slide

22. Widebullet - JSON-RPC gateway
    

    View Slide

23. View Slide

24. Widebullet - API gateway
    

    View Slide

25. Summary
    • has been building scalable infrastructure with nginx and Go
    • Why we use nginx and Go?
    • nginx
    • Scalable, Flexible, Extensible
    • Go
    • Simple, Enough, Powerful
    • These infrastructures are constructed by SREs and
    Backend System Engineers
    

    View Slide

26. We are hiring!
    • SRE
    • https://www.mercari.com/jp/jobs/sre/
    • Backend System engineer
    • https://www.mercari.com/jp/jobs/backend/
    

    View Slide

27. Summer Internship 2016
    https://www.wantedly.com/projects/58138
    

    View Slide