Building Scalable Infrastructure at Mercari

Transcript

1. Building scalable infrastructure with nginx and Go @ Tatsuhiko Kubo@cubicdaiya dots. 2016/06/23
2. @cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc. ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, widebullet, etc…
4. Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 RDS SPDY/HTTP2 SPDY/HTTP2 EC2
5. Scale @ • API requests (HTTPS) • 1,300,000 req / min • CDN Traffic • 10Gbps • at peak…
6. Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 RDS SPDY/HTTP2 SPDY/HTTP2 EC2
7. nginx • HTTP server for general purpose • event-driven, non-blocking architecture • 100+ nginx works @ • HTTP/2, SPDY Termination • Reverse Proxy, Load Balancer, …
8. Why we use nginx? • Scalable • serve over 10K connections always per 1 instance • Flexible • TLS Termination, Reverse Proxy, Load Balancing(L4 / L7), Content Caching, and so on… • Extensible • developer can extend nginx with C, Lua, JavaScript, mruby, … • selects C and Lua
9. Programming Language @ • API, Web • PHP • Backend • Go, Lua • sometimes PHP
10. Programming Language @ • API, Web • PHP • Backend • Go, Lua • sometimes PHP
11. Why we use Go? • Simple • build and deployment • Enough • scalability, robustness, productivity • Powerful • concurrency support with goroutine and channel
12. mercari.go • nginx-build - seamless nginx builder (github.com/cubicdaiya/nginx-build) • Gaurun - push notification server (github.com/mercari/gaurun) • Widebullet - API gateway (github.com/mercari/widebullet) • etc… ( e.g. CLI in-house operation tools)
13. Building scalable infrastructure with nginx and Go • has been building scalable infrastructure with nginx and Go • Gaurun - Push infrastructure • Pascal - Log analysis infrastructure • Widebullet - API gateway • etc… (Some infrastructure is powered by gRPC)
14. Gaurun - Push infrastructure ($. "1/T (BVSVO (BVSVO (BVSVO /(*/9 "1*
    4FSWFS
    "1*
    4FSWFS
    "1*
    4FSWFS
    #BUDI
    4FSWFS
    QVTI
    XJUI
    LFFQBMJWF )551
    1045
    QVTI
15. Gaurun - Push infrastructure • The events in are notified to users. • Purchase, Like, Comment, Shipment, … • Push notification to all users at once? • Over a few tens of millions pushʂ • High performance and scalability are required for pushing quickly
16. Gaurun • Push notification server in Go • github.com/mercari/gaurun • Provides HTTP APIs for push and metrics • Supports GCM and APNs ( HTTP/2, too) • Scalable • In , a few Gaurun instances process all pushes.
17. 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF
    #JH
    2VFSZ Developer Data Scientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events hydra(※) hydra(※) hydra(※) (※) fluent-agent-hydra Pascal - Log analysis infrastructure in_tail & out_forward
18. PascalʙLog analysis infrastructureʙ • Built with the software blocks below • Puree, OpenResty (ngx_lua), Fluentd, fluent-agent-hydra, Google BigQuery • Aggregate various logs to Google BigQuery • Event in app • A/B Testing • etc…
19. • Low resource usage is required • Pascal indicates modestly high workload • OpenResty processes a lot of JSONs and outputs various logs • Requests are come from not only device but internal subsystems (e.g. API). • OpenResty and fluent-agent-hydra are very fast and efficient PascalʙLog analysis infrastructureʙ
20. Widebullet - API gateway • In , subsystems have been increased day by day. • some subsystem must call multiple APIs at once • Call APIs concurrently for high performance • Widebullet proxies multiple API calls at once
21. Widebullet • API gateway in Go • github.com/mercari/widebullet • Protocol • based JSON-RPC ( a little extended)
22. Widebullet - JSON-RPC gateway
24. Widebullet - API gateway
25. Summary • has been building scalable infrastructure with nginx and Go • Why we use nginx and Go? • nginx • Scalable, Flexible, Extensible • Go • Simple, Enough, Powerful • These infrastructures are constructed by SREs and Backend System Engineers
26. We are hiring! • SRE • https://www.mercari.com/jp/jobs/sre/ • Backend System engineer • https://www.mercari.com/jp/jobs/backend/
27. Summer Internship 2016 https://www.wantedly.com/projects/58138