Building Scalable Infrastructure at Mercari

Building Scalable Infrastructure at Mercari

5d74d743eabd2bf7d4d2f68b9d3c727d?s=128

Tatsuhiko Kubo

June 24, 2016
Tweet

Transcript

  1. Building scalable infrastructure with nginx and Go @ Tatsuhiko Kubo@cubicdaiya

    dots. 2016/06/23
  2. @cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.

    ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, widebullet, etc…
  3. None
  4. Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or

    its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 RDS SPDY/HTTP2 SPDY/HTTP2 EC2
  5. Scale @ • API requests (HTTPS) • 1,300,000 req /

    min • CDN Traffic • 10Gbps • at peak…
  6. Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or

    its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 RDS SPDY/HTTP2 SPDY/HTTP2 EC2
  7. nginx • HTTP server for general purpose • event-driven, non-blocking

    architecture • 100+ nginx works @ • HTTP/2, SPDY Termination • Reverse Proxy, Load Balancer, …
  8. Why we use nginx? • Scalable • serve over 10K

    connections always per 1 instance • Flexible • TLS Termination, Reverse Proxy, Load Balancing(L4 / L7), Content Caching, and so on… • Extensible • developer can extend nginx with C, Lua, JavaScript, mruby, … • selects C and Lua
  9. Programming Language @ • API, Web • PHP • Backend

    • Go, Lua • sometimes PHP
  10. Programming Language @ • API, Web • PHP • Backend

    • Go, Lua • sometimes PHP
  11. Why we use Go? • Simple • build and deployment

    • Enough • scalability, robustness, productivity • Powerful • concurrency support with goroutine and channel
  12. mercari.go • nginx-build - seamless nginx builder (github.com/cubicdaiya/nginx-build) • Gaurun

    - push notification server (github.com/mercari/gaurun) • Widebullet - API gateway (github.com/mercari/widebullet) • etc… ( e.g. CLI in-house operation tools)
  13. Building scalable infrastructure with nginx and Go • has been

    building scalable infrastructure with nginx and Go • Gaurun - Push infrastructure • Pascal - Log analysis infrastructure • Widebullet - API gateway • etc… (Some infrastructure is powered by gRPC)
  14. Gaurun - Push infrastructure ($. "1/T (BVSVO (BVSVO (BVSVO /(*/9

    "1*4FSWFS "1*4FSWFS "1*4FSWFS #BUDI4FSWFS QVTI XJUI LFFQBMJWF )551 1045QVTI
  15. Gaurun - Push infrastructure • The events in are notified

    to users. • Purchase, Like, Comment, Shipment, … • Push notification to all users at once? • Over a few tens of millions pushʂ • High performance and scalability are required for pushing quickly
  16. Gaurun • Push notification server in Go • github.com/mercari/gaurun •

    Provides HTTP APIs for push and metrics • Supports GCM and APNs ( HTTP/2, too) • Scalable • In , a few Gaurun instances process all pushes.
  17. 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF#JH2VFSZ Developer Data Scientist Analyze by SQL

    send events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events hydra(※) hydra(※) hydra(※) (※) fluent-agent-hydra Pascal - Log analysis infrastructure in_tail & out_forward
  18. PascalʙLog analysis infrastructureʙ • Built with the software blocks below

    • Puree, OpenResty (ngx_lua), Fluentd, fluent-agent-hydra, Google BigQuery • Aggregate various logs to Google BigQuery • Event in app • A/B Testing • etc…
  19. • Low resource usage is required • Pascal indicates modestly

    high workload • OpenResty processes a lot of JSONs and outputs various logs • Requests are come from not only device but internal subsystems (e.g. API). • OpenResty and fluent-agent-hydra are very fast and efficient PascalʙLog analysis infrastructureʙ
  20. Widebullet - API gateway • In , subsystems have been

    increased day by day. • some subsystem must call multiple APIs at once • Call APIs concurrently for high performance • Widebullet proxies multiple API calls at once
  21. Widebullet • API gateway in Go • github.com/mercari/widebullet • Protocol

    • based JSON-RPC ( a little extended)
  22. Widebullet - JSON-RPC gateway

  23. None
  24. Widebullet - API gateway

  25. Summary • has been building scalable infrastructure with nginx and

    Go • Why we use nginx and Go? • nginx • Scalable, Flexible, Extensible • Go • Simple, Enough, Powerful • These infrastructures are constructed by SREs and Backend System Engineers
  26. We are hiring! • SRE • https://www.mercari.com/jp/jobs/sre/ • Backend System

    engineer • https://www.mercari.com/jp/jobs/backend/
  27. Summer Internship 2016 https://www.wantedly.com/projects/58138