Building Scalable Infrastructure at Mercari

Slide 1

Slide 1 text

Building scalable infrastructure with nginx and Go @ Tatsuhiko Kubo@cubicdaiya dots. 2016/06/23

Slide 2

Slide 2 text

@cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc. ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, widebullet, etc…

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 RDS SPDY/HTTP2 SPDY/HTTP2 EC2

Slide 5

Slide 5 text

Scale @ • API requests (HTTPS) • 1,300,000 req / min • CDN Trafﬁc • 10Gbps • at peak…

Slide 6

Slide 6 text

Slide 7

Slide 7 text

nginx • HTTP server for general purpose • event-driven, non-blocking architecture • 100+ nginx works @ • HTTP/2, SPDY Termination • Reverse Proxy, Load Balancer, …

Slide 8

Slide 8 text

Why we use nginx? • Scalable • serve over 10K connections always per 1 instance • Flexible • TLS Termination, Reverse Proxy, Load Balancing(L4 / L7), Content Caching, and so on… • Extensible • developer can extend nginx with C, Lua, JavaScript, mruby, … • selects C and Lua

Slide 9

Slide 9 text

Programming Language @ • API, Web • PHP • Backend • Go, Lua • sometimes PHP

Slide 10

Slide 10 text

Programming Language @ • API, Web • PHP • Backend • Go, Lua • sometimes PHP

Slide 11

Slide 11 text

Why we use Go? • Simple • build and deployment • Enough • scalability, robustness, productivity • Powerful • concurrency support with goroutine and channel

Slide 12

Slide 12 text

mercari.go • nginx-build - seamless nginx builder (github.com/cubicdaiya/nginx-build) • Gaurun - push notiﬁcation server (github.com/mercari/gaurun) • Widebullet - API gateway (github.com/mercari/widebullet) • etc… ( e.g. CLI in-house operation tools)

Slide 13

Slide 13 text

Building scalable infrastructure with nginx and Go • has been building scalable infrastructure with nginx and Go • Gaurun - Push infrastructure • Pascal - Log analysis infrastructure • Widebullet - API gateway • etc… (Some infrastructure is powered by gRPC)

Slide 14

Slide 14 text

Gaurun - Push infrastructure ($. "1/T (BVSVO (BVSVO (BVSVO /(*/9 "1*4FSWFS "1*4FSWFS "1*4FSWFS #BUDI4FSWFS QVTI XJUI LFFQBMJWF )551 1045QVTI

Slide 15

Slide 15 text

Gaurun - Push infrastructure • The events in are notiﬁed to users. • Purchase, Like, Comment, Shipment, … • Push notiﬁcation to all users at once? • Over a few tens of millions pushʂ • High performance and scalability are required for pushing quickly

Slide 16

Slide 16 text

Gaurun • Push notiﬁcation server in Go • github.com/mercari/gaurun • Provides HTTP APIs for push and metrics • Supports GCM and APNs ( HTTP/2, too) • Scalable • In , a few Gaurun instances process all pushes.

Slide 17

Slide 17 text

0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF#JH2VFSZ Developer Data Scientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events hydra(※) hydra(※) hydra(※) (※) ﬂuent-agent-hydra Pascal - Log analysis infrastructure in_tail & out_forward

Slide 18

Slide 18 text

PascalʙLog analysis infrastructureʙ • Built with the software blocks below • Puree, OpenResty (ngx_lua), Fluentd, ﬂuent-agent-hydra, Google BigQuery • Aggregate various logs to Google BigQuery • Event in app • A/B Testing • etc…

Slide 19

Slide 19 text

• Low resource usage is required • Pascal indicates modestly high workload • OpenResty processes a lot of JSONs and outputs various logs • Requests are come from not only device but internal subsystems (e.g. API). • OpenResty and ﬂuent-agent-hydra are very fast and efﬁcient PascalʙLog analysis infrastructureʙ

Slide 20

Slide 20 text

Widebullet - API gateway • In , subsystems have been increased day by day. • some subsystem must call multiple APIs at once • Call APIs concurrently for high performance • Widebullet proxies multiple API calls at once

Slide 21

Slide 21 text

Widebullet • API gateway in Go • github.com/mercari/widebullet • Protocol • based JSON-RPC ( a little extended)

Slide 22

Slide 22 text

Widebullet - JSON-RPC gateway

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Widebullet - API gateway

Slide 25

Slide 25 text

Summary • has been building scalable infrastructure with nginx and Go • Why we use nginx and Go? • nginx • Scalable, Flexible, Extensible • Go • Simple, Enough, Powerful • These infrastructures are constructed by SREs and Backend System Engineers

Slide 26

Slide 26 text

We are hiring! • SRE • https://www.mercari.com/jp/jobs/sre/ • Backend System engineer • https://www.mercari.com/jp/jobs/backend/

Slide 27

Slide 27 text

Summer Internship 2016 https://www.wantedly.com/projects/58138