Slide 1

Slide 1 text

Tatsuhiko Kubo@cubicdaiya Go 1.10 Release Party 2018/02/20 High Performance Count Up!

Slide 2

Slide 2 text

@cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Agenda • Building High Performance count-up server with Go • Backend for counting page view of item at Mercari • ※ The right image is dev version’s

Slide 5

Slide 5 text

Agenda • Building High Performance count-up server with Go • Backend for counting page view of item at Mercari • ※ The right image is dev version’s

Slide 6

Slide 6 text

Difficulty in counting up page view of item in large web service • Large web service handles too many requests • Large web service has too many items • In general, page view of item overrepresents in all requests • Usually, page view is only read-only processing except logging • If it changes to write processing? • Write's scaling is more difficult than read's • Asynchronous processing is essential

Slide 7

Slide 7 text

Case at Mercari • Many mercari code bases are still built by PHP • But, PHP is not good at asynchronous processing • We developed the service by Go to count up page view of item • The name is pvpool

Slide 8

Slide 8 text

pvpool provides 2 HTTP APIs • POST /items/pvc • Count up page view of given item IDs asynchronously • POST /items/pv • Return page view of given item IDs Payload is JSON

Slide 9

Slide 9 text

System architecture of pvpool • nginx • pvpoold • Powered by Go (net/http) • MySQL pvpoold pvpoold nginx MySQL pvpool.local by consul DNS : HTTP protocol : MySQL protocol The gopher was designed by Renée French

Slide 10

Slide 10 text

Requirements and Load Characteristics of pvpool • Requirements • Reflect latest page view as real-time as possible (e.g. within a few seconds) • Respond as low latency as possible (e.g. within a few milli-seconds) • Load Characteristics • High throughput (Equal to page view of all items plus alpha) • Write-heavy

Slide 11

Slide 11 text

pvpoold Internals POST /items/pvc { “item_ids": [ ... ] } PVCHandler Sharding Item IDs MySQL Slot Slot Slot Slot Slot Slot Pooling query Flushing periodically synchronous processing asynchronous processing

Slide 12

Slide 12 text

pvpoold Internals • PVCHandler • HTTP Handler function • Slot • Processing unit to count up page view of items • Represented by goroutine

Slide 13

Slide 13 text

pvpoold Internals • Characters in Slot • Slot -> chan types.ItemID • PVMap -> map[types.ItemID]int • Storer -> Fetch page view of item in Slot and store to PVMap • Flusher -> Fetch page view of item in PVMap and Issue SQL to MySQL periodically • Storer & Flusher are also represented by goroutine

Slide 14

Slide 14 text

Slot overview Slot Storer Flusher PVMap dequeue item ID Store pv per item Fetch pv per item periodically MySQL Issue SQL to count up

Slide 15

Slide 15 text

Storer behavior image func (slot *PVCSlot) storer() { for { itemID := <- slot.Slot slot.Lock() slot.PVMap[itemID]++ slot.Unock() } }

Slide 16

Slide 16 text

Flusher behavior image func (slot *PVCSlot) flusher(interval time.Duration) { ticker := time.NewTicker(interval) for { <- ticker.C slot.Lock() if err := slot.flush(); err != nil { // transaction // error handling } slot.Unlock() } }

Slide 17

Slide 17 text

Role of Storer & Flusher • Storer • Reduce issued SQLs to MySQL by aggregating page view per item • eg. UPDATE pv = pv + 5 instead of five of UPDATE pv = pv +1 • Flusher • Issue SQLs to MySQL periodically • Mitigate load of MySQL by timer control

Slide 18

Slide 18 text

Avoid dead lock between Slots • pvpoold do not store some item ID to multiple Slots • Slot in which some item ID is stored is uniquely identified by its value • Because Go's map iteration order is random • So flushed item ID order in each Slot is not fixed

Slide 19

Slide 19 text

Appendix • Fast logging • Powered by uber-go/zap • Tuning sql.DB • SetMaxOpenConns • SetMaxIdleConns • SetConnMaxLifetime • Great article by @methane • http://dsas.blog.klab.org/archives/2018-02/configure-sql-db.html

Slide 20

Slide 20 text

We are hiring Go Software Engineer! https://careers.mercari.com/job/