High Performance Count Up!

Transcript

1. Tatsuhiko Kubo@cubicdaiya
   Go 1.10 Release Party 2018/02/20
   High Performance Count Up!
   

   View Slide

2. @cubicdaiya / Tatsuhiko Kubo
   Principal Engineer, SRE @ Mercari, Inc.
   

   View Slide

3. View Slide

4. Agenda
   • Building High Performance count-up server with Go
   
   • Backend for counting page view of item at Mercari
   
   • ※ The right image is dev version’s
   

   View Slide

5. Agenda
   • Building High Performance count-up server with Go
   
   • Backend for counting page view of item at Mercari
   
   • ※ The right image is dev version’s
   

   View Slide

6. Difficulty in counting up page view of item in large web
   service
   • Large web service handles too many requests
   
   • Large web service has too many items
   
   • In general, page view of item overrepresents in all requests
   
   • Usually, page view is only read-only processing except logging
   
   • If it changes to write processing?
   
   • Write's scaling is more difficult than read's
   
   • Asynchronous processing is essential
   

   View Slide

7. Case at Mercari
   • Many mercari code bases are still built by PHP
   
   • But, PHP is not good at asynchronous processing
   
   • We developed the service by Go to count up page view of item
   
   • The name is pvpool
   

   View Slide

8. pvpool provides 2 HTTP APIs
   • POST /items/pvc
   • Count up page view of given item IDs asynchronously
   
   • POST /items/pv
   • Return page view of given item IDs
   Payload is JSON
   

   View Slide

9. System architecture of pvpool
   • nginx
   
   • pvpoold
   
   • Powered by Go (net/http)
   
   • MySQL
   pvpoold
   pvpoold
   nginx MySQL
   pvpool.local
   by consul DNS
   : HTTP protocol
   : MySQL protocol
   The gopher was designed by Renée French
   

   View Slide

10. Requirements and Load Characteristics of pvpool
    • Requirements
    
    • Reflect latest page view as real-time as possible (e.g. within a few
    seconds)
    
    • Respond as low latency as possible (e.g. within a few milli-seconds)
    
    • Load Characteristics
    
    • High throughput (Equal to page view of all items plus alpha)
    
    • Write-heavy
    

    View Slide

11. pvpoold Internals
    POST /items/pvc
    {
    “item_ids": [
    ...
    ]
    }
    PVCHandler
    Sharding
    Item IDs
    MySQL
    Slot
    Slot
    Slot
    Slot
    Slot
    Slot
    Pooling query
    Flushing
    periodically
    synchronous processing
    asynchronous processing
    

    View Slide

12. pvpoold Internals
    • PVCHandler
    
    • HTTP Handler function
    
    • Slot
    
    • Processing unit to count up page view of items
    
    • Represented by goroutine
    

    View Slide

13. pvpoold Internals
    • Characters in Slot
    
    • Slot -> chan types.ItemID
    
    • PVMap -> map[types.ItemID]int
    • Storer -> Fetch page view of item in Slot and store to PVMap
    
    • Flusher -> Fetch page view of item in PVMap and Issue SQL to MySQL
    periodically
    
    • Storer & Flusher are also represented by goroutine
    

    View Slide

14. Slot overview
    Slot
    Storer Flusher
    PVMap
    dequeue item ID
    Store pv per item Fetch pv per item
    periodically
    MySQL
    Issue SQL to count up
    

    View Slide

15. Storer behavior image
    func (slot *PVCSlot) storer() {
    for {
    itemID := <- slot.Slot
    slot.Lock()
    slot.PVMap[itemID]++
    slot.Unock()
    }
    }
    

    View Slide

16. Flusher behavior image
    func (slot *PVCSlot) flusher(interval time.Duration) {
    ticker := time.NewTicker(interval)
    for {
    <- ticker.C
    slot.Lock()
    if err := slot.flush(); err != nil { // transaction
    // error handling
    }
    slot.Unlock()
    }
    }
    

    View Slide

17. Role of Storer & Flusher
    • Storer
    • Reduce issued SQLs to MySQL by aggregating page view per item
    
    • eg. UPDATE pv = pv + 5 instead of five of UPDATE pv = pv +1
    
    • Flusher
    • Issue SQLs to MySQL periodically
    
    • Mitigate load of MySQL by timer control
    

    View Slide

18. Avoid dead lock between Slots
    • pvpoold do not store some item ID to multiple Slots
    
    • Slot in which some item ID is stored is uniquely identified by its value
    
    • Because Go's map iteration order is random
    
    • So flushed item ID order in each Slot is not fixed
    

    View Slide

19. Appendix
    • Fast logging
    
    • Powered by uber-go/zap
    
    • Tuning sql.DB
    
    • SetMaxOpenConns
    • SetMaxIdleConns
    • SetConnMaxLifetime
    • Great article by @methane
    
    • http://dsas.blog.klab.org/archives/2018-02/configure-sql-db.html
    

    View Slide

20. We are hiring Go Software Engineer!
    https://careers.mercari.com/job/
    

    View Slide