Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rewriting the Parse API in Go.

Rewriting the Parse API in Go.

GopherCon 2015 Talk on how we rewrote the Parse API

Abhishek Kona

July 09, 2015
Tweet

More Decks by Abhishek Kona

Other Decks in Programming

Transcript

  1. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 1/39 Rewriting

    the Parse API in Go GopherCon 2015, Denver 8 July 2015 Abhishek Kona Software Engineer at Parse and Facebook
  2. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 2/39 What

    is this talk about? Why we rewrote the Parse API in Go Tools and libraries we built
  3. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 3/39 What

    is Parse? Backend as a service SDKs for iOS, Android, JS, React, Windows, PHP Acquired by Facebook in 2013
  4. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 4/39 Rome

    was not built in a day - they did not use Parse
  5. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 5/39 Parse

    circa 2013 ~60K apps 10 engineers Ruby on Rails app
  6. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 6/39 Scalability

    Issues in 2013 Single popular app could take down Parse Fixed-size unicorn pool Lengthy deploy times Spooky action at a distance
  7. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 7/39 Solution:

    Rewrite in Go!
  8. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 8/39 Why

    Rewrite? Huge estimated reliability win Wanted easier deploys Needed faster tests Hard to evolve to existing Ruby codebase
  9. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 9/39 Why

    Go? Statically typed Good concurrency support Dynamic number of worker goroutines per HTTP server Easy to attract engineers
  10. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 10/39 Rules

    of the Rewrite Don't break backwards compatibility No downtime
  11. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 11/39 Initial

    Ports Hosting server Parse Push Notification Service (PPNS) Maintains long-lived push sockets with Android clients Concurrent conns per node increased from 250K to 1.5M
  12. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 12/39 Mongo

    Proxy github.com/facebookgo/dvara (https://github.com/facebookgo/dvara) Mongo used to limit max number of connections to 20000 We wrote our own proxy for Mongo in Go Made easy by Go runtime's use of non-blocking I/O
  13. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 13/39 Rollout

    Migrate endpoints one by one Diffed responses between old and new code using a shadow cluster Started with low-traffic read only endpoints Graduated to write endpoints
  14. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 14/39 Comment

    Goldmine / / N o t e : a n u n s e t c a c h e v e r s i o n i s t r e a t e d b y r u b y a s “ ” . / / B e c a u s e o f t h i s , d i r t y i n g t h i s i s n ’ t a s s i m p l e a s d e l e t i n g i t – w e n e e d t o / / a c t u a l l y s e t a n e w v a l u e . / / T h i s b y t e s e q u e n c e i s w h a t r u b y e x p e c t s . / / y e s t h a t ’ s a p a r e n a f t e r t h e s e c o n d 1 8 0 , p e r r u b y . / / I n s e r t i n g a n d h a v i n g a n o p i s k i n d a w e i r d : W e a l r e a d y k n o w / / s t a t e z e r o . B u t r u b y s u p p o r t s i t , s o g o d o e s t o o . / / s i n g l e g e o q u e r y , d o n ’ t d o a n y t h i n g . s t u p i d a n d d o e s n o t m a k e s e n s e / / b u t r u b y d o e s i t . C h a n g i n g t h i s w i l l b r e a k a l o t o f c l i e n t t e s t s . / / j u s t b e n i c e a n d f i x i t h e r e . / / R u b y s e t s v a r i o u s d e f a u l t s d i r e c t l y i n t h e s t r u c t u r e a n d e x p e c t s t h e m t o a p p e a r i n c a c h e . / / F o r c o n s i s t e n c y , w e ’ l l d o t h e s a m e t h i n g .
  15. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 15/39 A

    Young Language Some good libraries: mgo, memcache, etc. Some missing libraries
  16. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 16/39 Libraries

    / Tools
  17. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 17/39 Dependency

    Injection Helps instantiate implementations for test and production Easy to miss passing a dependency to a struct
  18. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 18/39 Introducing

    Inject github.com/facebookgo/inject (http://github.com/facebookgo/inject) Only occurs at process startup for singletons Dependencies declared using struct tags Fail instead of guessing
  19. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 19/39 Dependency

    Injection Example t y p e H a n d l e r s t r u c t { S c r i b e * s c r i b e . C l i e n t ` i n j e c t : " " ` L o g l o g g e r . L o g g e r ` i n j e c t : " " ` } / / S e r v e H T T P a s a m p l e i m p l e m e n t a t i o n f u n c ( h * H a n d l e r ) S e r v e H T T P ( w R e s p o n s e W r i t e r , r * R e q u e s t ) { p a r a m s : = e x t r a c t P a r a m s ( r ) h . S c r i b e . L o g ( p a r a m s ) h . L o g ( " e v e r y t h i n g o k " ) w . W r i t e ( r e s ) }
  20. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 20/39 Main

    for Inject f u n c m a i n ( ) { v a r g i n j e c t . G r a p h e r r : = g . P r o v i d e ( & i n j e c t . O b j e c t { V a l u e : s c r i b e . N e w H T T P S c r i b e C l i e n t ( ) } , & i n j e c t . O b j e c t { V a l u e : p a r s e . N e w L o g g e r ( ) } , ) i f e r r ! = n i l { f m t . F p r i n t l n ( o s . S t d e r r , e r r ) o s . E x i t ( 1 ) } i f e r r : = g . P o p u l a t e ( ) ; e r r ! = n i l { f m t . F p r i n t l n ( o s . S t d e r r , e r r ) o s . E x i t ( 1 ) } / / r e s t o f m a i n }
  21. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 21/39 Initializing

    and Destroying Injected Objects github.com/facebookgo/startstop (http://github.com/facebookgo/startstop) Traverses object graph At startup: calls S t a r t on each injected object in dependency order At shutdown: calls S t o p on each injected object in reverse dependency order Fails on cycles
  22. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 22/39 Start-Stop

    Example t y p e S c r i b e C l i e n t { T h r i f t * T h r i f t P o o l ` i n j e c t : " " ` } / / S c r i b e C l i e n t s t a r t w i l l b e c a l l e d a f t e r T h r i f t P o o l . S t a r t f u n c ( s * S c r i b e C l i e n t ) S t a r t ( ) e r r o r { f m t . P r i n t l n ( " s t a r t i n g s c r i b e c l i e n t " ) r e t u r n n i l } t y p e T h r i f t P o o l s t r u c t { / / } / / T h r i f t P o o l s t a r t w i l l b e c a l l e d f i r s t . f u n c ( t * T h r i f t P o o l ) S t a r t ( ) e r r o r { f m t . P r i n t l n ( " s t a r t i n g t h r i f t p o o l " ) r e t u r n t . t c p D i a l ( ) } f u n c ( t * T h r i f t P o o l ) S t o p ( ) e r r o r { f m t . P r i n t l n ( " s t o p p i n g t h r i f t p o o l " ) r e t u r n t . t c p C l o s e A l l ( ) }
  23. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 23/39 Graceful

    Restarts github.com/facebookgo/grace (https://github.com/facebookgo/grace) Restart servers gracefully on deploys On USR2, spawns new process and hands off listening socket
  24. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 24/39 Error

    Reporting github.com/facebookgo/stackerr (https://github.com/facebookgo/stackerr) Wrap error calls with stackerr Aggregate errors based on stack trace in an in-house system called Logview
  25. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 25/39 Stackerr

    Example f u n c m a i n ( ) { e r r : = e r r 2 ( ) f m t . P r i n t l n ( e r r ) } f u n c e r r 2 ( ) e r r o r { e r r : = e r r 1 ( ) i f e r r ! = n i l { r e t u r n s t a c k e r r . W r a p ( e r r ) } r e t u r n n i l } f u n c e r r 1 ( ) e r r o r { r e t u r n s t a c k e r r . W r a p ( e r r o r s . N e w ( " f a i l u r e " ) ) }
  26. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 26/39 Stackerr

    Output f a i l u r e / p r i v a t e / t m p / s t a c k e r r . g o : 2 5 e r r 1 / p r i v a t e / t m p / s t a c k e r r . g o : 1 7 e r r 2 / p r i v a t e / t m p / s t a c k e r r . g o : 1 2 m a i n / u s r / l o c a l / C e l l a r / g o / 1 . 4 . 2 / l i b e x e c / s r c / r u n t i m e / p r o c . g o : 7 2 m a i n / u s r / l o c a l / C e l l a r / g o / 1 . 4 . 2 / l i b e x e c / s r c / r u n t i m e / a s m _ a m d 6 4 . s : 2 2 3 3 g o e x i t ( S t a c k 2 ) / p r i v a t e / t m p / s t a c k e r r . g o : 1 9 e r r 2 / p r i v a t e / t m p / s t a c k e r r . g o : 1 2 m a i n / u s r / l o c a l / C e l l a r / g o / 1 . 4 . 2 / l i b e x e c / s r c / r u n t i m e / p r o c . g o : 7 2 m a i n / u s r / l o c a l / C e l l a r / g o / 1 . 4 . 2 / l i b e x e c / s r c / r u n t i m e / a s m _ a m d 6 4 . s : 2 2 3 3 g o e x i t
  27. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 27/39 Muster

    github.com/facebookgo/muster (https://github.com/facebookgo/muster) A library to perform operations in a batch Two tunables: M a x B a t c h S i z e and B a t c h T i m e o u t
  28. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 28/39 Generics

    github.com/facebookgo/generics (https://github.com/facebookgo/generics)
  29. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 29/39 More

    Libraries github.com/facebookgo (https://github.com/facebookgo) Many more small libraries httpcontrol, ensure, stack
  30. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 30/39 We

    Love Go github.com/daaku (github.com/daaku)
  31. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 31/39 Results

  32. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 32/39 Results

    ~175k LOC in Go vs ~130k LOC in Ruby ~3 minutes to run all the unit tests (down from 25min) Apps start in seconds instead of minutes Downsized API server pool by 90% Rolling restarts dropped from 30 minutes to 3 minutes
  33. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 33/39 Parse

    circa 2015 >500K apps built on Parse 2-3x YoY traffic growth Primarily a Go stack
  34. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 34/39 Observations

    Rewrites are hard ~4 engineers over 2 years
  35. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 35/39 Go

    Side Effects Deploying with static binaries is easy Developers are responsible for deploys, not ops
  36. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 36/39 Recap

  37. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 37/39 http://tiny.cc/parsego

  38. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 38/39 Thank

    you Abhishek Kona Software Engineer at Parse and Facebook abhishekk@fb.com (mailto:abhishekk@fb.com) http://sheki.in (http://sheki.in) @sheki (http://twitter.com/sheki)
  39. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 39/39