Rewriting the Parse API in Go.

GopherCon 2015 Talk on how we rewrote the Parse API

Transcript

1. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 1/39 Rewriting the Parse API in Go GopherCon 2015, Denver 8 July 2015 Abhishek Kona Software Engineer at Parse and Facebook
2. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 2/39 What is this talk about? Why we rewrote the Parse API in Go Tools and libraries we built
3. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 3/39 What is Parse? Backend as a service SDKs for iOS, Android, JS, React, Windows, PHP Acquired by Facebook in 2013
4. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 4/39 Rome was not built in a day - they did not use Parse
5. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 5/39 Parse circa 2013 ~60K apps 10 engineers Ruby on Rails app
6. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 6/39 Scalability Issues in 2013 Single popular app could take down Parse Fixed-size unicorn pool Lengthy deploy times Spooky action at a distance
7. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 7/39 Solution: Rewrite in Go!
8. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 8/39 Why Rewrite? Huge estimated reliability win Wanted easier deploys Needed faster tests Hard to evolve to existing Ruby codebase
9. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 9/39 Why Go? Statically typed Good concurrency support Dynamic number of worker goroutines per HTTP server Easy to attract engineers
10. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 10/39 Rules of the Rewrite Don't break backwards compatibility No downtime
11. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 11/39 Initial Ports Hosting server Parse Push Notification Service (PPNS) Maintains long-lived push sockets with Android clients Concurrent conns per node increased from 250K to 1.5M
12. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 12/39 Mongo Proxy github.com/facebookgo/dvara (https://github.com/facebookgo/dvara) Mongo used to limit max number of connections to 20000 We wrote our own proxy for Mongo in Go Made easy by Go runtime's use of non-blocking I/O
13. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 13/39 Rollout Migrate endpoints one by one Diffed responses between old and new code using a shadow cluster Started with low-traffic read only endpoints Graduated to write endpoints
14. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 14/39 Comment Goldmine / / N o t e : a n u n s e t c a c h e v e r s i o n i s t r e a t e d b y r u b y a s “ ” . / / B e c a u s e o f t h i s , d i r t y i n g t h i s i s n ’ t a s s i m p l e a s d e l e t i n g i t – w e n e e d t o / / a c t u a l l y s e t a n e w v a l u e . / / T h i s b y t e s e q u e n c e i s w h a t r u b y e x p e c t s . / / y e s t h a t ’ s a p a r e n a f t e r t h e s e c o n d 1 8 0 , p e r r u b y . / / I n s e r t i n g a n d h a v i n g a n o p i s k i n d a w e i r d : W e a l r e a d y k n o w / / s t a t e z e r o . B u t r u b y s u p p o r t s i t , s o g o d o e s t o o . / / s i n g l e g e o q u e r y , d o n ’ t d o a n y t h i n g . s t u p i d a n d d o e s n o t m a k e s e n s e / / b u t r u b y d o e s i t . C h a n g i n g t h i s w i l l b r e a k a l o t o f c l i e n t t e s t s . / / j u s t b e n i c e a n d f i x i t h e r e . / / R u b y s e t s v a r i o u s d e f a u l t s d i r e c t l y i n t h e s t r u c t u r e a n d e x p e c t s t h e m t o a p p e a r i n c a c h e . / / F o r c o n s i s t e n c y , w e ’ l l d o t h e s a m e t h i n g .
15. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 15/39 A Young Language Some good libraries: mgo, memcache, etc. Some missing libraries
16. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 16/39 Libraries / Tools
17. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 17/39 Dependency Injection Helps instantiate implementations for test and production Easy to miss passing a dependency to a struct
18. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 18/39 Introducing Inject github.com/facebookgo/inject (http://github.com/facebookgo/inject) Only occurs at process startup for singletons Dependencies declared using struct tags Fail instead of guessing
19. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 19/39 Dependency Injection Example t y p e H a n d l e r s t r u c t { S c r i b e * s c r i b e . C l i e n t ` i n j e c t : " " ` L o g l o g g e r . L o g g e r ` i n j e c t : " " ` } / / S e r v e H T T P a s a m p l e i m p l e m e n t a t i o n f u n c ( h * H a n d l e r ) S e r v e H T T P ( w R e s p o n s e W r i t e r , r * R e q u e s t ) { p a r a m s : = e x t r a c t P a r a m s ( r ) h . S c r i b e . L o g ( p a r a m s ) h . L o g ( " e v e r y t h i n g o k " ) w . W r i t e ( r e s ) }
20. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 20/39 Main for Inject f u n c m a i n ( ) { v a r g i n j e c t . G r a p h e r r : = g . P r o v i d e ( & i n j e c t . O b j e c t { V a l u e : s c r i b e . N e w H T T P S c r i b e C l i e n t ( ) } , & i n j e c t . O b j e c t { V a l u e : p a r s e . N e w L o g g e r ( ) } , ) i f e r r ! = n i l { f m t . F p r i n t l n ( o s . S t d e r r , e r r ) o s . E x i t ( 1 ) } i f e r r : = g . P o p u l a t e ( ) ; e r r ! = n i l { f m t . F p r i n t l n ( o s . S t d e r r , e r r ) o s . E x i t ( 1 ) } / / r e s t o f m a i n }
21. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 21/39 Initializing and Destroying Injected Objects github.com/facebookgo/startstop (http://github.com/facebookgo/startstop) Traverses object graph At startup: calls S t a r t on each injected object in dependency order At shutdown: calls S t o p on each injected object in reverse dependency order Fails on cycles
22. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 22/39 Start-Stop Example t y p e S c r i b e C l i e n t { T h r i f t * T h r i f t P o o l ` i n j e c t : " " ` } / / S c r i b e C l i e n t s t a r t w i l l b e c a l l e d a f t e r T h r i f t P o o l . S t a r t f u n c ( s * S c r i b e C l i e n t ) S t a r t ( ) e r r o r { f m t . P r i n t l n ( " s t a r t i n g s c r i b e c l i e n t " ) r e t u r n n i l } t y p e T h r i f t P o o l s t r u c t { / / } / / T h r i f t P o o l s t a r t w i l l b e c a l l e d f i r s t . f u n c ( t * T h r i f t P o o l ) S t a r t ( ) e r r o r { f m t . P r i n t l n ( " s t a r t i n g t h r i f t p o o l " ) r e t u r n t . t c p D i a l ( ) } f u n c ( t * T h r i f t P o o l ) S t o p ( ) e r r o r { f m t . P r i n t l n ( " s t o p p i n g t h r i f t p o o l " ) r e t u r n t . t c p C l o s e A l l ( ) }
23. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 23/39 Graceful Restarts github.com/facebookgo/grace (https://github.com/facebookgo/grace) Restart servers gracefully on deploys On USR2, spawns new process and hands off listening socket
24. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 24/39 Error Reporting github.com/facebookgo/stackerr (https://github.com/facebookgo/stackerr) Wrap error calls with stackerr Aggregate errors based on stack trace in an in-house system called Logview
25. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 25/39 Stackerr Example f u n c m a i n ( ) { e r r : = e r r 2 ( ) f m t . P r i n t l n ( e r r ) } f u n c e r r 2 ( ) e r r o r { e r r : = e r r 1 ( ) i f e r r ! = n i l { r e t u r n s t a c k e r r . W r a p ( e r r ) } r e t u r n n i l } f u n c e r r 1 ( ) e r r o r { r e t u r n s t a c k e r r . W r a p ( e r r o r s . N e w ( " f a i l u r e " ) ) }
26. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 26/39 Stackerr Output f a i l u r e / p r i v a t e / t m p / s t a c k e r r . g o : 2 5 e r r 1 / p r i v a t e / t m p / s t a c k e r r . g o : 1 7 e r r 2 / p r i v a t e / t m p / s t a c k e r r . g o : 1 2 m a i n / u s r / l o c a l / C e l l a r / g o / 1 . 4 . 2 / l i b e x e c / s r c / r u n t i m e / p r o c . g o : 7 2 m a i n / u s r / l o c a l / C e l l a r / g o / 1 . 4 . 2 / l i b e x e c / s r c / r u n t i m e / a s m _ a m d 6 4 . s : 2 2 3 3 g o e x i t ( S t a c k 2 ) / p r i v a t e / t m p / s t a c k e r r . g o : 1 9 e r r 2 / p r i v a t e / t m p / s t a c k e r r . g o : 1 2 m a i n / u s r / l o c a l / C e l l a r / g o / 1 . 4 . 2 / l i b e x e c / s r c / r u n t i m e / p r o c . g o : 7 2 m a i n / u s r / l o c a l / C e l l a r / g o / 1 . 4 . 2 / l i b e x e c / s r c / r u n t i m e / a s m _ a m d 6 4 . s : 2 2 3 3 g o e x i t
27. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 27/39 Muster github.com/facebookgo/muster (https://github.com/facebookgo/muster) A library to perform operations in a batch Two tunables: M a x B a t c h S i z e and B a t c h T i m e o u t
28. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 28/39 Generics github.com/facebookgo/generics (https://github.com/facebookgo/generics)
29. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 29/39 More Libraries github.com/facebookgo (https://github.com/facebookgo) Many more small libraries httpcontrol, ensure, stack
30. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 30/39 We Love Go github.com/daaku (github.com/daaku)
31. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 31/39 Results
32. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 32/39 Results ~175k LOC in Go vs ~130k LOC in Ruby ~3 minutes to run all the unit tests (down from 25min) Apps start in seconds instead of minutes Downsized API server pool by 90% Rolling restarts dropped from 30 minutes to 3 minutes
33. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 33/39 Parse circa 2015 >500K apps built on Parse 2-3x YoY traffic growth Primarily a Go stack
34. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 34/39 Observations Rewrites are hard ~4 engineers over 2 years
35. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 35/39 Go Side Effects Deploying with static binaries is easy Developers are responsible for deploys, not ops
36. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 36/39 Recap
37. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 37/39 http://tiny.cc/parsego
38. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 38/39 Thank you Abhishek Kona Software Engineer at Parse and Facebook abhishekk@fb.com (mailto:abhishekk@fb.com) http://sheki.in (http://sheki.in) @sheki (http://twitter.com/sheki)
39. 7/8/2015 Rewriting the Parse API in Go http://localhost:3999/gophercon_2015.slide#1 39/39