Rewriting the Parse API in Go.

7/8/2015 Rewriting the Parse API in Go
http://localhost:3999/gophercon_2015.slide#1 1/39
Rewriting the Parse API in Go
GopherCon 2015, Denver
8 July 2015
Abhishek Kona
Software Engineer at Parse and Facebook

View Slide

What is this talk about?
Why we rewrote the Parse API in Go
Tools and libraries we built

View Slide

What is Parse?
Backend as a service
SDKs for iOS, Android, JS, React, Windows, PHP
Acquired by Facebook in 2013

View Slide

Rome was not built in a day - they did not use Parse

View Slide

Parse circa 2013
~60K apps
10 engineers
Ruby on Rails app

View Slide

Scalability Issues in 2013
Single popular app could take down Parse
Fixed-size unicorn pool
Lengthy deploy times
Spooky action at a distance

View Slide

Solution: Rewrite in Go!

View Slide

Why Rewrite?
Huge estimated reliability win
Wanted easier deploys
Needed faster tests
Hard to evolve to existing Ruby codebase

View Slide

Why Go?
Statically typed
Good concurrency support
Dynamic number of worker goroutines per HTTP server
Easy to attract engineers

View Slide

Rules of the Rewrite
Don't break backwards compatibility
No downtime

View Slide

Initial Ports
Hosting server
Parse Push Notification Service (PPNS)
Maintains long-lived push sockets with Android clients
Concurrent conns per node increased from 250K to 1.5M

View Slide

Mongo Proxy
github.com/facebookgo/dvara (https://github.com/facebookgo/dvara)
Mongo used to limit max number of connections to 20000
We wrote our own proxy for Mongo in Go
Made easy by Go runtime's use of non-blocking I/O

View Slide

Rollout
Migrate endpoints one by one
Diffed responses between old and new code using a shadow cluster
Started with low-traffic read only endpoints
Graduated to write endpoints

View Slide

Comment Goldmine
/
/ N
o
t
e
: a
n u
n
s
e
t c
a
c
h
e v
e
r
s
i
o
n i
s t
r
e
a
t
e
d b
y r
u
b
y a
s “
”
.
/
/ B
e
c
a
u
s
e o
f t
h
i
s
, d
i
r
t
y
i
n
g t
h
i
s i
s
n
’
t a
s s
i
m
p
l
e a
s d
e
l
e
t
i
n
g i
t – w
e n
e
e
d t
o
/
/ a
c
t
u
a
l
l
y s
e
t a n
e
w v
a
l
u
e
.
/
/ T
h
i
s b
y
t
e s
e
q
u
e
n
c
e i
s w
h
a
t r
u
b
y e
x
p
e
c
t
s
.
/
/ y
e
s t
h
a
t
’
s a p
a
r
e
n a
f
t
e
r t
h
e s
e
c
o
n
d 1
8
0
, p
e
r r
u
b
y
.
/
/ I
n
s
e
r
t
i
n
g a
n
d h
a
v
i
n
g a
n o
p i
s k
i
n
d
a w
e
i
r
d
: W
e a
l
r
e
a
d
y k
n
o
w
/
/ s
t
a
t
e z
e
r
o
. B
u
t r
u
b
y s
u
p
p
o
r
t
s i
t
, s
o g
o d
o
e
s t
o
o
.
/
/ s
i
n
g
l
e g
e
o q
u
e
r
y
, d
o
n
’
t d
o a
n
y
t
h
i
n
g
. s
t
u
p
i
d a
n
d d
o
e
s n
o
t m
a
k
e s
e
n
s
e
/
/ b
u
t r
u
b
y d
o
e
s i
t
. C
h
a
n
g
i
n
g t
h
i
s w
i
l
l b
r
e
a
k a l
o
t o
f c
l
i
e
n
t t
e
s
t
s
.
/
/ j
u
s
t b
e n
i
c
e a
n
d f
i
x i
t h
e
r
e
.
/
/ R
u
b
y s
e
t
s v
a
r
i
o
u
s d
e
f
a
u
l
t
s d
i
r
e
c
t
l
y i
n t
h
e s
t
r
u
c
t
u
r
e a
n
d e
x
p
e
c
t
s t
h
e
m t
o a
p
p
e
a
r i
n c
a
c
h
e
.
/
/ F
o
r c
o
n
s
i
s
t
e
n
c
y
, w
e
’
l
l d
o t
h
e s
a
m
e t
h
i
n
g
.

View Slide

A Young Language
Some good libraries: mgo, memcache, etc.
Some missing libraries

View Slide

Libraries / Tools

View Slide

Dependency Injection
Helps instantiate implementations for test and production
Easy to miss passing a dependency to a struct

View Slide

Introducing Inject
github.com/facebookgo/inject (http://github.com/facebookgo/inject)
Only occurs at process startup for singletons
Dependencies declared using struct tags
Fail instead of guessing

View Slide

Dependency Injection Example
t
y
p
e H
a
n
d
l
e
r s
t
r
u
c
t {
S
c
r
i
b
e *
s
c
r
i
b
e
.
C
l
i
e
n
t `
i
n
j
e
c
t
:
"
"
`
L
o
g l
o
g
g
e
r
.
L
o
g
g
e
r `
i
n
j
e
c
t
:
"
"
`
}
/
/ S
e
r
v
e
H
T
T
P a s
a
m
p
l
e i
m
p
l
e
m
e
n
t
a
t
i
o
n
f
u
n
c (
h *
H
a
n
d
l
e
r
) S
e
r
v
e
H
T
T
P
(
w R
e
s
p
o
n
s
e
W
r
i
t
e
r
, r *
R
e
q
u
e
s
t
) {
p
a
r
a
m
s :
= e
x
t
r
a
c
t
P
a
r
a
m
s
(
r
)
h
.
S
c
r
i
b
e
.
L
o
g
(
p
a
r
a
m
s
)
h
.
L
o
g
(
"
e
v
e
r
y
t
h
i
n
g o
k
"
)
w
.
W
r
i
t
e
(
r
e
s
)
}

View Slide

Main for Inject
f
u
n
c m
a
i
n
(
) {
v
a
r g i
n
j
e
c
t
.
G
r
a
p
h
e
r
r :
= g
.
P
r
o
v
i
d
e
(
&
i
n
j
e
c
t
.
O
b
j
e
c
t
{
V
a
l
u
e
: s
c
r
i
b
e
.
N
e
w
H
T
T
P
S
c
r
i
b
e
C
l
i
e
n
t
(
)
}
,
&
i
n
j
e
c
t
.
O
b
j
e
c
t
{
V
a
l
u
e
: p
a
r
s
e
.
N
e
w
L
o
g
g
e
r
(
)
}
,
)
i
f e
r
r !
= n
i
l {
f
m
t
.
F
p
r
i
n
t
l
n
(
o
s
.
S
t
d
e
r
r
, e
r
r
)
o
s
.
E
x
i
t
(
1
)
}
i
f e
r
r :
= g
.
P
o
p
u
l
a
t
e
(
)
; e
r
r !
= n
i
l {
f
m
t
.
F
p
r
i
n
t
l
n
(
o
s
.
S
t
d
e
r
r
, e
r
r
)
o
s
.
E
x
i
t
(
1
)
}
/
/ r
e
s
t o
f m
a
i
n
}

View Slide

Initializing and Destroying Injected Objects
github.com/facebookgo/startstop (http://github.com/facebookgo/startstop)
Traverses object graph
At startup: calls S
t
a
r
t
on each injected object in dependency order
At shutdown: calls S
t
o
p
on each injected object in reverse dependency order
Fails on cycles

View Slide

Start-Stop Example
t
y
p
e S
c
r
i
b
e
C
l
i
e
n
t {
T
h
r
i
f
t *
T
h
r
i
f
t
P
o
o
l `
i
n
j
e
c
t
:
"
"
`
}
/
/ S
c
r
i
b
e
C
l
i
e
n
t s
t
a
r
t w
i
l
l b
e c
a
l
l
e
d a
f
t
e
r T
h
r
i
f
t
P
o
o
l
.
S
t
a
r
t
f
u
n
c (
s *
S
c
r
i
b
e
C
l
i
e
n
t
) S
t
a
r
t
(
) e
r
r
o
r {
f
m
t
.
P
r
i
n
t
l
n
(
"
s
t
a
r
t
i
n
g s
c
r
i
b
e c
l
i
e
n
t
"
)
r
e
t
u
r
n n
i
l
}
t
y
p
e T
h
r
i
f
t
P
o
o
l s
t
r
u
c
t {
/
/
}
/
/ T
h
r
i
f
t
P
o
o
l s
t
a
r
t w
i
l
l b
e c
a
l
l
e
d f
i
r
s
t
.
f
u
n
c (
t *
T
h
r
i
f
t
P
o
o
l
) S
t
a
r
t
(
) e
r
r
o
r {
f
m
t
.
P
r
i
n
t
l
n
(
"
s
t
a
r
t
i
n
g t
h
r
i
f
t p
o
o
l
"
)
r
e
t
u
r
n t
.
t
c
p
D
i
a
l
(
)
}
f
u
n
c (
t *
T
h
r
i
f
t
P
o
o
l
) S
t
o
p
(
) e
r
r
o
r {
f
m
t
.
P
r
i
n
t
l
n
(
"
s
t
o
p
p
i
n
g t
h
r
i
f
t p
o
o
l
"
)
r
e
t
u
r
n t
.
t
c
p
C
l
o
s
e
A
l
l
(
)
}

View Slide

Graceful Restarts
github.com/facebookgo/grace (https://github.com/facebookgo/grace)
Restart servers gracefully on deploys
On USR2, spawns new process and hands off listening socket

View Slide

Error Reporting
github.com/facebookgo/stackerr (https://github.com/facebookgo/stackerr)
Wrap error calls with stackerr
Aggregate errors based on stack trace in an in-house system called Logview

View Slide

Stackerr Example
f
u
n
c m
a
i
n
(
) {
e
r
r :
= e
r
r
2
(
)
f
m
t
.
P
r
i
n
t
l
n
(
e
r
r
)
}
f
u
n
c e
r
r
2
(
) e
r
r
o
r {
e
r
r :
= e
r
r
1
(
)
i
f e
r
r !
= n
i
l {
r
e
t
u
r
n s
t
a
c
k
e
r
r
.
W
r
a
p
(
e
r
r
)
}
r
e
t
u
r
n n
i
l
}
f
u
n
c e
r
r
1
(
) e
r
r
o
r {
r
e
t
u
r
n s
t
a
c
k
e
r
r
.
W
r
a
p
(
e
r
r
o
r
s
.
N
e
w
(
"
f
a
i
l
u
r
e
"
)
)
}

View Slide

Stackerr Output
f
a
i
l
u
r
e
/
p
r
i
v
a
t
e
/
t
m
p
/
s
t
a
c
k
e
r
r
.
g
o
:
2
5 e
r
r
1
/
p
r
i
v
a
t
e
/
t
m
p
/
s
t
a
c
k
e
r
r
.
g
o
:
1
7 e
r
r
2
/
p
r
i
v
a
t
e
/
t
m
p
/
s
t
a
c
k
e
r
r
.
g
o
:
1
2 m
a
i
n
/
u
s
r
/
l
o
c
a
l
/
C
e
l
l
a
r
/
g
o
/
1
.
4
.
2
/
l
i
b
e
x
e
c
/
s
r
c
/
r
u
n
t
i
m
e
/
p
r
o
c
.
g
o
:
7
2 m
a
i
n
/
u
s
r
/
l
o
c
a
l
/
C
e
l
l
a
r
/
g
o
/
1
.
4
.
2
/
l
i
b
e
x
e
c
/
s
r
c
/
r
u
n
t
i
m
e
/
a
s
m
_
a
m
d
6
4
.
s
:
2
2
3
3 g
o
e
x
i
t
(
S
t
a
c
k 2
)
/
p
r
i
v
a
t
e
/
t
m
p
/
s
t
a
c
k
e
r
r
.
g
o
:
1
9 e
r
r
2
/
p
r
i
v
a
t
e
/
t
m
p
/
s
t
a
c
k
e
r
r
.
g
o
:
1
2 m
a
i
n
/
u
s
r
/
l
o
c
a
l
/
C
e
l
l
a
r
/
g
o
/
1
.
4
.
2
/
l
i
b
e
x
e
c
/
s
r
c
/
r
u
n
t
i
m
e
/
p
r
o
c
.
g
o
:
7
2 m
a
i
n
/
u
s
r
/
l
o
c
a
l
/
C
e
l
l
a
r
/
g
o
/
1
.
4
.
2
/
l
i
b
e
x
e
c
/
s
r
c
/
r
u
n
t
i
m
e
/
a
s
m
_
a
m
d
6
4
.
s
:
2
2
3
3 g
o
e
x
i
t

View Slide

Muster
github.com/facebookgo/muster (https://github.com/facebookgo/muster)
A library to perform operations in a batch
Two tunables: M
a
x
B
a
t
c
h
S
i
z
e
and B
a
t
c
h
T
i
m
e
o
u
t

View Slide

Generics
github.com/facebookgo/generics (https://github.com/facebookgo/generics)

View Slide

More Libraries
github.com/facebookgo (https://github.com/facebookgo)
Many more small libraries
httpcontrol, ensure, stack

View Slide

We Love Go
github.com/daaku (github.com/daaku)

View Slide

Results

View Slide

Results
~175k LOC in Go vs ~130k LOC in Ruby
~3 minutes to run all the unit tests (down from 25min)
Apps start in seconds instead of minutes
Downsized API server pool by 90%
Rolling restarts dropped from 30 minutes to 3 minutes

View Slide

Parse circa 2015
>500K apps built on Parse
2-3x YoY traffic growth
Primarily a Go stack

View Slide

Observations
Rewrites are hard
~4 engineers over 2 years

View Slide

Go Side Effects
Deploying with static binaries is easy
Developers are responsible for deploys, not ops

View Slide

Recap

View Slide

http://tiny.cc/parsego

View Slide

Thank you
Abhishek Kona
Software Engineer at Parse and Facebook
[email protected] (mailto:[email protected])
http://sheki.in (http://sheki.in)
@sheki (http://twitter.com/sheki)

View Slide


View Slide

Rewriting the Parse API in Go.

Rewriting the Parse API in Go.

Abhishek Kona

More Decks by Abhishek Kona

Other Decks in Programming

Featured

Transcript