Scala Exchange 2014: Netflix + Scala

Slide 1

Slide 1 text

Netﬂix + Scala Brendan McAdams (@rit) bmcadams@netﬂix.com

Slide 2

Slide 2 text

What My Team Does a.k.a. "What do you use Scala for?"

Slide 3

Slide 3 text

APIs! » RESTful services comprising business & domain logic » UI built on top of REST using AngularJS » Scalatra with Swagger

Slide 4

Slide 4 text

Collecting Valuable Knowledge » Analyze information about devices running Netflix before public release » Display Capabilities: 3D? 720p? 1080p? UltraHD (4K)? » Remote Control: Dedicated Netflix button? » Lots of other metadata of value to Netflix

Slide 5

Slide 5 text

Collecting Valuable Knowledge » Used widely internally » Certiﬁcation: run accurate tests » UI: Optimum look, feel, and behavior » Support: debug customer issues

Slide 6

Slide 6 text

Why We Love Scala » Concise yet expressive language » Hybridisation of OO and FP provides ﬂexibility » Great libraries and toolchains » Solid Java interop » Stays crunchy in milk

Slide 7

Slide 7 text

The Right Tool(s) for the Job » Plays well with Netﬂix' Platform & JVM ecosystem » Reuse existing Java & Groovy code » Lots of powerful Java libraries in the Netﬂix toolbelt, easily accessible from Scala

Slide 8

Slide 8 text

The Right Tool(s) for the Job » Production containers to inject healthchecks, management hooks, etc... » Scalatra deploys into existing infrastructure as standard J2EE app » Existing Gradle build system & Jenkins CI well suited for JVM languages

Slide 9

Slide 9 text

The Right Tool(s) for the Job » A lot of Netﬂix' core tools are open source » A few: » Simian Army: Chaos Monkey helps ensure resilient systems » Asgard: AWS Service Management » Hysterix: Latency & Fault Tolerance helpers » http://netﬂix.github.io/

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

A Little Swagger In Our Step » Swagger makes API Development a lot easier for us » Since another group of developers interacts with our service » They have interactive, detailed documentation at their ﬁngertips

Slide 13

Slide 13 text

A Little Swagger In Our Step » Swagger is built around self documenting code » Documentation is always up to date. » Swagger itself is mostly a JSON based "Speciﬁcation" » Scalatra module for describing API, generates JSON » swagger-ui JSON Frontend explores API

Slide 14

Slide 14 text

Swagger at a Glance

Slide 15

Slide 15 text

"GET" get("/", operation(getTestAccounts)) { withAuthenticatedUser { authUser: AuthenticatedUser => val testAccounts = TestAccountLogManager.ﬁndByUser( authUser ) Ok( ApiResponse(testAccounts) ) } }

Slide 16

Slide 16 text

"GET" def getTestAccounts = authApiOperation[TestAccountResponse]( "getTestAccounts", """Returns a list of the test accounts created by the current user with this tool.""")

Slide 17

Slide 17 text

"GET" def getTestAccounts = authApiOperation[TestAccountResponse]( "getTestAccounts", """Returns a list of the test accounts created by the current user with this tool.""") » Swagger automatically documents the API from the authApiOperation method... » The base Swagger method is apiOperation, which we've extended internally to support our auth layer.

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

Creating a Test Account post("/", operation(createTestAccount)) { withAuthenticatedUser { user: AuthenticatedUser => withValidJsonBody[TestAccountRequest] { testAccountReq => createAcctForUser(user, testAccountReq ) } } }

Slide 22

Slide 22 text

Creating a Test Account def createTestAccount = authApiOperation[String]("createTestAccount", """Creates a test account, returning JSON result from the API""") .parameter( bodyParam[TestAccountRequest]("") .description( """The request includes the new Netﬂix test account's country of origin, email address, password and (test) user's ﬁrst name. """) )

Slide 23

Slide 23 text

"POST" def createTestAccount = authApiOperation[String]("createTestAccount", "Creates a test account, returning JSON result from the API") .parameter( bodyParam[TestAccountRequest]("").description( """The request includes the new Netﬂix test account's country of origin, email address, password and (test) user's ﬁrst name. """) ) » In the case of more complex methods, we can include parameter information » e.g. bodyParam, pathParam, queryParam, headerParam

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

Swagger documents our bodyParam

Slide 26

Slide 26 text

Hitting Model Schema shows a JSON schema of the Parameter

Slide 27

Slide 27 text

Software Deployments

Slide 28

Slide 28 text

Traditional Software Deployment » The "traditional" J2EE deployment model » WAR ﬁles deployed to Tomcat/Jetty/My Lovecraftian J2EE Horror

Slide 29

Slide 29 text

Traditional Software Deployment » Software changes? Just redeploy to the same old server » Rolling back to old versions isn't always straight forward » Chasing dependency conﬂicts is never fun

Slide 30

Slide 30 text

Traditional Software Deployment » Reproducibility is low: If the server changes, hard to solve problems » Did the new build break? » Did someone muck with the software on the server w/o telling anyone? » Was the server invaded by Gibbering Horrors from Beyond Spacetime?

Slide 31

Slide 31 text

Immutable Deployments A Better Way » Build server images as immutable AMIs » Never modiﬁed once deployed » Need to roll back? Pick an image of the last known working version...

Slide 32

Slide 32 text

Immutable Deployments » Building an image requires a valid Jenkins build » Only Jenkins can merge to 'master'

Slide 33

Slide 33 text

Immutable Deployments » In this setup, good testing is key to success » We have tools to manage the deployment of server images

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Immutable Deployments » Asgard (open source) helps manage Amazon services, including Auto Scaling Groups » Servers grouped into clusters ensures that each deployment group (e.g. Dev, Staging, Production) are of one cohesive version » Last working version available as a hot spare / backup

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

Smaller is Better » Build lots of small services » Move quickly, move independently » APIs are powerful in this model » API can feed UI, Data Services, etc without a monolithic dev cycle

Slide 40

Slide 40 text

Good Testing Matters » This should go without saying, but good tests matter quite a lot » Especially in an environment where you're building immutable images... » And relying upon CI to determine the validity of a build

Slide 41

Slide 41 text

Good Testing Matters » We use and love ScalaTest » Specs2 is pretty awesome as well

Slide 42

Slide 42 text

Scalatra Testing » Scalatra ships with some nice test helpers for ScalaTest, Specs2, and Scalacheck » How would we test that TestAccountService with ScalaTest?

Slide 43

Slide 43 text

Scalatra Testing class TestAccountServiceSpec extends BaseScalatraFlatSuite with Logging { addServlet(new TestAccountService, "/*") "Test Account Service" should "verify if email addresses use the Netﬂix domain" in { val body = """{ "country":"US", "password":"pass", "fname":"ScalaX" "email":"[email protected]", }""" postJsonWithToken("/", body)(TestUserTokens.brendanToken) { status should equal(BadRequest().status.code) log.debug("Returned, status = %s, body = %s", status, body) } } }

Slide 44

Slide 44 text

Testing Your API » Your API Framework of choice should provide good tools for testing it » If it doesn't, consider ﬁnding a better framework » Scalatra (the framework we're using) provides great bindings to Scalatest » This contributes greatly to feeling comfortable that what you've built works as documented

Slide 45

Slide 45 text

Write Lots of Tests » Positive Tests » Negative Tests » Service Level Tests (HTTP Calls) » Helper Level Tests (backend objects sans HTTP) » Parser Tests » Regression Tests » Voight-Kampff Tests

Slide 46

Slide 46 text

"Never Trust A Test You Haven't Seen Fail." - Rob Fletcher ( @rﬂetcherEW ) ... who says he nicked it from @tddmonkey (Colin Vipurs)

Slide 47

Slide 47 text

Errors! Errors! Errors!

Slide 48

Slide 48 text

ERRORS » Meaningless error messages lead to long days of slaving over error logs » Nothing sucks more than spending an hour digging through logs... » ... only to ﬁnd it was something you could have communicated with an error message

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Good Practices for Errors » If you can validate an entire request object, do so » Return every error you know about immediately, and clearly » If you can pass multiple errors up your stack and return them, do so... » JSON Supports Arrays, after all

Slide 51

Slide 51 text

Good Practices for Errors » Provide useful data, not a generic 500 Internal Server Error » If you have an exception, extract a message from it » However: Exceptions are not ﬂow control » Some useful tools out there in the Scala world for Error handling & passing...

Slide 52

Slide 52 text

We are Not Using the 'Zed' Word

Slide 53

Slide 53 text

We are Not Using the 'Zed' Word

Slide 54

Slide 54 text

scalaz is kind of awesome... » We've been playing with scalaz a bit lately » There's a lot in there, and a lot to absorb » The bits I've found most useful so far? » Validation » Disjunction ( \/ )

Slide 55

Slide 55 text

problems to solve » Providing clearer errors & validating input was a problem » API Server is part of a larger application: erroring is hard » 500s & generic exceptions, complicate frontend devs debugging

Slide 56

Slide 56 text

Helping developers help themselves » An Error Occurred » API received bad/invalid data? » Database failed? » What if multiple errors occurred? » How do we communicate this effectively?

Slide 57

Slide 57 text

» Validation + NonEmptyList is great for "These are all the things that went wrong" » Validation is Applicative: you can accrue the results of many Validations together » Using a NonEmptyList for Failure means the applicative adds each error » Operators for doing cool things » Implicits to convert things

Slide 58

Slide 58 text

» \/ (Disjunction), like Scala's Either, has a "Left" and a "Right" value » Unlike Scala's Either, \/ is Monadic: You can iterate over it » This is "right biased": like Option it unwraps if it is "Right" » Operators and Implicits for converting from things like Option

Slide 59

Slide 59 text

Come to my talk tomorrow: 'A Skeptic's Look at scalaz' "Gateway Drugs”': In Room 2 @ 1600!

Slide 60

Slide 60 text

Questions?