OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix

In this talk I’d like to introduce the Scala-based API stack at Partner Innovation Group at Netflix. After seeing a massive growth in the business model and the device ecosystem, we needed a system that could scale and be flexible at the same time. Scala provided the answer and we started with a basic set of APIs which, since then, has evolved towards complex but flexible business flows. Supporting metadata for over hundreds of brands and thousands of devices, the API development has followed a well thought-out, test-driven approach, git-flow, and what most API developers dread – documentation. I will talk about the architecture of the RESTful APIs, and the development + deployment process. We use Netflix-OSS components heavily in the architecture and cloud deployment, so I will cover them as well.

Swagger is what we used for type-safe documentation, which is really easy to use and integrate. I will briefly talk about customizations we’ve done to Swagger in order to make it far more usable at Netflix.

Throughout this effort there were lessons to be learnt, and plenty of best practices and recommendations for anyone starting out to build RESTful APIs, regardless of the platform or stack of choice. It’d be a great opportunity for me to walk through the architecture, and talk about the various components, technologies, and practices that are seeing increasing adoption in the modern, API driven landscape.

Transcript

1. API ecosystem with Scala, Scalatra, and Swagger at Netflix Manish Pandit @lobster1234
2. Manish Pandit Engineering Manager, Streaming Platforms Netflix @lobster1234
3. Thank you, OSCON!
7. 50 M+ Streaming Members 40 countries and counting Thousands of device types
8. Power the Consumer Electronics Partner Portal Enable Certification of Netflix Ready Devices Source of truth for all Device Data at Netflix Correlate Streaming Quality Metrics Our APIs
10. Devices Smart devices + Certification = Lots of Device Metadata!  
11. Model Firmware Screen Resolution Subtitle Support 3D DRM Remote Control Netflix SDK 4K HD …   Everything matters..
12. 58 Resources ~10 methods per resource
13. Background Netflix OSS Components Development Deployment/Delivery Open Floor Architecture HTTP  Layer,  and  Manager  Layer   Cassandra EVCache   Crowd/SSO RDS Astyanax Netflix OSS Cloud Components
14. Manager Layer = Business Code Protocol Independent Package-able as an artifact
15. HTTP Layer = Protocol Wrapper Map HTTP methods to Manager functions Transform JSON <=> Case Classes
16. The road trip from your IDE to a Production Node
18. Yes, it can be fun!
19. Background Netflix OSS Components Development Deployment/Delivery Open Floor
20. Background Netflix OSS Components Development Deployment/Delivery Open Floor Open Source Infrastructure Components Libraries Tools Frameworks PaaS with AWS IaaS
21. Background Netflix OSS Components Development Deployment/Delivery Open Floor
22. Scala Background Netflix OSS Components Development Deployment/Delivery Open Floor
23. Immutable by default Concise, yet expressive Type inference Multi paradigm Java ecosystem Functional
24. /** * Map results from a java.sql.ResultSet to a list of objects of another type * @param rs The result set * @param f Function that takes the result set and returns an instance of T * @tparam A The type * @return Sequence of instances of type A */ def mapResults[A](rs: ResultSet, f: ResultSet => A) : Seq[A] = { val array = ArrayBuffer[A]() while (rs.next) { array += f(rs) } array.toSeq }
25. /** * Function to return a cached value, or fetch it per the parameter f * @param key The key to look up in cache, or set against this key if not found * @param f The function that fetches the instance from some persistent store * @tparam A The type * @return None if not found in cache and persistent store, else Some[A] */ def withCache[A](key: String)(f: => Option[A]): Option[A] = { EVCacheGateway.get[A](key) match { case None => val data = f if (!data.isEmpty) EVCacheGateway.set(key, data.get) data case Some(x) => Some(x) } }
26. /** * Get a user given an ID * @param id The User ID * @return None if user is not found, else the user */ def getById(id: Long): Option[User] = { withCache[User](s"User$id") { val query = "select * from users where user_id = ?" DBHelper.queryWith(query, extract, id).headOption } }
27. Scalatra Background Netflix OSS Components Development Deployment/Delivery Open Floor
28. Lightweight HTTP wrapper .war based deployment Swagger support ScalatraSpec
29. /** * Get all the chipsets ordered by name. */ get("/providers", operation(getChipsetProviders)) { Ok( ChipsetManager.getProviders(0,20) ) }
30. ScalaTest Background Netflix OSS Components Development Deployment/Delivery Open Floor
31. Background Netflix OSS Components Development Deployment/Delivery Open Floor Simple Intuitive English-like Rich Promotes BDD
32. it should "return the chipset provider partner orgs" in { val data = ChipsetManager.getProviders(0,20) data.records.isEmpty should be (false) data.records.size should be (20) data.total should be >= 21l }
33. ScalatraSpec Background Netflix OSS Components Development Deployment/Delivery Open Floor Traits to take ScalaTest to the next level Helpers for JSON parsing Plenty of wrappers (body, headers..)
34. addServlet(new ChipsetService, "/*") it should "get a list of chipset providers" in { getWithToken("/providers")(TestUserTokens.testToken) { status should equal (200) body should include (""""start":0,"end":19,"count":20""") } }
35. Swagger Background Netflix OSS Components Development Deployment/Delivery Open Floor
36. Excellent support within Scalatra Type-safe documentation Contract First API Sandbox Machine Readable
37. get("/providers", operation(getAllChipsetProviders)) { //code } case class PartnerList(data: Seq[PartnerOrg], start: Int, end: Int, count: Long, total: Long) def getAllChipsetProviders = authApiOperation[PartnerList]("getChipsetProviders", "Get all chipset providers") .parameter(queryParam[Option[Int]]("start").description("Starting record count, defaults to 0")) .parameter(queryParam[Option[Int]]("count").description("The number of records requested, defaults to 20"))
38. Swagger
40. Deployment Background Netflix OSS Components Development Deployment/Delivery Open Floor Push to dev Jenkins runs dev build, tests, merges to master Jenkins runs master build, makes a .deb Aminator bakes an AMI from the .deb Asgard deploys the AMI in staging cloud
41. Production? Background Netflix OSS Components Development Deployment/Delivery Open Floor
42. lgml-mpandit:nrd-portal-api mpandit$ git status On branch dev Your branch is up-to-date with 'origin/dev'. Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory) modified: api/src/main/scala/com/netflix/nrdportal/http/ChipsetService.scala modified: api/src/test/scala/com/netflix/nrdportal/http/ChipsetServiceSpec.scala modified: api/src/main/scala/com/netflix/nrdportal/manager/ChipsetManager.scala modified: api/src/test/scala/com/netflix/nrdportal/manager/ChipsetManagerSpec.scala no changes added to commit (use "git add" and/or "git commit -a") Check in
43. [info] Passed: Total 1386, Failed 0, Errors 0, Passed 1386 [success] Total time: 1273 s, completed Jul 19, 2014 8:39:54 PM Build step 'Build using sbt' changed build result to SUCCESS Pushing HEAD to branch master of origin repository Returning node parameter for ssh-dynaslave-1291624c Triggering a new build of PPD-NRD-PORTAL-API-MASTER #1172 Notifying upstream projects of job completion Finished: SUCCESS Dev Build
44. Background Netflix OSS Components Development Deployment/Delivery Open Floor Integration Tests Hitting Real DB Refreshed every night with Production
45. Build Artifacts: nrd-portal-api_1.0-h1172.880e187_all.deb 127.71 MB [fingerprint] Changes: Added an endpoint to return a list of chipset provider partner orgs. NFLX-5514 (detail) Started by upstream project PPD-NRD-PORTAL-API-DEV build number 1923 originally caused by: Started by an SCM change Revision: 880e1873fef63d278b3180b49af7434e234dee40 origin/master Master Build
46. Finally!
47. “A node, once deployed, cannot be changed.” Immutable Deployments
48. Easy rollbacks Consistency Predictability Immutable Deployments
49. Summary Background Netflix OSS Components Development Deployment/Delivery Open Floor Bridge the gap between dev and deploy No such thing as too many tests Automate everything Document your APIs Best solutions are implementation agnostic
50. Netflix OSS - http://netflix.github.io/#repo Scalatra - http://www.scalatra.org/ Swagger - https://helloreverb.com/developers/swagger ScalaTest - http://www.scalatest.org/
51. Session Feedback and Rating slideshare.net/lobster1234 speakerdeck.com/mpandit