Scala at Netflix

My talk at Scala Bay Meetup at Netflix about Powering the Partner APIs with Scalatra and Netflix OSS. This talk was delivered on September 9th 2013, at Netflix, Los Gatos.

Transcript

1. Scala at Netflix Scala Bay Meetup Netflix, Sept 9th 2013 Manish Pandit mpandit@netflix.com @lobster1234
2. Agenda Background Netflix OSS Components Development Deployment/Delivery Open Floor
3. Partner Product Engineering Smart devices + Certification = Lots of Device Metadata!
4. Model Firmware Screen Resolution Subtitle Support 3D DRM Remote Control Netflix SDK …
5. Architecture Cassandra HTTP  Layer,  and  Manager  Layer   EVCache   Crowd/SSO RDS Astyanax Netflix OSS Cloud Components
6. Netflix OSS Components https://github.com/netflix http://techblog.netflix.com
7. Simian army
9. Asgard
10. Eureka
11. Karyon
12. Astyanax https://github.com/Netflix/astyanax Astyanax is a Java Client for Cassandra
13. EVCache
14. Development We to code.
15. Test first We write a ton of tests.
16. Test coverage is a measure of confidence.
17. ScalaTest simple intuitive english-like rich promotes BDD
18. Test first Manager Layer No HTTP requests, simple method invocation
19. /** Lets try to build a login endpoint. It should support a method called login(user:String,pass:String) that returns an Option[String]. */ class LoginManagerSpec extends FlatSpec with ShouldMatchers { }
20. /** Lets try to build a login endpoint. It should support a method called login(user:String,pass:String) that returns an Option[String]. */ class LoginManagerSpec extends FlatSpec with ShouldMatchers { it should " Be able to login a valid user and get a token " in { fail() } }
21. /** Lets try to build a login endpoint. It should support a method called login(user:String,pass:String) that returns an Option[String]. */ class LoginManagerSpec extends FlatSpec with ShouldMatchers { it should " Be able to login a valid user and get a token " in { val token = LoginManager.login("someuser", "somepassword") token should not be None } }
22. /** Lets try to build a login endpoint. It should support a method called login(user:String,pass:String) that returns an Option[String]. */ class LoginManagerSpec extends FlatSpec with ShouldMatchers { it should " Be able to login a valid user and get a token " in { val token = LoginManager.login("someuser", "somepassword") token should not be None } it should " Fail to login an invalid user " in { fail } }
23. /** Lets try to build a login endpoint. It should support a method called login(user:String,pass:String) that returns an Option[String]. */ class LoginManagerSpec extends FlatSpec with ShouldMatchers { it should " Be able to login a valid user and get a token " in { val token = LoginManager.login("someuser", "somepassword") token should not be None } it should " Fail to login an invalid user " in { val token = LoginManager.login("fail", "fail") token should be (None) } }
24. HTTP, or the Scalatra Layer
25. Scalatra very simple, Sinatra-inspired framework routes defined with code json4s Swagger ScalatraSpec
26. A Simple API in Scalatra class LoginService extends ScalatraServlet
27. A Simple API in Scalatra class LoginService extends ScalatraServlet { post("/") { } }
28. A Simple API in Scalatra class LoginService extends ScalatraServlet { before() { contentType = formats("json") } post("/") { val token = LoginManager.login(params("user"), params("password")) token match{ case None => Unauthorized(Map("message"->"Login failed")) case Some(x) => Ok(Map("token"->x)) } } }
29. Putting it all together.. class ScalatraBootstrap extends LifeCycle { override def init(context: ServletContext) { context.mount(new LoginService, "/login/*") } } <listener> <listener-class>org.scalatra.servlet.ScalatraListener</listener-class> </listener>
30. $ curl http://localhost:8080/login --form "user=foo&password=bar" {"token":"02055794-1928-11e3-9a3b-f23c91aec05e"}
31. ScalatraSpec traits to take ScalaTest to the next level support for all HTTP methods helpers for JSON parsing plenty of wrappers (body, headers..)
32. class LoginServiceSpec extends ScalatraFlatSpec { }
33. class LoginServiceSpec extends ScalatraFlatSpec { addServlet(classOf[LoginService], "/*") }
34. class LoginServiceSpec extends ScalatraFlatSpec { addServlet(classOf[LoginService], "/*") it should "log in valid users" in { post("/", body = """user=gooduser&password=goodpassword""") { status should equal(200) body should include "token" } } }
35. class LoginServiceSpec extends ScalatraFlatSpec { addServlet(classOf[LoginService], "/*") it should "log in valid users" in { post("/", body = """user=gooduser&password=goodpassword""") { status should equal(200) body should include "token" } } it should "not allow invalid users to log in" in { post("/", body = """user=baduser&password=badpassword""") { status should equal(401) body should include "message" } } }
36. APIs Best Practices Use Proper HTTP Response Codes Set Proper HTTP Headers Break up your data into groups
37. Pop Quiz! Lets do some HTTP response codes..
38. Pop Quiz! What is the response code for an async operation?
39. Pop Quiz! …forbidden?
40. Pop Quiz! …a delete?
41. Git Workflow work on the dev branch write tests leave the rest to Jenkins
42. Git Workflow $ git status # On branch dev # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: src/main/scala/com/netflix/nrdportal/http/DpiService.scala # modified: src/test/scala/com/netflix/nrdportal/http/DpiServiceSpec.scala
43. Automated Code Pushes Push to dev Jenkins runs dev build, tests, merges to master Jenkins runs master build, makes an RPM Aminator bakes an AMI from the RPM asgard deploys the AMI in staging cloud
44. Scala Best Practices Using Options Using Try[A] vs. Exceptions Wrappers Control Abstractions
45. The dreaded null public String willReturnNullForOdds(int x){ if(x%2==0) return "Even"; else return null; }
46. Using Options def willReturnNullForOdds(x: Int): Option[String] = { if (x % 2 == 0) Some("Even") else None }
47. Using Options as Wrappers …where you have to call code that can return null def returnNull(x:Int) = if(x%2 == 0) "Even" else null scala> Option(returnNull(3)) res01: Option[String] = None scala> Option(returnNull(2)) res02: Option[String] = Some(Even)
48. Exceptions? public String willThrowExceptionForOdds(int x){ if(x%2==0) return "Even"; else throw new IllegalArgumentException("Odd Number!"); }
49. Using Try[A] def someFunction(x: Int): Try[String] = { if (x % 2 == 0) Success("Even") else Failure(new IllegalArgumentException("Odd number!")) }
50. Control Abstractions def withAuthenticatedUser(f: (String) => ActionResult) = { getUserNameFromCookie match { case Some(userName) => f(userName) case None => Unauthorized("You are not logged in!") } } def printCurrentUserName = { withAuthenticatedUser { userName => Ok(s"Your username is ${userName}") } }
51. Finally… Avoid writing complex code at all costs – there are better ways to prove your awesomeness!
52. Manish Pandit @lobster1234 mpandit@netflix.com linkedin.com/in/mpandit slideshare.net/lobster1234