Evolution of the Netflix API - QCon SF 2013

Netflix streaming has been growing tremendously, now reaching more than 40M subscribers in over 40 countries. These subscribers are enjoying Netflix on more than 1,000 different device types, ranging from game consoles to set-top boxes, TVs and mobile devices. This is all made possible by a complex service-oriented architecture that handles billions of requests a day from those devices. At the center of that architecture is the Netflix API, which is the front door to the entire system.

This presentation will describe how the Netflix API evolved from a typical one-size-fits-all RESTful API designed to support public developers into a web service platform optimized to handle the diversity and variability of each device and user experience. The presentation will also address the challenges involving operations, deployment, performance, fault-tolerance, and rate of innovation at massive scale.

Presented at QCon SF 2013: http://www.infoq.com/presentations/netflix-api-evolution

Transcript

1. Evolution of the Netflix API Ben Christensen Software Engineer – Edge & Playback Services at Netflix @benjchristensen ! ! ! ! http://techblog.netflix.com/ QCon San Francisco - November 2013
3. More than 40 million Subscribers in 50+ Countries and Territories
4. Netflix accounts for 33% of Peak Downstream Internet Traffic in North America Netflix subscribers are watching more than 1 billion hours a month
5. API traffic has grown from ~20 million/day in 2010 to >2 billion/day 0 500 1000 1500 2000 2010 2011 2012 Today millions of API requests per day
6. Discovery Streaming
7. Netflix API Streaming
9. At the start … 0 500 1000 1500 2000 2008 2009 2010 2011 2012 Today millions of API requests per day
10. 2008 API Launch Targeted 100% at External Developers Open API Netflix Devices
11. 2008 API Launch Targeted 100% at External Developers Purpose A “Thousand Flowers” of 3rd party innovation ! Audience External Developers
12. Oracle Pre-Cloud Architecture API Service Service Service Service
13. 0 500 1000 1500 2000 2008 2009 2010 2011 2012 Today In 2011 … millions of API requests per day
14. 99.9% Netflix Devices Open API Netflix Devices
15. Targeted at Internal Developers Open API Netflix Devices
16. Targeted at Internal Developers Purpose Enable Netflix Experience ! Audience Netflix Device and UI Teams
17. Scale & Resilience ! Performance & Innovation
18. Scale & Resilience ! Performance & Innovation
19. Netflix API Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R
20. Netflix API Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R
25. AWS Availability Zone AWS Availability Zone AWS Availability Zone
30. User Request Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R
31. User Request Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R User request blocked by latency in single network call
32. At high volume all request threads can block in seconds User Request Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R User Request User Request User Request User Request User Request User Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33. ! Dozens of dependencies. ! One going bad takes everything down. ! 99.99%30 = 99.7% uptime ! 0.3% of 1 billion = 3,000,000 failures ! 2+ hours downtime/month ! ! ! Reality is generally worse.
34. CONSTRAINTS Speed of Iteration ! Client Libraries ! Mixed Environment
35. CONSTRAINTS Speed of Iteration ! Client Libraries ! Mixed Environment
36. CONSTRAINTS Speed of Iteration ! Client Libraries ! Mixed Environment
37. CONSTRAINTS Speed of Iteration ! Client Libraries ! Mixed Environment
38. User Request Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R User Request User Request User Request User Request User Request User Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39. cy D dency G ependency J Dependency M Dependency B Dependency E Dependency H Dependency K Dependency N Dependency C Dependency F Dependency I Dependency L Dependency O User Request User Request User Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Network Request - TCP/HTTP, latency, 4xx, 5xx, etc Deserialization - JSON/XML/Thrift/Protobuf/etc Logic - argument validation, caches, metrics, logging, multivariate testing, routing, etc Serialization - URL and/or body generation Logic - validation, decoration, object model, caching, metrics, logging, etc
40. "Timeout guard" daemon prio=10 tid=0x00002aaacd5e5000 nid=0x3aac runnable [0x00002aaac388f000] java.lang.Thread.State: RUNNABLE! at java.net.PlainSocketImpl.socketConnect(Native Method)! at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)! - locked <0x000000055c7e8bd8> (a java.net.SocksSocketImpl)! at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)! at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)! at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)! at java.net.Socket.connect(Socket.java:579)! at java.net.Socket.connect(Socket.java:528)! at java.net.Socket.(Socket.java:425)! at java.net.Socket.(Socket.java:280)! at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)! at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$1.doit(ControllerThreadSocketFactory.java:91)! at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$SocketTask.run(ControllerThreadSocketFactory.java:158) at java.lang.Thread.run(Thread.java:722) [Sat Jun 30 04:01:37 2012] [error] proxy: HTTP: disabled connection for (127.0.0.1) > 80% of requests rejected Median Latency
41. User Request Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R
42. User Request Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R
43. User Request Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R
44. User Request Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R
48. circle color and size represent health and traffic volume
49. 2 minutes of request rate to show relative changes in traffic circle color and size represent health and traffic volume
50. 2 minutes of request rate to show relative changes in traffic circle color and size represent health and traffic volume hosts reporting from cluster
51. last minute latency percentiles 2 minutes of request rate to show relative changes in traffic circle color and size represent health and traffic volume hosts reporting from cluster
52. last minute latency percentiles 2 minutes of request rate to show relative changes in traffic circle color and size represent health and traffic volume hosts reporting from cluster Circuit-breaker status
53. last minute latency percentiles Request rate 2 minutes of request rate to show relative changes in traffic circle color and size represent health and traffic volume hosts reporting from cluster Circuit-breaker status
54. Error percentage of last 10 seconds last minute latency percentiles Request rate 2 minutes of request rate to show relative changes in traffic circle color and size represent health and traffic volume hosts reporting from cluster Error percentage of last 10 seconds Circuit-breaker status
55. last minute latency percentiles Request rate 2 minutes of request rate to show relative changes in traffic circle color and size represent health and traffic volume hosts reporting from cluster Error percentage of last 10 seconds Rolling 10 second counters with 1 second granularity Failures/Exceptions Thread-pool Rejections Thread timeouts Successes Short-circuited (rejected) Circuit-breaker status
56. User Request Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R
58. Zuul Routing Layer Canary vs Baseline Squeeze Production "Coalmine"
59. Zuul Routing Layer Canary vs Baseline Squeeze Production "Coalmine"
60. Zuul Routing Layer Canary vs Baseline Squeeze Production "Coalmine"
61. Zuul Routing Layer Canary vs Baseline Squeeze Production "Coalmine"
62. Zuul Routing Layer Canary vs Baseline Squeeze Production "Coalmine"
63. User Request Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R System Relationship Over Network without Bulkhead
64. Zuul Routing Layer Canary vs Baseline Squeeze Production "Coalmine"
65. Zuul Routing Layer Canary vs Baseline Squeeze Production "Coalmine"
74. Predictive + Reactive Auto-Scaling
83. Scale & Resilience ! Performance & Innovation
84. Netflix API Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R
85. Netflix API Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R
86. One Size Fits All RESTful API 0 500 1000 1500 2000 2010 2011 2012 Today millions of API requests per day
87. One Size Fits All RESTful API 0 500 1000 1500 2000 2010 2011 2012 Today millions of API requests per day
88. One Size Fits All RESTful API 0 500 1000 1500 2000 2010 2011 2012 Today millions of API requests per day 1000+ Devices
89. We wanted to re-architecture our call patterns ...
90. ... to collapse network traffic into coarse API calls ... nested, conditional, concurrent execution
91. ... and we wanted to allow anybody to create endpoints, not just the “API Team”
93. Concurrency without each engineer reading and re-reading this → ! (awesome book ... everybody isn’t going to - or should have to - read it though, that’s the point)
94. Owner of api should retain control of concurrency behavior.
95. What if the implementation needs to change from synchronous to asynchronous? ! How should the client execute that method without blocking? spawn a thread? public  Data  getData(); Owner of api should retain control of concurrency behavior.
96. public  void  getData(Callback<T>  c);   ! public  Future<T>  getData();   ! public  Future<List<Future<T>>>  getData();   ! ! other options ... ? public  Data  getData();
97. Iterable pull Observable push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted() Reactive Programming with Rx
98. Iterable pull Observable push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted() (Functional) Reactive Programming with RxJava
99. Iterable pull Observable push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted() (Functional) Reactive Programming with RxJava
100. Iterable pull Observable push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted() (Functional) Reactive Programming with RxJava
101. Iterable pull Observable push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted() !  //  Iterable<String>      //  that  contains  75  Strings    getDataFromLocalMemory()      .skip(10)      .take(5)      .map({  s  -­‐>          return  s  +  "_transformed"})      .forEach(            {  println  "next  =>  "  +  it})   !  //  Observable<String>      //  that  emits  75  Strings    getDataFromNetwork()      .skip(10)      .take(5)      .map({  s  -­‐>          return  s  +  "_transformed"})      .subscribe(            {  println  "onNext  =>  "  +  it})  
102. Iterable pull Observable push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted() !  //  Iterable<String>      //  that  contains  75  Strings    getDataFromLocalMemory()      .skip(10)      .take(5)      .map({  s  -­‐>          return  s  +  "_transformed"})      .forEach(            {  println  "onNext  =>  "  +  it})   !  //  Observable<String>      //  that  emits  75  Strings    getDataFromNetwork()      .skip(10)      .take(5)      .map({  s  -­‐>          return  s  +  "_transformed"})      .subscribe(            {  println  "onNext  =>  "  +  it})  
103. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData()
104. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() String s = getData(args); if (s.equals(x)) { // do something } else { // do something else }
105. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Iterable<String> values = getData(args); for (String s : values) { if (s.equals(x)) { // do something } else { // do something else } }
106. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); if (s.get().equals(x)) { // do something } else { // do something else }
107. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); if (s.get().equals(x)) { // do something } else { // do something else }
108. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); Futures.addCallback(s, new FutureCallback<String> { public void onSuccess(String s) { if (s.equals(x)) { // do something } else { // do something else } } }, executor);
109. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); Futures.addCallback(s, new FutureCallback<String> { public void onSuccess(String s) { if (s.equals(x)) { // do something } else { // do something else } } }, executor);
110. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); Futures.addCallback(s, new FutureCallback<String> { public void onSuccess(String s) { if (s.equals(x)) { // do something } else { // do something else } } }, executor);
111. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() CompletableFuture<String> s = getData(args); s.thenApply((v) -> { if (v.equals(x)) { // do something } else { // do something else } });
112. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() CompletableFuture<String> s = getData(args); s.thenApply((v) -> { if (v.equals(x)) { // do something } else { // do something else } });
113. Future<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } }); Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData()
114. Future<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } }); Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData()
115. Future<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } }); Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData()
116. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Observable<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } });
117. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Observable<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } });
118. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Observable<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } });
119. class  VideoService  {        def  VideoList  getPersonalizedListOfMovies(userId);        def  VideoBookmark  getBookmark(userId,  videoId);        def  VideoRating  getRating(userId,  videoId);        def  VideoMetadata  getMetadata(videoId);   } class  VideoService  {        def  Observable<VideoList>  getPersonalizedListOfMovies(userId);        def  Observable<VideoBookmark>  getBookmark(userId,  videoId);        def  Observable<VideoRating>  getRating(userId,  videoId);        def  Observable<VideoMetadata>  getMetadata(videoId);   } ... create an observable api: instead of a blocking api ...
120. “a library for composing asynchronous and event-based programs using observable sequences for the Java VM” A Java port of Rx (Reactive Extensions) https://rx.codeplex.com (.Net and Javascript by Microsoft) RxJava http://github.com/Netflix/RxJava
121. client code treats all interactions with the api as asynchronous ! ! the api implementation chooses whether something is blocking or non-blocking and what resources it uses
128. Netflix API Device Server Optimize for Each Device. Leverage the Server
129. Code
130. SDK for API Web Service Development
131. SDK for API Web Service Development
132. SDK for API Web Service Development
133. api.servletResponse.writer.print("Hello")
134. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
135. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
136. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
137. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
138. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
139. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
140. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } }
141. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
142. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
143. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
144. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
145. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
146. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
147. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
148. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
149. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
150. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
151. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
152. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
153. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
154. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
155. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:38:51 UTC 2013", "revision": 1, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
156. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:38:51 UTC 2013", "revision": 1, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
157. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
158. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" } $ activateScript.py -e TEST --revision 2 /test/hello { "active": true, "allocationPercentage": 100, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "previousRevision": null, "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
159. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" } $ activateScript.py -e TEST --revision 2 /test/hello { "active": true, "allocationPercentage": 100, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "previousRevision": null, "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
160. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" } $ activateScript.py -e TEST --revision 2 /test/hello { "active": true, "allocationPercentage": 100, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "previousRevision": null, "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
161. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" } $ activateScript.py -e TEST --revision 2 /test/hello { "active": true, "allocationPercentage": 100, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "previousRevision": null, "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
162. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" } $ activateScript.py -e TEST --revision 3 /test/hello { "active": true, "allocationPercentage": 100, "creationDate": "Mon Nov 11 05:42:05 UTC 2013", "previousRevision": 2, "revision": 3, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
163. Future
164. millions of API requests per day 0 500 1000 1500 2000 2008 2009 2010 2011 2012 Today
172. /ps3/home Dependency F 10 Threads Dependency G 10 Threads Dependency H 10 Threads Dependency I 5 Threads Dependency J 8 Threads Dependency A 10 Threads Dependency B 8 Threads Dependency C 10 Threads Dependency D 15 Threads Dependency E 5 Threads Dependency K 15 Threads Dependency L 4 Threads Dependency M 5 Threads Dependency N 10 Threads Dependency O 10 Threads Dependency P 10 Threads Dependency Q 8 Threads Dependency R 10 Threads Dependency S 8 Threads Dependency T 10 Threads /android/home /tv/home Functional Reactive Dynamic Endpoints Asynchronous Java API
173. /ps3/home Dependency F 10 Threads Dependency G 10 Threads Dependency H 10 Threads Dependency I 5 Threads Dependency J 8 Threads Dependency A 10 Threads Dependency B 8 Threads Dependency C 10 Threads Dependency D 15 Threads Dependency E 5 Threads Dependency K 15 Threads Dependency L 4 Threads Dependency M 5 Threads Dependency N 10 Threads Dependency O 10 Threads Dependency P 10 Threads Dependency Q 8 Threads Dependency R 10 Threads Dependency S 8 Threads Dependency T 10 Threads /android/home /tv/home Functional Reactive Dynamic Endpoints Asynchronous Java API Hystrix fault-isolation layer
174. + Observable<User>  u  =  new  GetUserCommand(id).observe();   Observable<Geo>  g  =  new  GetGeoCommand(request).observe();   ! Observable.zip(u,  g,  {user,  geo  -­‐>                    return  [username:  user.getUsername(),                                    currentLocation:  geo.getCounty()]         }) RxJava in Hystrix 1.3+ https://github.com/Netflix/Hystrix
177. Optimizing the Netflix API http://techblog.netflix.com/2013/01/optimizing-netflix-api.html ! Why REST Keeps Me Up At Night http://blog.programmableweb.com/2012/05/15/why-rest-keeps-me-up-at-night/ ! Functional Reactive in the Netflix API with RxJava http://techblog.netflix.com/2013/02/rxjava-netflix-api.html ! Application Resilience in a Service-oriented Architecture http://programming.oreilly.com/2013/06/application-resilience-in-a-service-oriented-architecture.html ! Netflix API on Tech Blog http://techblog.netflix.com/search/label/api jobs.netflix.com Ben Christensen @benjchristensen http://www.linkedin.com/in/benjchristensen