Evolution of the Netflix API - QCon SF 2013

Netflix streaming has been growing tremendously, now reaching more than 40M subscribers in over 40 countries. These subscribers are enjoying Netflix on more than 1,000 different device types, ranging from game consoles to set-top boxes, TVs and mobile devices. This is all made possible by a complex service-oriented architecture that handles billions of requests a day from those devices. At the center of that architecture is the Netflix API, which is the front door to the entire system.

This presentation will describe how the Netflix API evolved from a typical one-size-fits-all RESTful API designed to support public developers into a web service platform optimized to handle the diversity and variability of each device and user experience. The presentation will also address the challenges involving operations, deployment, performance, fault-tolerance, and rate of innovation at massive scale.

Presented at QCon SF 2013: http://www.infoq.com/presentations/netflix-api-evolution

Transcript

1. Evolution of the Netflix API
   Ben Christensen
   Software Engineer – Edge & Playback Services at Netflix
   @benjchristensen
   !
   !
   !
   !
   http://techblog.netflix.com/
   QCon San Francisco - November 2013
   

   View Slide

2. View Slide

3. More than 40 million Subscribers
   in 50+ Countries and Territories
   

   View Slide

4. Netflix accounts for 33% of Peak Downstream
   Internet Traffic in North America
   Netflix subscribers are watching
   more than 1 billion hours a month
   

   View Slide

5. API traffic has grown from
   ~20 million/day in 2010 to >2 billion/day
   0
   500
   1000
   1500
   2000
   2010 2011 2012 Today
   millions of API requests per day
   

   View Slide

6. Discovery Streaming
   

   View Slide

7. Netflix API Streaming
   

   View Slide

8. View Slide

9. At the start …
   0
   500
   1000
   1500
   2000
   2008 2009 2010 2011 2012 Today
   millions of API requests per day
   

   View Slide

10. 2008 API Launch
    Targeted 100% at External Developers
    Open API Netflix Devices
    

    View Slide

11. 2008 API Launch
    Targeted 100% at External Developers
    Purpose
    A “Thousand Flowers” of
    3rd party innovation
    !
    Audience
    External Developers
    

    View Slide

12. Oracle
    Pre-Cloud Architecture
    API
    Service Service Service Service
    

    View Slide

13. 0
    500
    1000
    1500
    2000
    2008 2009 2010 2011 2012 Today
    In 2011 …
    millions of API requests per day
    

    View Slide

14. 99.9% Netflix Devices
    Open API Netflix Devices
    

    View Slide

15. Targeted at Internal Developers
    Open API Netflix Devices
    

    View Slide

16. Targeted at Internal Developers
    Purpose
    Enable Netflix Experience
    !
    Audience
    Netflix Device and UI Teams
    

    View Slide

17. Scale & Resilience
    !
    Performance & Innovation
    

    View Slide

18. Scale & Resilience
    !
    Performance & Innovation
    

    View Slide

19. Netflix API
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    

    View Slide

20. Netflix API
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    

    View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. AWS
    Availability Zone
    AWS
    Availability Zone
    AWS
    Availability Zone
    

    View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. User Request
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    

    View Slide

31. User Request
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    User request
    blocked by
    latency in
    single
    network call
    

    View Slide

32. At high
    volume
    all request
    threads can
    block in
    seconds
    User Request
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    User Request
    User Request
    User Request
    User Request
    User Request
    User Request
    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
    

    View Slide

33. !
    Dozens of dependencies.
    !
    One going bad takes everything down.
    !
    99.99%30 = 99.7% uptime
    !
    0.3% of 1 billion = 3,000,000 failures
    !
    2+ hours downtime/month
    !
    !
    !
    Reality is generally worse.
    

    View Slide

34. CONSTRAINTS
    Speed of Iteration
    !
    Client Libraries
    !
    Mixed Environment
    

    View Slide

35. CONSTRAINTS
    Speed of Iteration
    !
    Client Libraries
    !
    Mixed Environment
    

    View Slide

36. CONSTRAINTS
    Speed of Iteration
    !
    Client Libraries
    !
    Mixed Environment
    

    View Slide

37. CONSTRAINTS
    Speed of Iteration
    !
    Client Libraries
    !
    Mixed Environment
    

    View Slide

38. User Request
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    User Request
    User Request
    User Request
    User Request
    User Request
    User Request
    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
    

    View Slide

39. cy D
    dency G
    ependency J
    Dependency M
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    User Request
    User Request
    User Request
    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
    Network Request - TCP/HTTP, latency, 4xx, 5xx, etc
    Deserialization - JSON/XML/Thrift/Protobuf/etc
    Logic - argument validation, caches, metrics, logging,
    multivariate testing, routing, etc
    Serialization - URL and/or body generation
    Logic - validation, decoration, object model, caching,
    metrics, logging, etc
    

    View Slide

40. "Timeout guard" daemon prio=10 tid=0x00002aaacd5e5000 nid=0x3aac runnable [0x00002aaac388f000] java.lang.Thread.State: RUNNABLE!
    at java.net.PlainSocketImpl.socketConnect(Native Method)!
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)!
    - locked <0x000000055c7e8bd8> (a java.net.SocksSocketImpl)!
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)!
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)!
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)!
    at java.net.Socket.connect(Socket.java:579)!
    at java.net.Socket.connect(Socket.java:528)!
    at java.net.Socket.(Socket.java:425)!
    at java.net.Socket.(Socket.java:280)!
    at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)!
    at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$1.doit(ControllerThreadSocketFactory.java:91)!
    at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$SocketTask.run(ControllerThreadSocketFactory.java:158) at java.lang.Thread.run(Thread.java:722)
    [Sat Jun 30 04:01:37 2012] [error] proxy: HTTP: disabled connection for (127.0.0.1)
    > 80% of requests rejected
    Median
    Latency
    

    View Slide

41. User Request
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    

    View Slide

42. User Request
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    

    View Slide

43. User Request
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    

    View Slide

44. User Request
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    

    View Slide

45. View Slide

46. View Slide

47. View Slide

48. circle color and size represent
    health and traffic volume
    

    View Slide

49. 2 minutes of request rate to
    show relative changes in traffic
    circle color and size represent
    health and traffic volume
    

    View Slide

50. 2 minutes of request rate to
    show relative changes in traffic
    circle color and size represent
    health and traffic volume
    hosts reporting from cluster
    

    View Slide

51. last minute latency percentiles
    2 minutes of request rate to
    show relative changes in traffic
    circle color and size represent
    health and traffic volume
    hosts reporting from cluster
    

    View Slide

52. last minute latency percentiles
    2 minutes of request rate to
    show relative changes in traffic
    circle color and size represent
    health and traffic volume
    hosts reporting from cluster
    Circuit-breaker
    status
    

    View Slide

53. last minute latency percentiles
    Request rate
    2 minutes of request rate to
    show relative changes in traffic
    circle color and size represent
    health and traffic volume
    hosts reporting from cluster
    Circuit-breaker
    status
    

    View Slide

54. Error percentage of
    last 10 seconds
    last minute latency percentiles
    Request rate
    2 minutes of request rate to
    show relative changes in traffic
    circle color and size represent
    health and traffic volume
    hosts reporting from cluster
    Error percentage of
    last 10 seconds
    Circuit-breaker
    status
    

    View Slide

55. last minute latency percentiles
    Request rate
    2 minutes of request rate to
    show relative changes in traffic
    circle color and size represent
    health and traffic volume
    hosts reporting from cluster
    Error percentage of
    last 10 seconds
    Rolling 10 second counters
    with 1 second granularity
    Failures/Exceptions
    Thread-pool Rejections
    Thread timeouts
    Successes
    Short-circuited (rejected)
    Circuit-breaker
    status
    

    View Slide

56. User Request
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    

    View Slide

57. View Slide

58. Zuul Routing Layer
    Canary vs Baseline
    Squeeze
    Production
    "Coalmine"
    

    View Slide

59. Zuul Routing Layer
    Canary vs Baseline
    Squeeze
    Production
    "Coalmine"
    

    View Slide

60. Zuul Routing Layer
    Canary vs Baseline
    Squeeze
    Production
    "Coalmine"
    

    View Slide

61. Zuul Routing Layer
    Canary vs Baseline
    Squeeze
    Production
    "Coalmine"
    

    View Slide

62. Zuul Routing Layer
    Canary vs Baseline
    Squeeze
    Production
    "Coalmine"
    

    View Slide

63. User Request
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    System
    Relationship
    Over
    Network
    without
    Bulkhead
    

    View Slide

64. Zuul Routing Layer
    Canary vs Baseline
    Squeeze
    Production
    "Coalmine"
    

    View Slide

65. Zuul Routing Layer
    Canary vs Baseline
    Squeeze
    Production
    "Coalmine"
    

    View Slide

66. View Slide

67. View Slide

68. View Slide

69. View Slide

70. View Slide

71. View Slide

72. View Slide

73. View Slide

74. Predictive + Reactive Auto-Scaling
    

    View Slide

75. View Slide

76. View Slide

77. View Slide

78. View Slide

79. View Slide

80. View Slide

81. View Slide

82. View Slide

83. Scale & Resilience
    !
    Performance & Innovation
    

    View Slide

84. Netflix API
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    

    View Slide

85. Netflix API
    Dependency A
    Dependency D
    Dependency G
    Dependency J
    Dependency M
    Dependency P
    Dependency B
    Dependency E
    Dependency H
    Dependency K
    Dependency N
    Dependency Q
    Dependency C
    Dependency F
    Dependency I
    Dependency L
    Dependency O
    Dependency R
    

    View Slide

86. One Size Fits All RESTful API
    0
    500
    1000
    1500
    2000
    2010 2011 2012 Today
    millions of API requests per day
    

    View Slide

87. One Size Fits All RESTful API
    0
    500
    1000
    1500
    2000
    2010 2011 2012 Today
    millions of API requests per day
    

    View Slide

88. One Size Fits All RESTful API
    0
    500
    1000
    1500
    2000
    2010 2011 2012 Today
    millions of API requests per day
    1000+ Devices
    

    View Slide

89. We wanted to re-architecture our call patterns ...
    

    View Slide

90. ... to collapse network traffic into coarse API calls ...
    nested, conditional, concurrent execution
    

    View Slide

91. ... and we wanted to allow
    anybody to create
    endpoints, not just the
    “API Team”
    

    View Slide

92. View Slide

93. Concurrency without each engineer
    reading and re-reading this →
    !
    (awesome book ... everybody isn’t going to - or
    should have to - read it though, that’s the point)
    

    View Slide

94. Owner of api should retain control
    of concurrency behavior.
    

    View Slide

95. What if the implementation needs to change
    from synchronous to asynchronous?
    !
    How should the client execute that method
    without blocking? spawn a thread?
    public  Data  getData();
    Owner of api should retain control
    of concurrency behavior.
    

    View Slide

96. public  void  getData(Callback  c);  
    !
    public  Future  getData();  
    !
    public  Future>>  getData();  
    !
    !
    other options ... ?
    public  Data  getData();
    

    View Slide

97. Iterable
    pull
    Observable
    push
    T next()
    throws Exception
    returns;
    onNext(T)
    onError(Exception)
    onCompleted()
    Reactive Programming with Rx
    

    View Slide

98. Iterable
    pull
    Observable
    push
    T next()
    throws Exception
    returns;
    onNext(T)
    onError(Exception)
    onCompleted()
    (Functional) Reactive Programming with RxJava
    

    View Slide

99. Iterable
    pull
    Observable
    push
    T next()
    throws Exception
    returns;
    onNext(T)
    onError(Exception)
    onCompleted()
    (Functional) Reactive Programming with RxJava
    

    View Slide

100. Iterable
     pull
     Observable
     push
     T next()
     throws Exception
     returns;
     onNext(T)
     onError(Exception)
     onCompleted()
     (Functional) Reactive Programming with RxJava
     

     View Slide

101. Iterable
     pull
     Observable
     push
     T next()
     throws Exception
     returns;
     onNext(T)
     onError(Exception)
     onCompleted()
     !
      //  Iterable    
      //  that  contains  75  Strings  
      getDataFromLocalMemory()  
        .skip(10)  
        .take(5)  
        .map({  s  -­‐>    
          return  s  +  "_transformed"})  
        .forEach(  
              {  println  "next  =>  "  +  it})  
     !
      //  Observable    
      //  that  emits  75  Strings  
      getDataFromNetwork()  
        .skip(10)  
        .take(5)  
        .map({  s  -­‐>    
          return  s  +  "_transformed"})  
        .subscribe(  
              {  println  "onNext  =>  "  +  it})  
     

     View Slide

102. Iterable
     pull
     Observable
     push
     T next()
     throws Exception
     returns;
     onNext(T)
     onError(Exception)
     onCompleted()
     !
      //  Iterable    
      //  that  contains  75  Strings  
      getDataFromLocalMemory()  
        .skip(10)  
        .take(5)  
        .map({  s  -­‐>    
          return  s  +  "_transformed"})  
        .forEach(  
              {  println  "onNext  =>  "  +  it})  
     !
      //  Observable    
      //  that  emits  75  Strings  
      getDataFromNetwork()  
        .skip(10)  
        .take(5)  
        .map({  s  -­‐>    
          return  s  +  "_transformed"})  
        .subscribe(  
              {  println  "onNext  =>  "  +  it})  
     

     View Slide

103. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     

     View Slide

104. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     String s = getData(args);
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     

     View Slide

105. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     Iterable values = getData(args);
     for (String s : values) {
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     }
     

     View Slide

106. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     Future s = getData(args);
     if (s.get().equals(x)) {
     // do something
     } else {
     // do something else
     }
     

     View Slide

107. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     Future s = getData(args);
     if (s.get().equals(x)) {
     // do something
     } else {
     // do something else
     }
     

     View Slide

108. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     Future s = getData(args);
     Futures.addCallback(s,
     new FutureCallback {
     public void onSuccess(String s) {
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     }
     }, executor);
     

     View Slide

109. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     Future s = getData(args);
     Futures.addCallback(s,
     new FutureCallback {
     public void onSuccess(String s) {
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     }
     }, executor);
     

     View Slide

110. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     Future s = getData(args);
     Futures.addCallback(s,
     new FutureCallback {
     public void onSuccess(String s) {
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     }
     }, executor);
     

     View Slide

111. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     CompletableFuture s = getData(args);
     s.thenApply((v) -> {
     if (v.equals(x)) {
     // do something
     } else {
     // do something else
     }
     });
     

     View Slide

112. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     CompletableFuture s = getData(args);
     s.thenApply((v) -> {
     if (v.equals(x)) {
     // do something
     } else {
     // do something else
     }
     });
     

     View Slide

113. Future s = getData(args);
     s.map({ s ->
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     });
     Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     

     View Slide

114. Future s = getData(args);
     s.map({ s ->
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     });
     Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     

     View Slide

115. Future s = getData(args);
     s.map({ s ->
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     });
     Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     

     View Slide

116. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     Observable s = getData(args);
     s.map({ s ->
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     });
     

     View Slide

117. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     Observable s = getData(args);
     s.map({ s ->
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     });
     

     View Slide

118. Single Multiple
     Sync T getData() Iterable getData()
     Async Future getData() Observable getData()
     Observable s = getData(args);
     s.map({ s ->
     if (s.equals(x)) {
     // do something
     } else {
     // do something else
     }
     });
     

     View Slide

119. class  VideoService  {  
          def  VideoList  getPersonalizedListOfMovies(userId);  
          def  VideoBookmark  getBookmark(userId,  videoId);  
          def  VideoRating  getRating(userId,  videoId);  
          def  VideoMetadata  getMetadata(videoId);  
     }
     class  VideoService  {  
          def  Observable  getPersonalizedListOfMovies(userId);  
          def  Observable  getBookmark(userId,  videoId);  
          def  Observable  getRating(userId,  videoId);  
          def  Observable  getMetadata(videoId);  
     }
     ... create an observable api:
     instead of a blocking api ...
     

     View Slide

120. “a library for composing
     asynchronous and event-based
     programs using observable
     sequences for the Java VM”
     A Java port of Rx (Reactive Extensions)
     https://rx.codeplex.com (.Net and Javascript by Microsoft)
     RxJava
     http://github.com/Netflix/RxJava
     

     View Slide

121. client code treats all interactions
     with the api as asynchronous
     !
     !
     the api implementation chooses
     whether something is
     blocking or non-blocking
     and
     what resources it uses
     

     View Slide

122. View Slide

123. View Slide

124. View Slide

125. View Slide

126. View Slide

127. View Slide

128. Netflix API
     Device
     Server
     Optimize for Each Device. Leverage the Server
     

     View Slide

129. Code
     

     View Slide

130. SDK for API Web Service Development
     

     View Slide

131. SDK for API Web Service Development
     

     View Slide

132. SDK for API Web Service Development
     

     View Slide

133. api.servletResponse.writer.print("Hello")
     

     View Slide

134. api.servletResponse.writer.print("Hello")
     $runScript.py -e PROD sample.groovy
     Hello
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     [0ms] INFO: User => NULL
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     

     View Slide

135. api.servletResponse.writer.print("Hello")
     $runScript.py -e PROD sample.groovy
     Hello
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     [0ms] INFO: User => NULL
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     

     View Slide

136. api.servletResponse.writer.print("Hello")
     $runScript.py -e PROD sample.groovy
     Hello
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     [0ms] INFO: User => NULL
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     

     View Slide

137. api.servletResponse.writer.print("Hello")
     $runScript.py -e PROD sample.groovy
     Hello
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     [0ms] INFO: User => NULL
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     

     View Slide

138. api.servletResponse.writer.print("Hello")
     $runScript.py -e PROD sample.groovy
     Hello
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     [0ms] INFO: User => NULL
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     

     View Slide

139. api.servletResponse.writer.print("Hello")
     $runScript.py -e PROD sample.groovy
     Hello
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     [0ms] INFO: User => NULL
     ################################################################################################
     ####### Script Logger (enabled via debug=true request parameter)
     ################################################################################################
     

     View Slide

140. public class HelloEndpoint extends APIEndpoint {
     @Override
     public void execute(APIRequest api) throws Throwable {
     // get a request parameter from servlet request
     String alias = api.servletRequest.getParameter("name");
     !
     // set content type and write something to servlet response
     api.servletResponse.setContentType("application/json");
     api.servletResponse.writer.print(
     JsonUtility.toJson(
     [ alias : alias, name : api.user.firstName ]
     )
     )
     !
     }
     }
     

     View Slide

141. public class HelloEndpoint extends APIEndpoint {
     @Override
     public void execute(APIRequest api) throws Throwable {
     // get a request parameter from servlet request
     String alias = api.servletRequest.getParameter("name");
     !
     // set content type and write something to servlet response
     api.servletResponse.setContentType("application/json");
     api.servletResponse.writer.print(
     JsonUtility.toJson(
     [ alias : alias, name : api.user.firstName ]
     )
     )
     !
     }
     }
     runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
     

     View Slide

142. public class HelloEndpoint extends APIEndpoint {
     @Override
     public void execute(APIRequest api) throws Throwable {
     // get a request parameter from servlet request
     String alias = api.servletRequest.getParameter("name");
     !
     // set content type and write something to servlet response
     api.servletResponse.setContentType("application/json");
     api.servletResponse.writer.print(
     JsonUtility.toJson(
     [ alias : alias, name : api.user.firstName ]
     )
     )
     !
     }
     }
     runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
     {"alias":"bob","name":"Old"}
     

     View Slide

143. public class HelloEndpoint extends APIEndpoint {
     @Override
     public void execute(APIRequest api) throws Throwable {
     // get a request parameter from servlet request
     String alias = api.servletRequest.getParameter("name");
     !
     // set content type and write something to servlet response
     api.servletResponse.setContentType("application/json");
     api.servletResponse.writer.print(
     JsonUtility.toJson(
     [ alias : alias, name : api.user.firstName ]
     )
     )
     !
     }
     }
     runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
     {"alias":"bob","name":"Old"}
     

     View Slide

144. public class HelloEndpoint extends APIEndpoint {
     @Override
     public void execute(APIRequest api) throws Throwable {
     // get a request parameter from servlet request
     String alias = api.servletRequest.getParameter("name");
     !
     // set content type and write something to servlet response
     api.servletResponse.setContentType("application/json");
     api.servletResponse.writer.print(
     JsonUtility.toJson(
     [ alias : alias, name : api.user.firstName ]
     )
     )
     !
     }
     }
     runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
     {"alias":"bob","name":"Old"}
     

     View Slide

145. public class HelloEndpoint extends APIEndpoint {
     @Override
     public void execute(APIRequest api) throws Throwable {
     // get a request parameter from servlet request
     String alias = api.servletRequest.getParameter("name");
     !
     // set content type and write something to servlet response
     api.servletResponse.setContentType("application/json");
     api.servletResponse.writer.print(
     JsonUtility.toJson(
     [ alias : alias, name : api.user.firstName ]
     )
     )
     !
     }
     }
     runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
     {"alias":"bob","name":"Old"}
     

     View Slide

146. public class HelloEndpoint extends APIEndpoint {
     @Override
     public void execute(APIRequest api) throws Throwable {
     // get a request parameter from servlet request
     String alias = api.servletRequest.getParameter("name");
     !
     // set content type and write something to servlet response
     api.servletResponse.setContentType("application/json");
     api.servletResponse.writer.print(
     JsonUtility.toJson(
     [ alias : alias, name : api.user.firstName ]
     )
     )
     !
     }
     }
     runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
     {"alias":"bob","name":"Old"}
     

     View Slide

147. public class HelloEndpoint extends APIEndpoint {
     @Override
     public void execute(APIRequest api) throws Throwable {
     // get a request parameter from servlet request
     String alias = api.servletRequest.getParameter("name");
     !
     // set content type and write something to servlet response
     api.servletResponse.setContentType("application/json");
     api.servletResponse.writer.print(
     JsonUtility.toJson(
     [ alias : alias, name : api.user.firstName ]
     )
     )
     !
     }
     }
     runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
     {"alias":"bob","name":"Old"}
     

     View Slide

148. public Observable getVideoSummary(APIVideo video) {
     // get id, title
     def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
     !
     // get bookmark
     def bookmarkObservable = getBookmark(video)
     !
     // get artwork
     def artworkObservable = getArtworkImageUrl(video)
     !
     // merge them and accumulate into the seed.
     return Observable.merge(bookmarkObservable, artworkObservable)
     .reduce(seed, { aggregate, current -> aggregate << current})
     .map({ [ (video.id.toString()) : it]})
     }
     

     View Slide

149. public Observable getVideoSummary(APIVideo video) {
     // get id, title
     def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
     !
     // get bookmark
     def bookmarkObservable = getBookmark(video)
     !
     // get artwork
     def artworkObservable = getArtworkImageUrl(video)
     !
     // merge them and accumulate into the seed.
     return Observable.merge(bookmarkObservable, artworkObservable)
     .reduce(seed, { aggregate, current -> aggregate << current})
     .map({ [ (video.id.toString()) : it]})
     }
     

     View Slide

150. public Observable getVideoSummary(APIVideo video) {
     // get id, title
     def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
     !
     // get bookmark
     def bookmarkObservable = getBookmark(video)
     !
     // get artwork
     def artworkObservable = getArtworkImageUrl(video)
     !
     // merge them and accumulate into the seed.
     return Observable.merge(bookmarkObservable, artworkObservable)
     .reduce(seed, { aggregate, current -> aggregate << current})
     .map({ [ (video.id.toString()) : it]})
     }
     

     View Slide

151. public Observable getVideoSummary(APIVideo video) {
     // get id, title
     def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
     !
     // get bookmark
     def bookmarkObservable = getBookmark(video)
     !
     // get artwork
     def artworkObservable = getArtworkImageUrl(video)
     !
     // merge them and accumulate into the seed.
     return Observable.merge(bookmarkObservable, artworkObservable)
     .reduce(seed, { aggregate, current -> aggregate << current})
     .map({ [ (video.id.toString()) : it]})
     }
     

     View Slide

152. public Observable getVideoSummary(APIVideo video) {
     // get id, title
     def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
     !
     // get bookmark
     def bookmarkObservable = getBookmark(video)
     !
     // get artwork
     def artworkObservable = getArtworkImageUrl(video)
     !
     // merge them and accumulate into the seed.
     return Observable.merge(bookmarkObservable, artworkObservable)
     .reduce(seed, { aggregate, current -> aggregate << current})
     .map({ [ (video.id.toString()) : it]})
     }
     

     View Slide

153. public Observable getVideoSummary(APIVideo video) {
     // get id, title
     def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
     !
     // get bookmark
     def bookmarkObservable = getBookmark(video)
     !
     // get artwork
     def artworkObservable = getArtworkImageUrl(video)
     !
     // merge them and accumulate into the seed.
     return Observable.merge(bookmarkObservable, artworkObservable)
     .reduce(seed, { aggregate, current -> aggregate << current})
     .map({ [ (video.id.toString()) : it]})
     }
     

     View Slide

154. public Observable getVideoSummary(APIVideo video) {
     // get id, title
     def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
     !
     // get bookmark
     def bookmarkObservable = getBookmark(video)
     !
     // get artwork
     def artworkObservable = getArtworkImageUrl(video)
     !
     // merge them and accumulate into the seed.
     return Observable.merge(bookmarkObservable, artworkObservable)
     .reduce(seed, { aggregate, current -> aggregate << current})
     .map({ [ (video.id.toString()) : it]})
     }
     

     View Slide

155. $ uploadScript.py -e TEST /test/hello sample.groovy
     {
     "active": false,
     "allocationPercentage": 0,
     "creationDate": "Mon Nov 11 05:38:51 UTC 2013",
     "revision": 1,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     

     View Slide

156. $ uploadScript.py -e TEST /test/hello sample.groovy
     {
     "active": false,
     "allocationPercentage": 0,
     "creationDate": "Mon Nov 11 05:38:51 UTC 2013",
     "revision": 1,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     

     View Slide

157. $ uploadScript.py -e TEST /test/hello sample.groovy
     {
     "active": false,
     "allocationPercentage": 0,
     "creationDate": "Mon Nov 11 05:40:26 UTC 2013",
     "revision": 2,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     

     View Slide

158. $ uploadScript.py -e TEST /test/hello sample.groovy
     {
     "active": false,
     "allocationPercentage": 0,
     "creationDate": "Mon Nov 11 05:40:26 UTC 2013",
     "revision": 2,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     $ activateScript.py -e TEST --revision 2 /test/hello
     {
     "active": true,
     "allocationPercentage": 100,
     "creationDate": "Mon Nov 11 05:40:26 UTC 2013",
     "previousRevision": null,
     "revision": 2,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     

     View Slide

159. $ uploadScript.py -e TEST /test/hello sample.groovy
     {
     "active": false,
     "allocationPercentage": 0,
     "creationDate": "Mon Nov 11 05:40:26 UTC 2013",
     "revision": 2,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     $ activateScript.py -e TEST --revision 2 /test/hello
     {
     "active": true,
     "allocationPercentage": 100,
     "creationDate": "Mon Nov 11 05:40:26 UTC 2013",
     "previousRevision": null,
     "revision": 2,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     

     View Slide

160. $ uploadScript.py -e TEST /test/hello sample.groovy
     {
     "active": false,
     "allocationPercentage": 0,
     "creationDate": "Mon Nov 11 05:40:26 UTC 2013",
     "revision": 2,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     $ activateScript.py -e TEST --revision 2 /test/hello
     {
     "active": true,
     "allocationPercentage": 100,
     "creationDate": "Mon Nov 11 05:40:26 UTC 2013",
     "previousRevision": null,
     "revision": 2,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     

     View Slide

161. $ uploadScript.py -e TEST /test/hello sample.groovy
     {
     "active": false,
     "allocationPercentage": 0,
     "creationDate": "Mon Nov 11 05:40:26 UTC 2013",
     "revision": 2,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     $ activateScript.py -e TEST --revision 2 /test/hello
     {
     "active": true,
     "allocationPercentage": 100,
     "creationDate": "Mon Nov 11 05:40:26 UTC 2013",
     "previousRevision": null,
     "revision": 2,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     

     View Slide

162. $ uploadScript.py -e TEST /test/hello sample.groovy
     {
     "active": false,
     "allocationPercentage": 0,
     "creationDate": "Mon Nov 11 05:40:26 UTC 2013",
     "revision": 2,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     $ activateScript.py -e TEST --revision 3 /test/hello
     {
     "active": true,
     "allocationPercentage": 100,
     "creationDate": "Mon Nov 11 05:42:05 UTC 2013",
     "previousRevision": 2,
     "revision": 3,
     "userAuthorizationRequired": true,
     "userAuthorizationType": "https"
     }
     

     View Slide

163. Future
     

     View Slide

164. millions of API requests per day
     0
     500
     1000
     1500
     2000
     2008 2009 2010 2011 2012 Today
     

     View Slide

165. View Slide

166. View Slide

167. View Slide

168. View Slide

169. View Slide

170. View Slide

171. View Slide

172. /ps3/home
     Dependency F
     10 Threads
     Dependency G
     10 Threads
     Dependency H
     10 Threads
     Dependency I
     5 Threads
     Dependency J
     8 Threads
     Dependency A
     10 Threads
     Dependency B
     8 Threads
     Dependency C
     10 Threads
     Dependency D
     15 Threads
     Dependency E
     5 Threads
     Dependency K
     15 Threads
     Dependency L
     4 Threads
     Dependency M
     5 Threads
     Dependency N
     10 Threads
     Dependency O
     10 Threads
     Dependency P
     10 Threads
     Dependency Q
     8 Threads
     Dependency R
     10 Threads
     Dependency S
     8 Threads
     Dependency T
     10 Threads
     /android/home
     /tv/home
     Functional Reactive Dynamic Endpoints
     Asynchronous Java API
     

     View Slide

173. /ps3/home
     Dependency F
     10 Threads
     Dependency G
     10 Threads
     Dependency H
     10 Threads
     Dependency I
     5 Threads
     Dependency J
     8 Threads
     Dependency A
     10 Threads
     Dependency B
     8 Threads
     Dependency C
     10 Threads
     Dependency D
     15 Threads
     Dependency E
     5 Threads
     Dependency K
     15 Threads
     Dependency L
     4 Threads
     Dependency M
     5 Threads
     Dependency N
     10 Threads
     Dependency O
     10 Threads
     Dependency P
     10 Threads
     Dependency Q
     8 Threads
     Dependency R
     10 Threads
     Dependency S
     8 Threads
     Dependency T
     10 Threads
     /android/home
     /tv/home
     Functional Reactive Dynamic Endpoints
     Asynchronous Java API
     Hystrix
     fault-isolation layer
     

     View Slide

174. +
     Observable  u  =  new  GetUserCommand(id).observe();  
     Observable  g  =  new  GetGeoCommand(request).observe();  
     !
     Observable.zip(u,  g,  {user,  geo  -­‐>  
                      return  [username:  user.getUsername(),  
                                      currentLocation:  geo.getCounty()]        
     })
     RxJava in Hystrix 1.3+
     https://github.com/Netflix/Hystrix
     

     View Slide

175. View Slide

176. View Slide

177. Optimizing the Netflix API
     http://techblog.netflix.com/2013/01/optimizing-netflix-api.html
     !
     Why REST Keeps Me Up At Night
     http://blog.programmableweb.com/2012/05/15/why-rest-keeps-me-up-at-night/
     !
     Functional Reactive in the Netflix API with RxJava
     http://techblog.netflix.com/2013/02/rxjava-netflix-api.html
     !
     Application Resilience in a Service-oriented Architecture
     http://programming.oreilly.com/2013/06/application-resilience-in-a-service-oriented-architecture.html
     !
     Netflix API on Tech Blog
     http://techblog.netflix.com/search/label/api
     jobs.netflix.com
     Ben Christensen
     @benjchristensen
     http://www.linkedin.com/in/benjchristensen
     

     View Slide