Implementing Location Independent Invocation

Implementing Location Independent Invocation

Papers We Love, Paris
April 2016

3e09fee7b359be847ed5fa48f524a3d3?s=128

Christopher Meiklejohn

April 26, 2016
Tweet

Transcript

  1. Implementing Location Independent Invocation 1 Andrew P. Black and Yeshayahu

    Artsy Digital Equipment Corporation Distributed Systems Advanced Development Group IEEE Transactions on Parallel and Distributed Systems 1990
  2. None
  3. History • Digital Equipment Corporation 
 Work performed at the

    Distributed Systems Advanced Development Group 3
  4. History • Digital Equipment Corporation 
 Work performed at the

    Distributed Systems Advanced Development Group • IEEE ICDCS 1989
 Preliminary version at 9th International Conference on Distributed Computing Systems 3
  5. History • Digital Equipment Corporation 
 Work performed at the

    Distributed Systems Advanced Development Group • IEEE ICDCS 1989
 Preliminary version at 9th International Conference on Distributed Computing Systems • IEEE TPDS 1990
 Final version in the IEEE Transactions on Parallel and Distributed Systems 3
  6. Motivation Remote Procedure Call 4

  7. Remote Procedure Call • Remote Procedure Call (RPC)
 Eases the

    development of distributed applications 5
  8. Remote Procedure Call • Remote Procedure Call (RPC)
 Eases the

    development of distributed applications • Transfer of control
 Between, instead of within, address spaces 5
  9. Remote Procedure Call • Remote Procedure Call (RPC)
 Eases the

    development of distributed applications • Transfer of control
 Between, instead of within, address spaces • “Alleviate the need to be aware…”
 Abstraction hides away network protocols, parameter marshaling, external data representations… 5
  10. Intrinsic Differences • Different address spaces
 Makes passing objects by

    reference difficult, given the reference will no longer be valid 6
  11. Intrinsic Differences • Different address spaces
 Makes passing objects by

    reference difficult, given the reference will no longer be valid • Failure modes
 Both the caller and callee can fail independently 6
  12. Most Fundamental Difference? 7 Binding

  13. Which address space should the call be directed at? 8

  14. Two Methods For Binding • Default binding
 Server is chosen

    automatically 9
  15. Two Methods For Binding • Default binding
 Server is chosen

    automatically • Single server, no other choice 9
  16. Two Methods For Binding • Default binding
 Server is chosen

    automatically • Single server, no other choice • All servers are semantically equivalent 9
  17. Two Methods For Binding • Default binding
 Server is chosen

    automatically • Single server, no other choice • All servers are semantically equivalent • Clerks
 Application specific subroutines or packages 9
  18. Two Methods For Binding • Default binding
 Server is chosen

    automatically • Single server, no other choice • All servers are semantically equivalent • Clerks
 Application specific subroutines or packages • Each application must provide one 9
  19. Location Independent Invocation • Removes the binding step
 Abstraction above

    RPC that hides the explicit binding from the application developer 10
  20. Location Independent Invocation • Removes the binding step
 Abstraction above

    RPC that hides the explicit binding from the application developer • Conceptual presentation
 Provides a conceptual presentation, without specifics of previous implementations 10
  21. Location Independent Invocation • Removes the binding step
 Abstraction above

    RPC that hides the explicit binding from the application developer • Conceptual presentation
 Provides a conceptual presentation, without specifics of previous implementations • Emerald
 Distributed programming language with runtime system 10
  22. Location Independent Invocation • Removes the binding step
 Abstraction above

    RPC that hides the explicit binding from the application developer • Conceptual presentation
 Provides a conceptual presentation, without specifics of previous implementations • Emerald
 Distributed programming language with runtime system • Eden
 Distributed operating system 10
  23. When is LII useful? • Pure Functions / Binding as

    Load Balancing
 Example: a fast Fourier transform where all servers are equivalent and selection is for load balancing 11
  24. When is LII useful? • Pure Functions / Binding as

    Load Balancing
 Example: a fast Fourier transform where all servers are equivalent and selection is for load balancing • Application Data
 Selection based on correctness and performance 11
  25. When is LII useful? • Pure Functions / Binding as

    Load Balancing
 Example: a fast Fourier transform where all servers are equivalent and selection is for load balancing • Application Data
 Selection based on correctness and performance • Partitioned data and correctness
 Not all servers can answer for all requests 11
  26. When is LII useful? • Pure Functions / Binding as

    Load Balancing
 Example: a fast Fourier transform where all servers are equivalent and selection is for load balancing • Application Data
 Selection based on correctness and performance • Partitioned data and correctness
 Not all servers can answer for all requests • Replicated data and performance (or availability)
 Choice can be based on desired performance or inherent availability trade-off with consistency 11
  27. The Registry Problem • Mapping
 If service instances can move,

    we need to keep track of where they are running 12
  28. The Registry Problem • Mapping
 If service instances can move,

    we need to keep track of where they are running • Churn Rate
 Based on what the churn rate is, different mechanisms for tracking might be required 12
  29. “constant” to “very occasionally” 13 each server has a function

    that computes a map
  30. “more than very occasionally” 14 global naming service (with inherent

    CAP tradeoffs)
  31. “frequently” 15 automation when instances "move frequently enough to make

    the implementation of a static mapping function impractical."
  32. Why do objects move? 16 [see Emerald, Eden, …]

  33. Process Migration 17

  34. Process Migration • Sharing memory
 Better utilization of memory across

    a cluster 17
  35. Process Migration • Sharing memory
 Better utilization of memory across

    a cluster • Reducing communication costs
 Co-location of processes that work together on a task 17
  36. Process Migration • Sharing memory
 Better utilization of memory across

    a cluster • Reducing communication costs
 Co-location of processes that work together on a task • Increasing availability
 Replication or mobility to increase fault-tolerance 17
  37. Process Migration • Sharing memory
 Better utilization of memory across

    a cluster • Reducing communication costs
 Co-location of processes that work together on a task • Increasing availability
 Replication or mobility to increase fault-tolerance • Reconfigurability
 Reconfiguration of machines, application, or network topology 17
  38. Process Migration • Sharing memory
 Better utilization of memory across

    a cluster • Reducing communication costs
 Co-location of processes that work together on a task • Increasing availability
 Replication or mobility to increase fault-tolerance • Reconfigurability
 Reconfiguration of machines, application, or network topology • Special capabilities
 Access to specialized hardware only available on certain machines 17
  39. concurrent access 18 may result in thrashing

  40. and the code that manipulates that data moves along with

    it 19 data moves
  41. Object Migration • Locating the object
 Need to ensure object

    can be found at its new home 20
  42. Object Migration • Locating the object
 Need to ensure object

    can be found at its new home • Recovery of object
 In the event of a failure, the object has to be able to be recovered safely 20
  43. Object Migration • Locating the object
 Need to ensure object

    can be found at its new home • Recovery of object
 In the event of a failure, the object has to be able to be recovered safely • Required by some applications
 Objects may have to be local for some operations to succeed 20
  44. Object Migration • Locating the object
 Need to ensure object

    can be found at its new home • Recovery of object
 In the event of a failure, the object has to be able to be recovered safely • Required by some applications
 Objects may have to be local for some operations to succeed • “Quality” improvement in interaction
 “Cost is justified” when moving objects if interaction quality increases 20
  45. Contributions The Hermes System 21

  46. Overview • Expense voucher system at Digital
 Corporation-wide, real-life application

    22
  47. Overview • Expense voucher system at Digital
 Corporation-wide, real-life application

    • Process 22
  48. Overview • Expense voucher system at Digital
 Corporation-wide, real-life application

    • Process • Vouchers filled in by employees 22
  49. Overview • Expense voucher system at Digital
 Corporation-wide, real-life application

    • Process • Vouchers filled in by employees • Approved or rejected by managers 22
  50. Overview • Expense voucher system at Digital
 Corporation-wide, real-life application

    • Process • Vouchers filled in by employees • Approved or rejected by managers • If approved, cash disbursement is made and forms are archived 22
  51. Overview • Expense voucher system at Digital
 Corporation-wide, real-life application

    • Process • Vouchers filled in by employees • Approved or rejected by managers • If approved, cash disbursement is made and forms are archived • Remote, geo-distributed, asynchronous process
 Actions can occur on the order of minutes or days, at any location in Digital’s 36,000 node global, internal network 22
  52. “These requirements make it infeasible to store all the forms

    in a single centralized database, or even in a number of geographically dispersed databases.” 23
  53. Represent the data and code of each form as an

    object that can move around the network as the application demands. 24
  54. (Mobile) Objects • Objects have: 25

  55. (Mobile) Objects • Objects have: • State 25

  56. (Mobile) Objects • Objects have: • State • Methods 25

  57. (Mobile) Objects • Objects have: • State • Methods •

    Objects control: 25
  58. (Mobile) Objects • Objects have: • State • Methods •

    Objects control: • Persistence
 How state should be persisted for each object 25
  59. (Mobile) Objects • Objects have: • State • Methods •

    Objects control: • Persistence
 How state should be persisted for each object • Recovery
 How objects can be recovered, if they happen to fail 25
  60. (Mobile) Objects • Objects have: • State • Methods •

    Objects control: • Persistence
 How state should be persisted for each object • Recovery
 How objects can be recovered, if they happen to fail • Placement
 Where objects should be located on the network 25
  61. (Mobile) Objects • Objects have: • State • Methods •

    Objects control: • Persistence
 How state should be persisted for each object • Recovery
 How objects can be recovered, if they happen to fail • Placement
 Where objects should be located on the network • Remove invocation
 Objects can invoke methods on other objects 25
  62. "Thus, the view of the world presented to application programmers

    is a distributed ocean in which application-dependent objects of their own design can be floated." 26
  63. Node Services • Active Objects
 Objects that are active are

    stored in virtual memory 27
  64. Node Services • Active Objects
 Objects that are active are

    stored in virtual memory • Stable Storage
 Stable storage is used for objects that are not currently referenced 27
  65. Node Makeup • Supervisor
 Object creation, location, and relocation 28

  66. Node Makeup • Supervisor
 Object creation, location, and relocation •

    Application Objects
 Virtual memory containing the active objects in the system for each application 28
  67. Node Makeup • Supervisor
 Object creation, location, and relocation •

    Application Objects
 Virtual memory containing the active objects in the system for each application • Intra-object Communication System
 Location Independent Invocation system with underlying RPC mechanism 28
  68. None
  69. Storesites • Checkpointing
 Objects periodically checkpoint their state at storesites.

    30
  70. Storesites • Checkpointing
 Objects periodically checkpoint their state at storesites.

    • Recovery
 Storesites support object recovery in the event of node failure 30
  71. Storesites • Checkpointing
 Objects periodically checkpoint their state at storesites.

    • Recovery
 Storesites support object recovery in the event of node failure • Recovery safety and liveness
 Object’s current location stored at the storesite, and until object is confirmed dead, recovery is prevented 30
  72. Additional Services • Name Service
 Locating storesites, supervisors, and objects

    by name. 31
  73. Additional Services • Name Service
 Locating storesites, supervisors, and objects

    by name. • Authentication Service
 For authentication between supervisors 31
  74. How’s it built? • Window-based interface
 For interacting with objects

    and their local Hermes node 32
  75. How’s it built? • Window-based interface
 For interacting with objects

    and their local Hermes node • Processes
 Objects as processes that are distributed across the network 32
  76. How’s it built? • Window-based interface
 For interacting with objects

    and their local Hermes node • Processes
 Objects as processes that are distributed across the network • Modula-2+
 RPC and multithreading support from Digital SRC 32
  77. Contributions Locating Objects 33

  78. None
  79. Objects • Globally unique identifiers
 Each object has a globally

    unique identifier for its lifetime 35
  80. Objects • Globally unique identifiers
 Each object has a globally

    unique identifier for its lifetime • Age
 Each object has an age containing a monotonic counter that advances when the object attempt to move between nodes 35
  81. Objects • Globally unique identifiers
 Each object has a globally

    unique identifier for its lifetime • Age
 Each object has an age containing a monotonic counter that advances when the object attempt to move between nodes • Location and storesite
 Each object contains a current location and a current storesite 35
  82. Temporal Address Descriptors • Temporal Address Descriptors (tad)
 Pair of

    node identifier and its age that represents an objects location at some point in time 36
  83. Temporal Address Descriptors • Temporal Address Descriptors (tad)
 Pair of

    node identifier and its age that represents an objects location at some point in time • Passed implicitly
 Passed implicitly along with the guid when the object is passed by reference 36
  84. Temporal Address Descriptors • Temporal Address Descriptors (tad)
 Pair of

    node identifier and its age that represents an objects location at some point in time • Passed implicitly
 Passed implicitly along with the guid when the object is passed by reference • Cached
 Supervisors cache tad’s locally for currently, and previously, referenced objects 36
  85. Locating an Object for Invocation • Local invocation attempted first


    Invocation if local; else, we must follow the tad to identify the current location 37
  86. Locating an Object for Invocation • Local invocation attempted first


    Invocation if local; else, we must follow the tad to identify the current location • Follow path of tads until object located
 Follow tads, returning the most recent tad back to the forwarding node until the object is found 37
  87. Locating an Object for Invocation • Local invocation attempted first


    Invocation if local; else, we must follow the tad to identify the current location • Follow path of tads until object located
 Follow tads, returning the most recent tad back to the forwarding node until the object is found • Update cache
 Each node updates its local cache of tad’s to optimize subsequent invocations 37
  88. End of Forwarding Chain • Forwarding chain doesn’t locate object


    If we can’t find the object via forwarding, we need to resort to asking the storesite for the current location 38
  89. End of Forwarding Chain • Forwarding chain doesn’t locate object


    If we can’t find the object via forwarding, we need to resort to asking the storesite for the current location • Forwarding pointers are still necessary
 However, since objects can move immediately after we access the storesite, forwarding pointers are still necessary 38
  90. End of Forwarding Chain • Forwarding chain doesn’t locate object


    If we can’t find the object via forwarding, we need to resort to asking the storesite for the current location • Forwarding pointers are still necessary
 However, since objects can move immediately after we access the storesite, forwarding pointers are still necessary • Resort to name service
 If objects migrate storesites, it may be required to contact the global name service to identify the current storesite 38
  91. RPC marshalling allows for guid/tad maintenance in structured responses! 39

    Cool!
  92. Invocation and location are combined to prevent an object from

    moving after identifying it’s location. 40
  93. None
  94. Idempotence and Sequencing • Invocations can succeed but fail prior

    to response
 Processing of an invocation and response to an invocation are not atomic 42
  95. Idempotence and Sequencing • Invocations can succeed but fail prior

    to response
 Processing of an invocation and response to an invocation are not atomic • Idempotence is one solution
 Given recovery may trigger duplicate invocation, ensuring idempotence in methods is essential 42
  96. Idempotence and Sequencing • Invocations can succeed but fail prior

    to response
 Processing of an invocation and response to an invocation are not atomic • Idempotence is one solution
 Given recovery may trigger duplicate invocation, ensuring idempotence in methods is essential • Otherwise, sequencing
 …or, simply put, you could just use consensus. 42
  97. Contributions An Example 43

  98. A B C

  99. A B C X Y

  100. A B C X Y migration (A, 0) -> (B,

    1) Y
  101. A B C X Y migration (A, 0) -> (B,

    1) Y Y migration (B, 1) -> (C, 2)
  102. A B C X Y migration (A, 0) -> (B,

    1) Y Y migration (B, 1) -> (C, 2) local invocation fails
  103. A B C X Y migration (A, 0) -> (B,

    1) Y Y migration (B, 1) -> (C, 2) local invocation fails invoke on Y with tad (B, 1) updated tad (C, 2)
  104. A B C X Y migration (A, 0) -> (B,

    1) Y Y migration (B, 1) -> (C, 2) local invocation fails invoke on Y with tad (B, 1) updated tad (C, 2) invoke on Y with tad (C, 2)
  105. A B C X Y migration (A, 0) -> (B,

    1) Y Y migration (B, 1) -> (C, 2) local invocation fails invoke on Y with tad (B, 1) updated tad (C, 2) invoke on Y with tad (C, 2) send response!
  106. Contributions Hybrid Approach 52

  107. Two approaches for following forwarding chains 53

  108. Throw Error • Return error if tad is out of

    date
 If temporal address descriptor is out of date, return an error immediately 54
  109. Throw Error • Return error if tad is out of

    date
 If temporal address descriptor is out of date, return an error immediately • Simplifies failure handling
 Control is returned to the invoker immediately; thread of control is not consumed 54
  110. Throw Error • Return error if tad is out of

    date
 If temporal address descriptor is out of date, return an error immediately • Simplifies failure handling
 Control is returned to the invoker immediately; thread of control is not consumed • Invoker must retry with new tad
 Invoker must update local information with new tad and repeat invocation 54
  111. Invocation Propagation • Propagation of invocation to location referenced by

    tad
 Propagation of the invocation to the location that is referenced by the nodes temporal address descriptor 55
  112. Invocation Propagation • Propagation of invocation to location referenced by

    tad
 Propagation of the invocation to the location that is referenced by the nodes temporal address descriptor • Ties up thread of control
 Until object located, thread of control is tied up 55
  113. Invocation Propagation • Propagation of invocation to location referenced by

    tad
 Propagation of the invocation to the location that is referenced by the nodes temporal address descriptor • Ties up thread of control
 Until object located, thread of control is tied up • More prone to disruption
 Failures in the middle of the chain can cause loss of availability 55
  114. Invocation Propagation • Propagation of invocation to location referenced by

    tad
 Propagation of the invocation to the location that is referenced by the nodes temporal address descriptor • Ties up thread of control
 Until object located, thread of control is tied up • More prone to disruption
 Failures in the middle of the chain can cause loss of availability • Allows cache maintenance
 All nodes along the path can be updated with up-to-date tads as forwarding occurs 55
  115. Hybrid Approach • Propagation is used for finite hop count


    Propagation of the invocation is done for a finite number of hops, until a hop count is exceeded 56
  116. Hybrid Approach • Propagation is used for finite hop count


    Propagation of the invocation is done for a finite number of hops, until a hop count is exceeded • Return error to invoker
 Error is returned to the invoker, and the invoker must try again 56
  117. Hybrid Approach • Propagation is used for finite hop count


    Propagation of the invocation is done for a finite number of hops, until a hop count is exceeded • Return error to invoker
 Error is returned to the invoker, and the invoker must try again • Short forwarding chains
 Performed under the belief that forwarding chains will typically be short 56
  118. Tad Maintenance • Stash
 Local cache always keeps the most

    recent tad for a given object 57
  119. Tad Maintenance • Stash
 Local cache always keeps the most

    recent tad for a given object • Long chains reduce performance 57
  120. Tad Maintenance • Stash
 Local cache always keeps the most

    recent tad for a given object • Long chains reduce performance • Latency & throughput are reduced linearly 57
  121. Tad Maintenance • Stash
 Local cache always keeps the most

    recent tad for a given object • Long chains reduce performance • Latency & throughput are reduced linearly • Availability is reduced exponentially 57
  122. Contributions Storesites 58

  123. Locating through Storesites • Solution for broken forwarding chains
 Fallback

    to using storesites for locating objects 59
  124. Locating through Storesites • Solution for broken forwarding chains
 Fallback

    to using storesites for locating objects • Initial storesite
 When created, objects have an initial storesite; this location along with 2PC is used to track object location after migrations 59
  125. Locating through Storesites • Solution for broken forwarding chains
 Fallback

    to using storesites for locating objects • Initial storesite
 When created, objects have an initial storesite; this location along with 2PC is used to track object location after migrations • Invocation can race with migration
 Forwarding pointers are still required for finding the current location of a process 59
  126. Storesite Migration • Forwarding pointers
 Support object migration from storesites

    through the use of forwarding pointers 60
  127. Storesite Migration • Forwarding pointers
 Support object migration from storesites

    through the use of forwarding pointers • Encode initial and track after first migration
 Encode the storesite in the globally unique identifier and register with the name service after first migration 60
  128. Storesite Migration • Forwarding pointers
 Support object migration from storesites

    through the use of forwarding pointers • Encode initial and track after first migration
 Encode the storesite in the globally unique identifier and register with the name service after first migration • Query in parallel
 Query both the storesite and global name service in parallel to reduce latency for locating objects 60
  129. Contributions Related Work 61

  130. Previous Systems • Eden: OS-level approach
 Process migration supported with

    ‘hints’; fallback to stable storage and network broadcast for identifying current location. 62
  131. Previous Systems • Eden: OS-level approach
 Process migration supported with

    ‘hints’; fallback to stable storage and network broadcast for identifying current location. • Emerald: language-level approach
 Process migration supported with forwarding pointers; fallback to stable storage with broadcast and pairwise inspection for identifying current location. 62
  132. Emerald was technically superior 63 process migration during invocation

  133. Emerald was technically superior 64 dynamic type checking

  134. Emerald was technically superior 65 dynamic software update

  135. “While an attractive paradigm for the future, we judge that

    we were unlikely to successfully introduce a new programming language for commercial distributed applications.” 66
  136. Related Systems • Demos/MP
 Unidirectional links for process migration which

    are extremely similar to temporal address descriptors 67
  137. Related Systems • Demos/MP
 Unidirectional links for process migration which

    are extremely similar to temporal address descriptors • Locus, MOS, R*
 “Home” machine tracks current location of processes that were birthed there which is a similar idea to storesites 67
  138. Contributions Evaluation 68

  139. Implementation • Modula-2+ specific
 Many details in the implementation are

    specific to Modula-2+ 69
  140. Implementation • Modula-2+ specific
 Many details in the implementation are

    specific to Modula-2+ • RPC ‘stubs’
 Local and remote stubs are used to wrap calls with the code for performing maintenance of the tad lifecycle: caching, forwarding, etc. 69
  141. Implementation • Modula-2+ specific
 Many details in the implementation are

    specific to Modula-2+ • RPC ‘stubs’
 Local and remote stubs are used to wrap calls with the code for performing maintenance of the tad lifecycle: caching, forwarding, etc. • Fix and unfix
 Objects are ‘fixed’ during the duration of the call, as to prevent object migration during invocation 69
  142. Questionable Evaluation • Actual, real-life system!
 Hermes is an actual

    system that had a working implementation! (albeit, in a laboratory) 70
  143. Questionable Evaluation • Actual, real-life system!
 Hermes is an actual

    system that had a working implementation! (albeit, in a laboratory) • Evaluation is somewhat questionable
 Wide variance in latencies without explanation; large forwarding chains are never evaluated; hard to understand where certain latency is coming from 70
  144. Questionable Evaluation • Actual, real-life system!
 Hermes is an actual

    system that had a working implementation! (albeit, in a laboratory) • Evaluation is somewhat questionable
 Wide variance in latencies without explanation; large forwarding chains are never evaluated; hard to understand where certain latency is coming from • "cost of communication is outweighed by the gain in parallelism."
 Unclear where the parallelism gains originate from in the system, or why these would override the cost of communication in latency penalties 70
  145. Partitions • Object unavailability
 Objects in the system will become

    temporarily unavailable under network partitions 71
  146. Partitions • Object unavailability
 Objects in the system will become

    temporarily unavailable under network partitions • Disruption to forwarding chains
 Partitions can also be very disruptive to forwarding chains where intermediary nodes may be unavailable 71
  147. Partitions • Object unavailability
 Objects in the system will become

    temporarily unavailable under network partitions • Disruption to forwarding chains
 Partitions can also be very disruptive to forwarding chains where intermediary nodes may be unavailable • Fallback
 We can fallback to the object’s store site or using the name service, but these are also susceptible to network partitions as well 71
  148. “Exactly-Once” Semantics • Invocation can fail for many reasons
 Both

    invoker and invokee can fail at any point; invokee can fail after performing side-effect but before responding 72
  149. “Exactly-Once” Semantics • Invocation can fail for many reasons
 Both

    invoker and invokee can fail at any point; invokee can fail after performing side-effect but before responding • Recovery from stable storage
 Recovery does not guarantee “exactly-once” semantics; some invocations may be retried upon recovery if they were performed before a checkpoint 72
  150. “Exactly-Once” Semantics • Invocation can fail for many reasons
 Both

    invoker and invokee can fail at any point; invokee can fail after performing side-effect but before responding • Recovery from stable storage
 Recovery does not guarantee “exactly-once” semantics; some invocations may be retried upon recovery if they were performed before a checkpoint • Idempotence
 Idempotence is the best strategy for mitigation of these issues 72
  151. In Conclusion 73

  152. In Summary • Addresses how to locate objects that are

    mobile
 Through the use of forwarding chains and “home” sites, eliminate the explicit binding step 74
  153. In Summary • Addresses how to locate objects that are

    mobile
 Through the use of forwarding chains and “home” sites, eliminate the explicit binding step • Selection and placement are still up to the user
 Developers still need to be concerned with where to place objects, and how to select the target objects of invocation 74
  154. In Summary • Addresses how to locate objects that are

    mobile
 Through the use of forwarding chains and “home” sites, eliminate the explicit binding step • Selection and placement are still up to the user
 Developers still need to be concerned with where to place objects, and how to select the target objects of invocation • RPC is still a problematic paradigm
 Issues with duplicate invocation, idempotence, and sequencing of operations remain challenges for the developer 74
  155. Further Reading • Lee, Collin, Seo Jin Park, Ankita Kejriwal,

    Satoshi Matsushita, and John Ousterhout. 2015. “Implementing Linearizability at Large Scale and Low Latency.” 75
  156. Further Reading • Lee, Collin, Seo Jin Park, Ankita Kejriwal,

    Satoshi Matsushita, and John Ousterhout. 2015. “Implementing Linearizability at Large Scale and Low Latency.” • Helland, Pat. 2012. “Idempotence Is Not a Medical Condition.” 75
  157. Further Reading • Lee, Collin, Seo Jin Park, Ankita Kejriwal,

    Satoshi Matsushita, and John Ousterhout. 2015. “Implementing Linearizability at Large Scale and Low Latency.” • Helland, Pat. 2012. “Idempotence Is Not a Medical Condition.” • Kendall, Samuel C, Jim Waldo, Ann Wollrath, and Geoff Wyant. 1994. “A Note on Distributed Computing.” 75
  158. Further Reading • Lee, Collin, Seo Jin Park, Ankita Kejriwal,

    Satoshi Matsushita, and John Ousterhout. 2015. “Implementing Linearizability at Large Scale and Low Latency.” • Helland, Pat. 2012. “Idempotence Is Not a Medical Condition.” • Kendall, Samuel C, Jim Waldo, Ann Wollrath, and Geoff Wyant. 1994. “A Note on Distributed Computing.” • Black, Andrew P, Norman C Hutchinson, Eric Jul, and Henry M Levy. 2007. “The Development of the Emerald Programming Language.” 75
  159. 76 Christopher Meiklejohn @cmeik Thanks!