Tweet
Tweet

Slide 1

Slide 1 text

Philipp Haller Philipp Haller 1 Docent Lecture Robust, Large-scale Concurrent and Distributed Programming KTH Royal Institute of Technology Stockholm, Sweden August 17th, 2018

Slide 2

Slide 2 text

Philipp Haller Motivation 2

Slide 3

Slide 3 text

Philipp Haller Motivation Demands of new and emerging software applications: 2

Slide 4

Slide 4 text

Philipp Haller Motivation Demands of new and emerging software applications: • Rapidly increasing scale of workloads: 2

Slide 5

Slide 5 text

Philipp Haller Motivation Demands of new and emerging software applications: • Rapidly increasing scale of workloads: – CERN amassed about 200 PB of data from over 800 trillion collisions searching for the Higgs boson. [1] – Steam, a digital content distribution service, delivers 16.9 PB per week to users in Germany (USA: 46.9 PB) [2] – Twitter has about 330 million monthly active users [3] • Reacting at the speed of the environment (guaranteed timely responses) – Example: autonomous driving • High availability • Fault tolerance 2

Slide 6

Slide 6 text

Philipp Haller 3

Slide 7

Slide 7 text

Philipp Haller 3 Steam delivers 16.9 PB per week to users in Germany (USA: 46.9 PB) [2]

Slide 8

Slide 8 text

Philipp Haller Motivation 4

Slide 9

Slide 9 text

Philipp Haller Motivation Demands of new and emerging software applications: • Rapidly increasing scale of workloads: – CERN amassed about 200 PB of data from over 800 trillion collisions searching for the Higgs boson. [1] – Steam, a digital content distribution service, delivers 16.9 PB per week to users in Germany (USA: 46.9 PB) [2] 4 February 2018

Slide 10

Slide 10 text

Philipp Haller Motivation Demands of new and emerging software applications: • Rapidly increasing scale of workloads: – CERN amassed about 200 PB of data from over 800 trillion collisions searching for the Higgs boson. [1] – Steam, a digital content distribution service, delivers 16.9 PB per week to users in Germany (USA: 46.9 PB) [2] – Twitter has about 330 million monthly active users [3] 4 February 2018 Q4, 2017

Slide 11

Slide 11 text

Philipp Haller Motivation Demands of new and emerging software applications: • Rapidly increasing scale of workloads: – CERN amassed about 200 PB of data from over 800 trillion collisions searching for the Higgs boson. [1] – Steam, a digital content distribution service, delivers 16.9 PB per week to users in Germany (USA: 46.9 PB) [2] – Twitter has about 330 million monthly active users [3] • Reacting at the speed of the environment (guaranteed timely responses) 4 February 2018 Q4, 2017

Slide 12

Slide 12 text

Philipp Haller Motivation Demands of new and emerging software applications: • Rapidly increasing scale of workloads: – CERN amassed about 200 PB of data from over 800 trillion collisions searching for the Higgs boson. [1] – Steam, a digital content distribution service, delivers 16.9 PB per week to users in Germany (USA: 46.9 PB) [2] – Twitter has about 330 million monthly active users [3] • Reacting at the speed of the environment (guaranteed timely responses) – Example: autonomous driving 4 February 2018 Q4, 2017

Slide 13

Slide 13 text

Philipp Haller Motivation Demands of new and emerging software applications: • Rapidly increasing scale of workloads: – CERN amassed about 200 PB of data from over 800 trillion collisions searching for the Higgs boson. [1] – Steam, a digital content distribution service, delivers 16.9 PB per week to users in Germany (USA: 46.9 PB) [2] – Twitter has about 330 million monthly active users [3] • Reacting at the speed of the environment (guaranteed timely responses) – Example: autonomous driving • High availability 4 February 2018 Q4, 2017

Slide 14

Slide 14 text

Philipp Haller Motivation Demands of new and emerging software applications: • Rapidly increasing scale of workloads: – CERN amassed about 200 PB of data from over 800 trillion collisions searching for the Higgs boson. [1] – Steam, a digital content distribution service, delivers 16.9 PB per week to users in Germany (USA: 46.9 PB) [2] – Twitter has about 330 million monthly active users [3] • Reacting at the speed of the environment (guaranteed timely responses) – Example: autonomous driving • High availability • Fault tolerance 4 February 2018 Q4, 2017

Slide 15

Slide 15 text

Philipp Haller Distributed Programming: A Solution 5

Slide 16

Slide 16 text

Philipp Haller Distributed Programming: A Solution • Enables construcing systems that are: 5

Slide 17

Slide 17 text

Philipp Haller Distributed Programming: A Solution • Enables construcing systems that are: – physically distributed, e.g. Internet of Things 5

Slide 18

Slide 18 text

Philipp Haller Distributed Programming: A Solution • Enables construcing systems that are: – physically distributed, e.g. Internet of Things – fault-tolerant 5

Slide 19

Slide 19 text

Philipp Haller Distributed Programming: A Solution • Enables construcing systems that are: – physically distributed, e.g. Internet of Things – fault-tolerant – highly available 5

Slide 20

Slide 20 text

Philipp Haller Distributed Programming: A Solution • Enables construcing systems that are: – physically distributed, e.g. Internet of Things – fault-tolerant – highly available – elastic (subsumes scalable) 5

Slide 21

Slide 21 text

Philipp Haller Distributed Programming: A Challenge 6

Slide 22

Slide 22 text

Philipp Haller Distributed Programming: A Challenge • Programmers must master the complex interplay of: 6

Slide 23

Slide 23 text

Philipp Haller Distributed Programming: A Challenge • Programmers must master the complex interplay of: – concurrency of computations 6

Slide 24

Slide 24 text

Philipp Haller Distributed Programming: A Challenge • Programmers must master the complex interplay of: – concurrency of computations – asynchronicity of events 6

Slide 25

Slide 25 text

Philipp Haller Distributed Programming: A Challenge • Programmers must master the complex interplay of: – concurrency of computations – asynchronicity of events – failure of communication and/or systems 6

Slide 26

Slide 26 text

Philipp Haller Distributed Programming: A Challenge • Programmers must master the complex interplay of: – concurrency of computations – asynchronicity of events – failure of communication and/or systems • An extreme challenge even for expert programmers 6

Slide 27

Slide 27 text

Philipp Haller Overview • Motivation • Part 1: Type systems for data-race safe concurrency • Part 2: Practical deterministic concurrency • Part 3: Lineage-based distributed programming • Ongoing and future work • Conclusion 7

Slide 28

Slide 28 text

Philipp Haller Overview • Motivation • Part 1: Type systems for data-race safe concurrency • Part 2: Practical deterministic concurrency • Part 3: Lineage-based distributed programming • Ongoing and future work • Conclusion 8

Slide 29

Slide 29 text

Philipp Haller Data Races: A Concurrency Hazard 9

Slide 30

Slide 30 text

Philipp Haller Data Races: A Concurrency Hazard • What is a data race? 9

Slide 31

Slide 31 text

Philipp Haller Data Races: A Concurrency Hazard • What is a data race? • A data race occurs 9

Slide 32

Slide 32 text

Philipp Haller Data Races: A Concurrency Hazard • What is a data race? • A data race occurs – when two tasks (threads, processes, actors) concurrently access the same shared variable (or object field) and 9

Slide 33

Slide 33 text

Philipp Haller Data Races: A Concurrency Hazard • What is a data race? • A data race occurs – when two tasks (threads, processes, actors) concurrently access the same shared variable (or object field) and – at least one of the accesses is a write (an assignment) 9

Slide 34

Slide 34 text

Philipp Haller Data Races: A Concurrency Hazard • What is a data race? • A data race occurs – when two tasks (threads, processes, actors) concurrently access the same shared variable (or object field) and – at least one of the accesses is a write (an assignment) • In practice, data races are difficult to find and fix 9

Slide 35

Slide 35 text

Philipp Haller Data Races: A Concurrency Hazard • What is a data race? • A data race occurs – when two tasks (threads, processes, actors) concurrently access the same shared variable (or object field) and – at least one of the accesses is a write (an assignment) • In practice, data races are difficult to find and fix • They can have dramatic consequences… 9

Slide 36

Slide 36 text

Philipp Haller 10

Slide 37

Slide 37 text

Philipp Haller 10

Slide 38

Slide 38 text

Philipp Haller 10

Slide 39

Slide 39 text

Philipp Haller 10 The Northeast blackout of 2003: a widespread power outage throughout parts of the Northeastern and Midwestern US and the Canadian province of Ontario on August 14, 2003 Primary cause: a data-race bug in the alarm system at the control room of FirstEnergy Corporation

Slide 40

Slide 40 text

Philipp Haller Problem Most widely-used pogramming languages cannot ensure data-race safety for their provided or enabled concurrency abstractions 11

Slide 41

Slide 41 text

Philipp Haller Problem Most widely-used pogramming languages cannot ensure data-race safety for their provided or enabled concurrency abstractions 11 IEEE Spectrum ranking "Top Programming Languages 2018" ("Trending" preset) https://spectrum.ieee.org/static/interactive-the-top-programming-languages-2018

Slide 42

Slide 42 text

Philipp Haller Goal 12

Slide 43

Slide 43 text

Philipp Haller Goal Static prevention of data races 12

Slide 44

Slide 44 text

Philipp Haller Goal Static prevention of data races • using a lightweight type system 12

Slide 45

Slide 45 text

Philipp Haller Goal Static prevention of data races • using a lightweight type system • that minimizes the effort to reuse existing code 12

Slide 46

Slide 46 text

Philipp Haller Goal Static prevention of data races • using a lightweight type system • that minimizes the effort to reuse existing code Focus: 12

Slide 47

Slide 47 text

Philipp Haller Goal Static prevention of data races • using a lightweight type system • that minimizes the effort to reuse existing code Focus: • Existing, full-featured languages like Scala 12

Slide 48

Slide 48 text

Philipp Haller Goal Static prevention of data races • using a lightweight type system • that minimizes the effort to reuse existing code Focus: • Existing, full-featured languages like Scala 12 In contrast to new language designs like Rust

Slide 49

Slide 49 text

Philipp Haller State of the Art 13

Slide 50

Slide 50 text

Philipp Haller State of the Art • A lot of progress in type systems for safe concurrency (linear and affine types, static capabilities, uniqueness types, ownership types, region inference, etc.) 13

Slide 51

Slide 51 text

Philipp Haller State of the Art • A lot of progress in type systems for safe concurrency (linear and affine types, static capabilities, uniqueness types, ownership types, region inference, etc.) • Challenges: 13

Slide 52

Slide 52 text

Philipp Haller State of the Art • A lot of progress in type systems for safe concurrency (linear and affine types, static capabilities, uniqueness types, ownership types, region inference, etc.) • Challenges: – Sound integration with advanced type system features 13

Slide 53

Slide 53 text

Philipp Haller State of the Art • A lot of progress in type systems for safe concurrency (linear and affine types, static capabilities, uniqueness types, ownership types, region inference, etc.) • Challenges: – Sound integration with advanced type system features 13 Example: local type inference

Slide 54

Slide 54 text

Philipp Haller State of the Art • A lot of progress in type systems for safe concurrency (linear and affine types, static capabilities, uniqueness types, ownership types, region inference, etc.) • Challenges: – Sound integration with advanced type system features – Adoption on large scale 13 Example: local type inference

Slide 55

Slide 55 text

Philipp Haller State of the Art • A lot of progress in type systems for safe concurrency (linear and affine types, static capabilities, uniqueness types, ownership types, region inference, etc.) • Challenges: – Sound integration with advanced type system features – Adoption on large scale • Key: reuse of existing code 13 Example: local type inference

Slide 56

Slide 56 text

Philipp Haller Example Image data 14

Slide 57

Slide 57 text

Philipp Haller Example Image data apply filter 14

Slide 58

Slide 58 text

Philipp Haller Example Image data apply filter 14

Slide 59

Slide 59 text

Philipp Haller Example Image data apply filter Image processing pipeline: filter 1 filter 2 14

Slide 60

Slide 60 text

Philipp Haller Example Image data apply filter Image processing pipeline: filter 1 filter 2 14 Pipeline stages run concurrently

Slide 61

Slide 61 text

Philipp Haller Example: Implementation 15

Slide 62

Slide 62 text

Philipp Haller Example: Implementation • Assumptions: – Main memory expensive – Image data large 15

Slide 63

Slide 63 text

Philipp Haller Example: Implementation • Assumptions: – Main memory expensive – Image data large • Approach for high performance: – Each pipeline stage is a concurrent actor – In-place update of image buffers – Pass mutable buffers by reference between actors 15

Slide 64

Slide 64 text

Philipp Haller Example: Problem Easy to produce data races: 1. Stage 1 sends a reference to a buffer to stage 2 2. Following the send, both stages have a reference to the same buffer 3. Stages can concurrently access the buffer 16

Slide 65

Slide 65 text

Philipp Haller Preventing Data Races 17

Slide 66

Slide 66 text

Philipp Haller Preventing Data Races Approach: 17

Slide 67

Slide 67 text

Philipp Haller Preventing Data Races Approach: – Extend type system with affine types and object capabilities 17

Slide 68

Slide 68 text

Philipp Haller Preventing Data Races Approach: – Extend type system with affine types and object capabilities – Affine types: 17

Slide 69

Slide 69 text

Philipp Haller Preventing Data Races Approach: – Extend type system with affine types and object capabilities – Affine types: • Variables of affine type may be "used" at most once 17

Slide 70

Slide 70 text

Philipp Haller Preventing Data Races Approach: – Extend type system with affine types and object capabilities – Affine types: • Variables of affine type may be "used" at most once • "Used" = consumed 17

Slide 71

Slide 71 text

Philipp Haller Preventing Data Races Approach: – Extend type system with affine types and object capabilities – Affine types: • Variables of affine type may be "used" at most once • "Used" = consumed • A consumed variable cannot be accessed any more 17

Slide 72

Slide 72 text

Philipp Haller Preventing Data Races Approach: – Extend type system with affine types and object capabilities – Affine types: • Variables of affine type may be "used" at most once • "Used" = consumed • A consumed variable cannot be accessed any more – Values of affine type are called permissions in our system 17

Slide 73

Slide 73 text

Philipp Haller Preventing Data Races Approach: – Extend type system with affine types and object capabilities – Affine types: • Variables of affine type may be "used" at most once • "Used" = consumed • A consumed variable cannot be accessed any more – Values of affine type are called permissions in our system – Permissions control access to transferable objects 17

Slide 74

Slide 74 text

Philipp Haller Guarantee of the Type System 18 Exchanging transferable objects between actors preserves actor isolation

Slide 75

Slide 75 text

Philipp Haller LaCasa: An Extension of Scala with Affine Types and Object Capabilities Transferable objects: instances of a new generic type Box[T] 19

Slide 76

Slide 76 text

Philipp Haller LaCasa: An Extension of Scala with Affine Types and Object Capabilities Transferable objects: instances of a new generic type Box[T] 19 def receive(box: Box[Message]): Unit = { box open { msg => msg.buffer = Array(1, 2, 3, 4) } ... } class Message { var buffer: Array[Byte] = _ }

Slide 77

Slide 77 text

Philipp Haller LaCasa: An Extension of Scala with Affine Types and Object Capabilities Transferable objects: instances of a new generic type Box[T] 19 def receive(box: Box[Message]): Unit = { box open { msg => msg.buffer = Array(1, 2, 3, 4) } ... } class Message { var buffer: Array[Byte] = _ } Accessing an encapsulated object requires the use of open

Slide 78

Slide 78 text

Philipp Haller LaCasa: An Extension of Scala with Affine Types and Object Capabilities Transferable objects: instances of a new generic type Box[T] 19 def receive(box: Box[Message]): Unit = { box open { msg => msg.buffer = Array(1, 2, 3, 4) } ... } class Message { var buffer: Array[Byte] = _ } msg is the encapsulated object Accessing an encapsulated object requires the use of open

Slide 79

Slide 79 text

Philipp Haller Permissions 20

Slide 80

Slide 80 text

Philipp Haller Permissions • The above code is still incomplete: opening a box requires a
 corresponding permission provided by the context 20

Slide 81

Slide 81 text

Philipp Haller Permissions • The above code is still incomplete: opening a box requires a
 corresponding permission provided by the context • Invoking open on box requires a permission with the following type: 20

Slide 82

Slide 82 text

Philipp Haller Permissions • The above code is still incomplete: opening a box requires a
 corresponding permission provided by the context • Invoking open on box requires a permission with the following type: 20 CanAccess { type C = box.C }

Slide 83

Slide 83 text

Philipp Haller Permissions • The above code is still incomplete: opening a box requires a
 corresponding permission provided by the context • Invoking open on box requires a permission with the following type: 20 CanAccess { type C = box.C } Dependent type

Slide 84

Slide 84 text

Philipp Haller Permissions • The above code is still incomplete: opening a box requires a
 corresponding permission provided by the context • Invoking open on box requires a permission with the following type: • Type member C links the permission type to a specific box 20 CanAccess { type C = box.C } Dependent type

Slide 85

Slide 85 text

Philipp Haller Permissions • The above code is still incomplete: opening a box requires a
 corresponding permission provided by the context • Invoking open on box requires a permission with the following type: • Type member C links the permission type to a specific box • A permission type CanAccess { type C = låda.C } would only be compatible with box iff 20 CanAccess { type C = box.C } Dependent type

Slide 86

Slide 86 text

Philipp Haller Permissions • The above code is still incomplete: opening a box requires a
 corresponding permission provided by the context • Invoking open on box requires a permission with the following type: • Type member C links the permission type to a specific box • A permission type CanAccess { type C = låda.C } would only be compatible with box iff – box and låda are aliases (statically-known) 20 CanAccess { type C = box.C } Dependent type

Slide 87

Slide 87 text

Philipp Haller Permissions (2) Making permissions available in the context via implicit parameters: 21

Slide 88

Slide 88 text

Philipp Haller Permissions (2) Making permissions available in the context via implicit parameters: 21 def receive(box: Box[Message]) (implicit p: CanAccess { type C = box.C }): Unit = { box open { msg => msg.buffer = Array(1, 2, 3, 4) } ... }

Slide 89

Slide 89 text

Philipp Haller Consuming Permissions Transfering a box from one actor to another consumes its access permission: 22

Slide 90

Slide 90 text

Philipp Haller Consuming Permissions Transfering a box from one actor to another consumes its access permission: 22 def receive(box: Box[Message]) (implicit p: CanAccess { type C = box.C }): Unit = { box open { msg => msg.buffer = Array(1, 2, 3, 4) } ... someActor.send(box) { // `p` unavailable here! ... } }

Slide 91

Slide 91 text

Philipp Haller Encapsulation Problem: not all types safe to transfer! 23

Slide 92

Slide 92 text

Philipp Haller Encapsulation Problem: not all types safe to transfer! 23 class Message { var buffer: Array[Int] = _ def leak(): Unit = { SomeObject.fld = buffer } } object SomeObject { var fld: Array[Int] = _ }

Slide 93

Slide 93 text

Philipp Haller Encapsulation 24 * simplified

Slide 94

Slide 94 text

Philipp Haller Encapsulation • Ensuring absence of data races requires restricting types put into boxes 24 * simplified

Slide 95

Slide 95 text

Philipp Haller Encapsulation • Ensuring absence of data races requires restricting types put into boxes • Requirements for “safe” classes:* – Methods only access parameters and this – Method parameter types are “safe” – Methods only instantiate “safe” classes – Types of fields are “safe” 24 * simplified

Slide 96

Slide 96 text

Philipp Haller Encapsulation • Ensuring absence of data races requires restricting types put into boxes • Requirements for “safe” classes:* – Methods only access parameters and this – Method parameter types are “safe” – Methods only instantiate “safe” classes – Types of fields are “safe” 24 “Safe” = conforms to object capability model [4] * simplified [4] Mark S. Miller. Robust Composition: Towards a Unified Approach to Access Control and Concurrency Control. PhD thesis, 2006

Slide 97

Slide 97 text

Philipp Haller Object Capabilities in Scala 25

Slide 98

Slide 98 text

Philipp Haller Object Capabilities in Scala • How common is object-capability safe code in Scala? 25

Slide 99

Slide 99 text

Philipp Haller Object Capabilities in Scala • How common is object-capability safe code in Scala? • Empirical study of over 75,000 SLOC of open-source Scala code: 25

Slide 100

Slide 100 text

Philipp Haller Object Capabilities in Scala • How common is object-capability safe code in Scala? • Empirical study of over 75,000 SLOC of open-source Scala code: 25 Project Version SLOC GitHub stats Scala stdlib 2.11.7 33,107 ✭5,795 257 Signal/Collect 8.0.6 10,159 ✭123 11 GeoTrellis 0.10.0-RC2 35,351 ✭400 38 -engine 3,868 -raster 22,291 -spark 9,192

Slide 101

Slide 101 text

Philipp Haller Object Capabilities in Scala Results of empirical study: 26 Project #classes/traits #ocap (%) #dir. insec. (%) Scala stdlib 1,505 644 (43%) 212/861 (25%) Signal/ Collect 236 159 (67%) 60/77 (78%) GeoTrellis -engine 190 40 (21%) 124/150 (83%) -raster 670 233 (35%) 325/437 (74%) -spark 326 101 (31%) 167/225 (74%) Total 2,927 1,177 (40%) 888/1,750 (51%)

Slide 102

Slide 102 text

Philipp Haller Object Capabilities in Scala Results of empirical study: 26 Project #classes/traits #ocap (%) #dir. insec. (%) Scala stdlib 1,505 644 (43%) 212/861 (25%) Signal/ Collect 236 159 (67%) 60/77 (78%) GeoTrellis -engine 190 40 (21%) 124/150 (83%) -raster 670 233 (35%) 325/437 (74%) -spark 326 101 (31%) 167/225 (74%) Total 2,927 1,177 (40%) 888/1,750 (51%)

Slide 103

Slide 103 text

Philipp Haller Object Capabilities in Scala Results of empirical study: 26 Project #classes/traits #ocap (%) #dir. insec. (%) Scala stdlib 1,505 644 (43%) 212/861 (25%) Signal/ Collect 236 159 (67%) 60/77 (78%) GeoTrellis -engine 190 40 (21%) 124/150 (83%) -raster 670 233 (35%) 325/437 (74%) -spark 326 101 (31%) 167/225 (74%) Total 2,927 1,177 (40%) 888/1,750 (51%) Immutability inference increases these percentages!

Slide 104

Slide 104 text

Philipp Haller Further Results 27

Slide 105

Slide 105 text

Philipp Haller Further Results • Object-oriented core languages 27

Slide 106

Slide 106 text

Philipp Haller Further Results • Object-oriented core languages – Formalization of object capabilities (type-based), uniqueness, separation, concurrency 27

Slide 107

Slide 107 text

Philipp Haller Further Results • Object-oriented core languages – Formalization of object capabilities (type-based), uniqueness, separation, concurrency • Meta theory 27

Slide 108

Slide 108 text

Philipp Haller Further Results • Object-oriented core languages – Formalization of object capabilities (type-based), uniqueness, separation, concurrency • Meta theory – Type soundness 27

Slide 109

Slide 109 text

Philipp Haller Further Results • Object-oriented core languages – Formalization of object capabilities (type-based), uniqueness, separation, concurrency • Meta theory – Type soundness – Isolation theorem for processes with shared heap 27

Slide 110

Slide 110 text

Philipp Haller Further Results • Object-oriented core languages – Formalization of object capabilities (type-based), uniqueness, separation, concurrency • Meta theory – Type soundness – Isolation theorem for processes with shared heap • Paper: 27 [5] Haller and Loiko. LaCasa: Lightweight affinity and object capabilities in Scala. OOPSLA 2016

Slide 111

Slide 111 text

Philipp Haller Ongoing Work 28

Slide 112

Slide 112 text

Philipp Haller Ongoing Work • Flow-sensitive type checking 28

Slide 113

Slide 113 text

Philipp Haller Ongoing Work • Flow-sensitive type checking 28 [6] Erik Reimers. Lightweight Software Isolation via Flow-Sensitive Capabilities in Scala. Master's thesis, KTH, 2017 (supervisor Philipp Haller)

Slide 114

Slide 114 text

Philipp Haller Ongoing Work • Flow-sensitive type checking • Empirical studies 28 [6] Erik Reimers. Lightweight Software Isolation via Flow-Sensitive Capabilities in Scala. Master's thesis, KTH, 2017 (supervisor Philipp Haller)

Slide 115

Slide 115 text

Philipp Haller Ongoing Work • Flow-sensitive type checking • Empirical studies – How much effort to change existing code? 28 [6] Erik Reimers. Lightweight Software Isolation via Flow-Sensitive Capabilities in Scala. Master's thesis, KTH, 2017 (supervisor Philipp Haller)

Slide 116

Slide 116 text

Philipp Haller Ongoing Work • Flow-sensitive type checking • Empirical studies – How much effort to change existing code? 28 [6] Erik Reimers. Lightweight Software Isolation via Flow-Sensitive Capabilities in Scala. Master's thesis, KTH, 2017 (supervisor Philipp Haller) [7] Haller, Sommar. Towards an Empirical Study of Affine Types for Isolated Actors in Scala. PLACES@ETAPS 2017

Slide 117

Slide 117 text

Philipp Haller Ongoing Work • Flow-sensitive type checking • Empirical studies – How much effort to change existing code? • Complete mechanization of meta-theory 28 [6] Erik Reimers. Lightweight Software Isolation via Flow-Sensitive Capabilities in Scala. Master's thesis, KTH, 2017 (supervisor Philipp Haller) [7] Haller, Sommar. Towards an Empirical Study of Affine Types for Isolated Actors in Scala. PLACES@ETAPS 2017

Slide 118

Slide 118 text

Philipp Haller LaCasa: Conclusion 29

Slide 119

Slide 119 text

Philipp Haller LaCasa: Conclusion • Preserving actor isolation is possible when transfering objects conforming to the object capability discipline 29

Slide 120

Slide 120 text

Philipp Haller LaCasa: Conclusion • Preserving actor isolation is possible when transfering objects conforming to the object capability discipline – Binary check whether a class is reusable unchanged 29

Slide 121

Slide 121 text

Philipp Haller LaCasa: Conclusion • Preserving actor isolation is possible when transfering objects conforming to the object capability discipline – Binary check whether a class is reusable unchanged • Integration with the full Scala language 29

Slide 122

Slide 122 text

Philipp Haller LaCasa: Conclusion • Preserving actor isolation is possible when transfering objects conforming to the object capability discipline – Binary check whether a class is reusable unchanged • Integration with the full Scala language • In medium to large open-source Scala projects, 21-67% of all classes conform to the object capability discipline 29

Slide 123

Slide 123 text

Philipp Haller Overview • Motivation • Part 1: Type systems for data-race safe concurrency • Part 2: Practical deterministic concurrency • Part 3: Lineage-based distributed programming • Ongoing and future work • Conclusion 30

Slide 124

Slide 124 text

Philipp Haller From Data-Race Freedom to Determinism • LaCasa: prevent data races via type system • However, due to non-determinism concurrent programs difficult to reason about even when they are data-race free 31

Slide 125

Slide 125 text

Philipp Haller Non-Determinism: Example 32

Slide 126

Slide 126 text

Philipp Haller Non-Determinism: Example Example 1: 32 @volatile var x = 0 def m(): Unit = { Future { x = 1 } Future { x = 2 } .. // does not access x }

Slide 127

Slide 127 text

Philipp Haller Non-Determinism: Example Example 1: 32 @volatile var x = 0 def m(): Unit = { Future { x = 1 } Future { x = 2 } .. // does not access x } What’s the value of x when an invocation of m returns?

Slide 128

Slide 128 text

Philipp Haller Reordering not always a problem Example 2: 33

Slide 129

Slide 129 text

Philipp Haller Reordering not always a problem Example 2: 33 val set = Set.empty[Int] Future { set.put(1) } set.put(2)

Slide 130

Slide 130 text

Philipp Haller Reordering not always a problem Example 2: 33 val set = Set.empty[Int] Future { set.put(1) } set.put(2) Assume: concurrent set

Slide 131

Slide 131 text

Philipp Haller Reordering not always a problem Example 2: 33 val set = Set.empty[Int] Future { set.put(1) } set.put(2) Eventually, set contains both 1 and 2, always Assume: concurrent set

Slide 132

Slide 132 text

Philipp Haller Reordering not always a problem Example 2: 33 val set = Set.empty[Int] Future { set.put(1) } set.put(2) Eventually, set contains both 1 and 2, always Bottom line: it depends on the datatype Assume: concurrent set

Slide 133

Slide 133 text

Philipp Haller Non-Commutative Operations Example 3: 34

Slide 134

Slide 134 text

Philipp Haller Non-Commutative Operations Example 3: 34 val set = Set.empty[Int] Future { set.put(1) } Future { if (set.contains(1)) { .. } } set.put(2)

Slide 135

Slide 135 text

Philipp Haller Non-Commutative Operations Example 3: 34 val set = Set.empty[Int] Future { set.put(1) } Future { if (set.contains(1)) { .. } } set.put(2) Result depends on schedule!

Slide 136

Slide 136 text

Philipp Haller Goal 35

Slide 137

Slide 137 text

Philipp Haller Goal • Programming model providing static determinism guarantees 35

Slide 138

Slide 138 text

Philipp Haller Goal • Programming model providing static determinism guarantees • More precisely: 35

Slide 139

Slide 139 text

Philipp Haller Goal • Programming model providing static determinism guarantees • More precisely: 35 "All non-failing executions compute the same result."

Slide 140

Slide 140 text

Philipp Haller Goal • Programming model providing static determinism guarantees • More precisely: 35 "All non-failing executions compute the same result." "Quasi-determinism" [8] [8] Kuper et al. Freeze after writing: quasi-deterministic parallel programming with LVars. POPL 2014

Slide 141

Slide 141 text

Philipp Haller Important Concerns 36

Slide 142

Slide 142 text

Philipp Haller Important Concerns • Starting from imperative, object-oriented language 36

Slide 143

Slide 143 text

Philipp Haller Important Concerns • Starting from imperative, object-oriented language – global state 36

Slide 144

Slide 144 text

Philipp Haller Important Concerns • Starting from imperative, object-oriented language – global state – pervasive aliasing 36

Slide 145

Slide 145 text

Philipp Haller Important Concerns • Starting from imperative, object-oriented language – global state – pervasive aliasing 36 Potential application to widely- used languages

Slide 146

Slide 146 text

Philipp Haller Important Concerns • Starting from imperative, object-oriented language – global state – pervasive aliasing • Important concerns: expressivity and performance 36 Potential application to widely- used languages

Slide 147

Slide 147 text

Philipp Haller Reactive Async: Approach 37

Slide 148

Slide 148 text

Philipp Haller Reactive Async: Approach • New programming model building on: 37

Slide 149

Slide 149 text

Philipp Haller Reactive Async: Approach • New programming model building on: – event-driven concurrency (similar to futures and promises) 37

Slide 150

Slide 150 text

Philipp Haller Reactive Async: Approach • New programming model building on: – event-driven concurrency (similar to futures and promises) – lattice-based data types 37

Slide 151

Slide 151 text

Philipp Haller Reactive Async: Approach • New programming model building on: – event-driven concurrency (similar to futures and promises) – lattice-based data types – reactive programming 37

Slide 152

Slide 152 text

Philipp Haller Reactive Async: Approach • New programming model building on: – event-driven concurrency (similar to futures and promises) – lattice-based data types – reactive programming • Build on LaCasa's type system to provide quasi-determinism guarantee at compile time 37

Slide 153

Slide 153 text

Philipp Haller Application: Static Program Analysis Example: return type analysis 38

Slide 154

Slide 154 text

Philipp Haller Application: Static Program Analysis Example: return type analysis 38 class C { def f(x: Int): D = if (x <= 0) g(x) else h(x-1) def g(y: Int): E = new E(y) def h(z: Int): D = if (z == 0) new F else f(z) }

Slide 155

Slide 155 text

Philipp Haller Application: Static Program Analysis Example: return type analysis 38 Class type hierarchy: D E F G class C { def f(x: Int): D = if (x <= 0) g(x) else h(x-1) def g(y: Int): E = new E(y) def h(z: Int): D = if (z == 0) new F else f(z) }

Slide 156

Slide 156 text

Philipp Haller Application: Static Program Analysis Example: return type analysis 38 Which types does method f possibly return? Class type hierarchy: D E F G class C { def f(x: Int): D = if (x <= 0) g(x) else h(x-1) def g(y: Int): E = new E(y) def h(z: Int): D = if (z == 0) new F else f(z) }

Slide 157

Slide 157 text

Philipp Haller Return Type Analysis (cont'd) 39 class C { def f(x: Int): D = if (x <= 0) g(x) else h(x-1) def g(y: Int): E = new E(y) def h(z: Int): D = if (z == 0) new F else f(z) }

Slide 158

Slide 158 text

Philipp Haller Return Type Analysis (cont'd) • Method f calls methods g and h; method h calls method f 39 class C { def f(x: Int): D = if (x <= 0) g(x) else h(x-1) def g(y: Int): E = new E(y) def h(z: Int): D = if (z == 0) new F else f(z) }

Slide 159

Slide 159 text

Philipp Haller Return Type Analysis (cont'd) • Method f calls methods g and h; method h calls method f • Programming model let's us express the resulting dependencies, forming a directed graph: 39 class C { def f(x: Int): D = if (x <= 0) g(x) else h(x-1) def g(y: Int): E = new E(y) def h(z: Int): D = if (z == 0) new F else f(z) }

Slide 160

Slide 160 text

Philipp Haller Return Type Analysis (cont'd) • Method f calls methods g and h; method h calls method f • Programming model let's us express the resulting dependencies, forming a directed graph: 39 class C { def f(x: Int): D = if (x <= 0) g(x) else h(x-1) def g(y: Int): E = new E(y) def h(z: Int): D = if (z == 0) new F else f(z) } f g h

Slide 161

Slide 161 text

Philipp Haller Return Type Analysis (cont'd) • Method f calls methods g and h; method h calls method f • Programming model let's us express the resulting dependencies, forming a directed graph: 39 class C { def f(x: Int): D = if (x <= 0) g(x) else h(x-1) def g(y: Int): E = new E(y) def h(z: Int): D = if (z == 0) new F else f(z) } f g h "calls"

Slide 162

Slide 162 text

Philipp Haller Evaluation 40

Slide 163

Slide 163 text

Philipp Haller Evaluation • Static analysis of JVM bytecode using the OPAL framework (OPen extensible Analysis Library) 40

Slide 164

Slide 164 text

Philipp Haller Evaluation • Static analysis of JVM bytecode using the OPAL framework (OPen extensible Analysis Library) – Concurrent design 40

Slide 165

Slide 165 text

Philipp Haller Evaluation • Static analysis of JVM bytecode using the OPAL framework (OPen extensible Analysis Library) – Concurrent design • Rewrote purity analysis and immutability analysis 40

Slide 166

Slide 166 text

Philipp Haller Evaluation • Static analysis of JVM bytecode using the OPAL framework (OPen extensible Analysis Library) – Concurrent design • Rewrote purity analysis and immutability analysis 40 http://www.opal-project.de

Slide 167

Slide 167 text

Philipp Haller Evaluation • Static analysis of JVM bytecode using the OPAL framework (OPen extensible Analysis Library) – Concurrent design • Rewrote purity analysis and immutability analysis • Ran analysis on JDK 8 update 45 40 http://www.opal-project.de

Slide 168

Slide 168 text

Philipp Haller Evaluation • Static analysis of JVM bytecode using the OPAL framework (OPen extensible Analysis Library) – Concurrent design • Rewrote purity analysis and immutability analysis • Ran analysis on JDK 8 update 45 – 18’591 class files, 163’268 methods, 77’128 fields 40 http://www.opal-project.de

Slide 169

Slide 169 text

Philipp Haller Results: Immutability Analysis • RA about 10x faster than FPCF (OPAL's fixed point computation framework) • RA = 294 LOC, FPCF = 424 LOC (1.44x) 41 FPCF (secs.) 1.0 1.5 2.0 2.5 Reactive-Async (secs.) 0.1 0.2 0.3 1 Thread 2 Threads 4 Threads 8 Threads 16 Threads 2.15 2.20 2.25 2.30 2.35 1.15 1.20 1.25 1.30 1.35 0.290 0.295 0.300 0.105 0.110 0.115

Slide 170

Slide 170 text

Philipp Haller Results: Immutability Analysis • RA about 10x faster than FPCF (OPAL's fixed point computation framework) • RA = 294 LOC, FPCF = 424 LOC (1.44x) 41 FPCF (secs.) 1.0 1.5 2.0 2.5 Reactive-Async (secs.) 0.1 0.2 0.3 1 Thread 2 Threads 4 Threads 8 Threads 16 Threads 2.15 2.20 2.25 2.30 2.35 1.15 1.20 1.25 1.30 1.35 0.290 0.295 0.300 0.105 0.110 0.115 box plot: whiskers = min/max
 top/bottom of box = 
 1st and 3rd quartile band in box: median

Slide 171

Slide 171 text

Philipp Haller Reactive Async: Conclusion • Deterministic concurrent programming model – Extension of imperative, object-oriented base language – Resolution of cyclic dependencies – Type system for object capabilities for safety • Experimental evaluation using large-scale, concurrent static analysis • Prototype implementation: https://github.com/phaller/reactive-async 42

Slide 172

Slide 172 text

Philipp Haller Reactive Async: Conclusion • Deterministic concurrent programming model – Extension of imperative, object-oriented base language – Resolution of cyclic dependencies – Type system for object capabilities for safety • Experimental evaluation using large-scale, concurrent static analysis • Prototype implementation: https://github.com/phaller/reactive-async 42 [9] Haller, Geries, Eichberg, and Salvaneschi.
 Reactive Async: Expressive deterministic concurrency. Scala Symposium 2016

Slide 173

Slide 173 text

Philipp Haller Overview • Motivation • Part 1: Type systems for data-race safe concurrency • Part 2: Practical deterministic concurrency • Part 3: Lineage-based distributed programming • Ongoing and future work • Conclusion 43

Slide 174

Slide 174 text

Philipp Haller Lineage Which resources are required for producing a particular expected result? Lineage may record information about: Data sets read/transformed for producing result data set 44 Etc. Services used for producing response

Slide 175

Slide 175 text

Philipp Haller Distributed programming with functional lineages a.k.a. function passing New data-centric programming model for functional processing of distributed data. 45

Slide 176

Slide 176 text

Philipp Haller Distributed programming with functional lineages a.k.a. function passing New data-centric programming model for functional processing of distributed data. Key ideas: 45

Slide 177

Slide 177 text

Philipp Haller Distributed programming with functional lineages a.k.a. function passing New data-centric programming model for functional processing of distributed data. Key ideas: 45 Utilize lineages for fault recovery

Slide 178

Slide 178 text

Philipp Haller Distributed programming with functional lineages a.k.a. function passing New data-centric programming model for functional processing of distributed data. Key ideas: 45 Provide lineages by programming abstractions Utilize lineages for fault recovery

Slide 179

Slide 179 text

Philipp Haller Distributed programming with functional lineages a.k.a. function passing New data-centric programming model for functional processing of distributed data. Key ideas: 45 Provide lineages by programming abstractions Keep data stationary (if possible), send functions Utilize lineages for fault recovery

Slide 180

Slide 180 text

Philipp Haller The function passing model Introducing Consists of 3 parts: Silos: stationary, typed, immutable data containers SiloRefs: references to local or remote Silos. Spores [10]: safe, serializable functions. 46

Slide 181

Slide 181 text

Philipp Haller The function passing model Introducing Consists of 3 parts: Silos: stationary, typed, immutable data containers SiloRefs: references to local or remote Silos. Spores [10]: safe, serializable functions. 46 [10] Miller, Haller, and Odersky. Spores: a type-based foundation for closures in the age of concurrency and distribution. ECOOP 2014

Slide 182

Slide 182 text

Philipp Haller The function passing model Some visual intuition of Master Worker 47

Slide 183

Slide 183 text

Philipp Haller The function passing model Some visual intuition of Silo SiloRef Master Worker 47

Slide 184

Slide 184 text

Philipp Haller The function passing model Some visual intuition of Master Worker 47

Slide 185

Slide 185 text

Philipp Haller The function passing model Some visual intuition of Master Worker 47

Slide 186

Slide 186 text

Philipp Haller Silos What are they? Silo[T] T SiloRef[T] Two parts. def apply def send def persist def unpersist SiloRef. Handle to a Silo. Silo. Typed, stationary data container. User interacts with SiloRef. SiloRefs come with 4 primitive operations. 48

Slide 187

Slide 187 text

Philipp Haller Silos What are they? Silo[T] T SiloRef[T] Primitive: apply Takes a function that is to be applied to the data in the silo associated with the SiloRef. Creates new silo to contain the data that the user- defined function returns; evaluation is deferred def apply[S](fun: T => SiloRef[S]): SiloRef[S] Enables interesting computation DAGs Deferred def apply def send def persist def unpersist 49

Slide 188

Slide 188 text

Philipp Haller Silos What are they? Silo[T] T SiloRef[T] Primitive: send Forces the built-up computation DAG to be sent to the associated node and applied. Future is completed with the result of the computation. def send(): Future[T] EAGER def apply def send def persist def unpersist 50

Slide 189

Slide 189 text

Philipp Haller Silos Silo[T] T SiloRef[T] Silo factories: Creates silo on given host populated with given value/text file/… object SiloRef { def populate[T](host: Host, value: T): SiloRef[T] def fromTextFile(host: Host, file: File): SiloRef[List[String]] ... } def apply def send def persist def unpersist Deferred What are they? 51

Slide 190

Slide 190 text

Philipp Haller Basic idea: apply/send Silo[T] Machine 1 Machine 2 SiloRef[T] T 52

Slide 191

Slide 191 text

Philipp Haller Basic idea: apply/send Silo[T] Machine 1 Machine 2 SiloRef[T] T SiloRef[S] ) 52

Slide 192

Slide 192 text

Philipp Haller Basic idea: apply/send Silo[T] Machine 1 Machine 2 SiloRef[T] λ T SiloRef[S] ) 52

Slide 193

Slide 193 text

Philipp Haller ) Basic idea: apply/send Silo[T] Machine 1 Machine 2 SiloRef[T] λ T SiloRef[S] S Silo[S] ) 52

Slide 194

Slide 194 text

Philipp Haller More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... 53

Slide 195

Slide 195 text

Philipp Haller More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) adults 54

Slide 196

Slide 196 text

Philipp Haller More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(spore { val localVehicles = vehicles // spore header ps => localVehicles.apply(spore { val localps = ps // spore header vs => SiloRef.populate(currentHost, localps.flatMap(p => // list of (p, v) for a single person p vs.flatMap { v => if (v.owner.name == p.name) List((p, v)) else Nil } ) adults owners vehicles val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) 55

Slide 197

Slide 197 text

Philipp Haller More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(...) adults owners vehicles val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) 56

Slide 198

Slide 198 text

Philipp Haller More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(...) adults owners vehicles val sorted = adults.apply(spore { ps => SiloRef.populate(currentHost, ps.sortWith(p => p.age)) }) val labels = sorted.apply(spore { ps => SiloRef.populate(currentHost, ps.map(p => "Hi, " + p.name)) }) sorted labels val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) 57

Slide 199

Slide 199 text

Philipp Haller More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(...) adults owners vehicles sorted labels so far we just staged computation, we haven’t yet “kicked it off”. val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) val sorted = adults.apply(spore { ps => SiloRef.populate(currentHost, ps.sortWith(p => p.age)) }) val labels = sorted.apply(spore { ps => SiloRef.populate(currentHost, ps.map(p => "Hi, " + p.name)) }) 58

Slide 200

Slide 200 text

Philipp Haller More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(...) adults owners vehicles sorted labels val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) val sorted = adults.apply(spore { ps => SiloRef.populate(currentHost, ps.sortWith(p => p.age)) }) val labels = sorted.apply(spore { ps => SiloRef.populate(currentHost, ps.map(p => "Hi, " + p.name)) }) labels.send() 59

Slide 201

Slide 201 text

Philipp Haller More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(...) adults owners vehicles sorted labels λ List[Person]㱺List[String] val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) val sorted = adults.apply(spore { ps => SiloRef.populate(currentHost, ps.sortWith(p => p.age)) }) val labels = sorted.apply(spore { ps => SiloRef.populate(currentHost, ps.map(p => "Hi, " + p.name)) }) labels.send() 59

Slide 202

Slide 202 text

Philipp Haller More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(...) adults owners vehicles sorted labels Silo[List[String]] val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) val sorted = adults.apply(spore { ps => SiloRef.populate(currentHost, ps.sortWith(p => p.age)) }) val labels = sorted.apply(spore { ps => SiloRef.populate(currentHost, ps.map(p => "Hi, " + p.name)) }) labels.send() 59

Slide 203

Slide 203 text

Philipp Haller A functional design for fault-tolerance A SiloRef is a lineage, a persistent (in the sense of functional programming) data structure. The lineage is the DAG of operations used to derive the data of a silo. Since the lineage is composed of spores, it is serializable. This means it can be persisted or transferred to other machines. Putting lineages to work 60

Slide 204

Slide 204 text

Philipp Haller Next: we formalize lineages, a concept from the database + systems communities, in the context of PL. Natural fit in context of functional programming! A functional design for fault-tolerance Putting lineages to work Formalization: typed, distributed core language with spores, silos, and futures. 61

Slide 205

Slide 205 text

Philipp Haller 62 Abstract syntax

Slide 206

Slide 206 text

Philipp Haller 63 Local reduction and lineages

Slide 207

Slide 207 text

Philipp Haller 63 Local reduction and lineages

Slide 208

Slide 208 text

Philipp Haller 64 Distributed reduction

Slide 209

Slide 209 text

Philipp Haller 65 Type assignment

Slide 210

Slide 210 text

Philipp Haller Properties of function passing model Formalization 66

Slide 211

Slide 211 text

Philipp Haller Properties of function passing model Formalization Subject reduction theorem guarantees preservation of types under reduction, as well as preservation of lineage mobility. 66

Slide 212

Slide 212 text

Philipp Haller Properties of function passing model Formalization Subject reduction theorem guarantees preservation of types under reduction, as well as preservation of lineage mobility. Progress theorem guarantees the finite materialization of remote, lineage-based data. 66

Slide 213

Slide 213 text

Philipp Haller Properties of function passing model Formalization Subject reduction theorem guarantees preservation of types under reduction, as well as preservation of lineage mobility. Progress theorem guarantees the finite materialization of remote, lineage-based data. 66 First correctness results for a programming model for lineage-based distributed computation.

Slide 214

Slide 214 text

Philipp Haller Liveness property: finite materialization Properties 67

Slide 215

Slide 215 text

Philipp Haller Liveness property: finite materialization Properties 67

Slide 216

Slide 216 text

Philipp Haller Liveness property: finite materialization Properties 67

Slide 217

Slide 217 text

Philipp Haller Paper Details, proofs, etc. 68 [11] Haller, Miller, and Müller. A programming model and foundation for lineage- based distributed computation. Journal of Functional Programming 28 (2018): e7

Slide 218

Slide 218 text

Philipp Haller Summary 69

Slide 219

Slide 219 text

Philipp Haller Summary • LaCasa: provably safe software isolation, code reuse via object capabilities 69

Slide 220

Slide 220 text

Philipp Haller Summary • LaCasa: provably safe software isolation, code reuse via object capabilities • Reactive Async: practical deterministic concurrency 69

Slide 221

Slide 221 text

Philipp Haller Summary • LaCasa: provably safe software isolation, code reuse via object capabilities • Reactive Async: practical deterministic concurrency – For an imperative, object-oriented language 69

Slide 222

Slide 222 text

Philipp Haller Summary • LaCasa: provably safe software isolation, code reuse via object capabilities • Reactive Async: practical deterministic concurrency – For an imperative, object-oriented language – Type system guarantees quasi-determinism at compile time 69

Slide 223

Slide 223 text

Philipp Haller Summary • LaCasa: provably safe software isolation, code reuse via object capabilities • Reactive Async: practical deterministic concurrency – For an imperative, object-oriented language – Type system guarantees quasi-determinism at compile time • Lineage-based distributed programming 69

Slide 224

Slide 224 text

Philipp Haller Summary • LaCasa: provably safe software isolation, code reuse via object capabilities • Reactive Async: practical deterministic concurrency – For an imperative, object-oriented language – Type system guarantees quasi-determinism at compile time • Lineage-based distributed programming – First correctness results for a lineage-based distributed programming model 69

Slide 225

Slide 225 text

Philipp Haller Summary • LaCasa: provably safe software isolation, code reuse via object capabilities • Reactive Async: practical deterministic concurrency – For an imperative, object-oriented language – Type system guarantees quasi-determinism at compile time • Lineage-based distributed programming – First correctness results for a lineage-based distributed programming model • Finite materialization of distributed, lineage-based data 69

Slide 226

Slide 226 text

Philipp Haller Ongoing and Future Work 70

Slide 227

Slide 227 text

Philipp Haller Ongoing and Future Work 70 Consistency, availability, partition tolerance Determinism Distributed Shared State

Slide 228

Slide 228 text

Philipp Haller Ongoing and Future Work 70 Consistency, availability, partition tolerance Determinism Distributed Shared State [12] Zhao, Haller. Observable atomic consistency for CvRDTs. CoRR abs/1802.09462 (2018)

Slide 229

Slide 229 text

Philipp Haller Ongoing and Future Work 70 Consistency, availability, partition tolerance Determinism Distributed Shared State Security & Privacy Privacy-aware distribution Information- flow security [12] Zhao, Haller. Observable atomic consistency for CvRDTs. CoRR abs/1802.09462 (2018)

Slide 230

Slide 230 text

Philipp Haller Ongoing and Future Work 70 Consistency, availability, partition tolerance Determinism Distributed Shared State Security & Privacy Privacy-aware distribution Information- flow security [13] Salvaneschi, Köhler, Haller, Erdweg, and Mezini. Language-Integrated Privacy-Aware Distributed Queries. 2018, draft [12] Zhao, Haller. Observable atomic consistency for CvRDTs. CoRR abs/1802.09462 (2018)

Slide 231

Slide 231 text

Philipp Haller Ongoing and Future Work 70 Consistency, availability, partition tolerance Determinism Distributed Shared State Security & Privacy Privacy-aware distribution Information- flow security [13] Salvaneschi, Köhler, Haller, Erdweg, and Mezini. Language-Integrated Privacy-Aware Distributed Queries. 2018, draft [12] Zhao, Haller. Observable atomic consistency for CvRDTs. CoRR abs/1802.09462 (2018) Chaos Engineering Testing hypotheses about resilience in production systems

Slide 232

Slide 232 text

Philipp Haller Ongoing and Future Work 70 Consistency, availability, partition tolerance Determinism Distributed Shared State Security & Privacy Privacy-aware distribution Information- flow security [13] Salvaneschi, Köhler, Haller, Erdweg, and Mezini. Language-Integrated Privacy-Aware Distributed Queries. 2018, draft [12] Zhao, Haller. Observable atomic consistency for CvRDTs. CoRR abs/1802.09462 (2018) Chaos Engineering Testing hypotheses about resilience in production systems [14] Zhang, Morin, Haller, Baudry, Monperrus. A Chaos Engineering System for Live Analysis and Falsification of Exception-handling in the JVM. CoRR abs/1805.05246 (2018)

Slide 233

Slide 233 text

Philipp Haller Organization of Scientific Meetings • NII Shonan Meeting:
 Haller, Salvaneschi, Watanabe, Agha.
 "Programming Languages for Distributed Systems", May 27–30, 2019 • Dagstuhl Seminar:
 Haller, Lopes, Markl, Salvaneschi.
 "Programming Languages for Distributed Systems and Distributed Data Management" (65-0618), October 28–31, 2019 71

Slide 234

Slide 234 text

Philipp Haller References • [1]: http://store.steampowered.com/stats/content/ • [2]: https://www.itbusinessedge.com/cm/blogs/lawson/the-big-data-software-problem-behind-cerns-higgs-boson-hunt/?cs=50736 • [3]: https://www.statista.com/statistics/282087/number-of-monthly-active-twitter-users/ • [4] Mark S. Miller. Robust Composition: Towards a Unified Approach to Access Control and Concurrency Control. PhD thesis, 2006 • [5] Haller and Loiko. LaCasa: Lightweight affinity and object capabilities in Scala. OOPSLA 2016 • [6] Erik Reimers. Lightweight Software Isolation via Flow-Sensitive Capabilities in Scala. Master's thesis, KTH, 2017 (supervisor Philipp Haller) • [7] Haller, Sommar. Towards an Empirical Study of Affine Types for Isolated Actors in Scala. PLACES@ETAPS 2017 • [8] Kuper et al. Freeze after writing: quasi-deterministic parallel programming with LVars. POPL 2014 • [9] Haller, Geries, Eichberg, and Salvaneschi. Reactive Async: Expressive Deterministic Concurrency. Scala Symposium 2016 • [10] Miller, Haller, and Odersky. Spores: a type-based foundation for closures in the age of concurrency and distribution. ECOOP 2014 • [11] Haller, Miller, and Müller. A programming model and foundation for lineage-based distributed computation. Journal of Functional Programming 28 (2018): e7 • [12] Zhao, Haller. Observable atomic consistency for CvRDTs. CoRR abs/1802.09462 (2018) • [13] Salvaneschi, Köhler, Haller, Erdweg, and Mezini. Language-Integrated Privacy-Aware Distributed Queries. 2018, draft • [14] Zhang, Morin, Haller, Baudry, Monperrus. A Chaos Engineering System for Live Analysis and Falsification of Exception-handling in the JVM. CoRR abs/1805.05246 (2018) 72

Slide 235

Slide 235 text

Philipp Haller Conclusion 73

Slide 236

Slide 236 text

Philipp Haller Conclusion • Goal:
 Robust construction of large-scale concurrent and distributed systems providing high availability, high scalability, and fault tolerance 73

Slide 237

Slide 237 text

Philipp Haller Conclusion • Goal:
 Robust construction of large-scale concurrent and distributed systems providing high availability, high scalability, and fault tolerance 73 Sound foundations and provable guarantees!

Slide 238

Slide 238 text

Philipp Haller Conclusion • Goal:
 Robust construction of large-scale concurrent and distributed systems providing high availability, high scalability, and fault tolerance • Methods: 73 Sound foundations and provable guarantees!

Slide 239

Slide 239 text

Philipp Haller Conclusion • Goal:
 Robust construction of large-scale concurrent and distributed systems providing high availability, high scalability, and fault tolerance • Methods: – Type systems: theory & practice 73 Sound foundations and provable guarantees!

Slide 240

Slide 240 text

Philipp Haller Conclusion • Goal:
 Robust construction of large-scale concurrent and distributed systems providing high availability, high scalability, and fault tolerance • Methods: – Type systems: theory & practice – Design and implementation of programming systems 73 Sound foundations and provable guarantees!

Slide 241

Slide 241 text

Philipp Haller Conclusion • Goal:
 Robust construction of large-scale concurrent and distributed systems providing high availability, high scalability, and fault tolerance • Methods: – Type systems: theory & practice – Design and implementation of programming systems – Empirical studies 73 Sound foundations and provable guarantees!

Slide 242

Slide 242 text

Philipp Haller Conclusion • Goal:
 Robust construction of large-scale concurrent and distributed systems providing high availability, high scalability, and fault tolerance • Methods: – Type systems: theory & practice – Design and implementation of programming systems – Empirical studies 73 Sound foundations and provable guarantees! Thank You!