A Foundation for Lineage-Based Distributed Computation Philipp Haller
 KTH Royal Institute of Technology Stockholm, Sweden Scala Stockholm
 November 21st, 2017 1 Joint work with Heather Miller and Normen Müller 

Distributed Programming is Everywhere! Large-scale web applications, IoT applications, serverless computing, etc. Distribution essential for: Resilience 2 Elasticity (subsumes scalability) Physically distributed systems 

Shameless Plug 3 CHAOS ENGINEERING & DEVOPS MEETUP @KTH STOCKHOLM, DEC 6 2017 REGISTER AT: https://www.chaos.conf.kth.se/ 

Functional Programming Gaining Ground in Distributed Systems MapReduce (2004), Apache Hadoop MapReduce (2011) DryadLINQ (2008), Apache Spark (2010), Twitter Scalding (2012), Apache Flink (2014), Akka Streams (2015), … 4 embrace FP concepts and FP language features do not embrace functional programming features embrace functional programming concepts 

Where's the Problem? Little reuse across systems and frameworks Complexity 5 E.g., Spark 2.2.0 (just core): 94922 SLOC (Scala) Reusable components rather large: HDFS, Mesos, … E.g., each system implements fault recovery & tolerance No formal guarantees Safety, e.g.: never crashes due to ClassCastException Liveness, e.g.: operations complete eventually 

Goals of a New Substrate for Distributed Programming Reusable abstractions Simplicity 6 Small core E.g., fault recovery less monolithic Formal guarantees Safety, e.g.: never crashes due to ClassCastException Liveness, e.g.: operations complete eventually Small reusable components 

A Foundation for Lineage-Based Distributed Computation 7 

High-level picture: wikipedia reduced, 48.4GB 8 

High-level picture: wikipedia reduced, 48.4GB Chunk up the data… 9 

High-level picture: Distribute it over your cluster of machines. 10 

High-level picture: From there, think of your distributed data like a single collection... wiki val wiki: RDD[WikiArticle] = ... wiki.map { article => article.text.toLowerCase } Example: Transform the text of all wiki articles to lowercase. 11 

Then, why do we build these systems using RPC or message passing? 2) Fault recovery not a natural fit 1) Computational pattern: send functions to data 12 

Idea: Capitalize on the structure of the problem: Simplifies fault tolerance by design Functional data structure falls out of this 13 

Distributed Programming with Functional Lineages a.k.a. Function Passing Key idea: inversion of the actor model. New data-centric programming model for functional processing of distributed data. 14 

Key idea: Inversion of the actor model. Actors: Encapsulate state and behavior. Are stationary. (Only mobile references to actors.) Actors exchange data/commands through asynchronous messaging. 15 

Key idea: Inversion of the actor model. Actors: keep functionality stationary, send data. Function passing: keep data stationary, send functionality. 16 

Key idea: Inversion of the actor model. Function passing: Stateless. Persistent data structures. Keep data stationary. Functions are exchanged through asynchronous messaging. 17 

The Function Passing Model Introducing Consists of 3 parts: Silos: stationary, typed, immutable data containers SiloRefs: references to local or remote Silos. Spores: safe, serializable functions. 18 

The Function Passing Model Some Visual Intuition of Silo SiloRef Master Worker 19 

Silos What are they? Silo[T] T SiloRef[T] Two parts. def apply def send def persist def unpersist SiloRef. Handle to a Silo. Silo. Typed, stationary data container. User interacts with SiloRef. SiloRefs come with 4 primitive operations. 20 

Silos What are they? Silo[T] T SiloRef[T] Primitive: apply Takes a function that is to be applied to the data in the silo associated with the SiloRef. Creates new silo to contain the data that the user- defined function returns; evaluation is deferred def apply[S](fun: T => SiloRef[S]): SiloRef[S] Enables interesting computation DAGs Deferred def apply def send def persist def unpersist 21 

Silos What are they? Silo[T] T SiloRef[T] Primitive: send Forces the built-up computation DAG to be sent to the associated node and applied. Future is completed with the result of the computation. def send(): Future[T] EAGER def apply def send def persist def unpersist 22 

Silos What are they? Silo[T] T SiloRef[T] Primitive: persist Ensures silo is cached in memory. def persist(): SiloRef[T] def apply def send def persist def unpersist Deferred 23 

Silos What are they? Silo[T] T SiloRef[T] Primitive: unpersist Enables silo to be removed from memory. def unpersist(): SiloRef[T] def apply def send def persist def unpersist Deferred 24 

Silos Silo[T] T SiloRef[T] Silo factories: Creates silo on given host containing given value/text file/… object SiloRef { def populate[T](host: Host, value: T): SiloRef[T] def fromTextFile(host: Host, file: File): SiloRef[List[String]] ... } def apply def send def persist def unpersist Deferred What are they? 25 

) Basic idea: apply/send Silo[T] Machine 1 Machine 2 SiloRef[T] λ T SiloRef[S] S Silo[S] ) T㱺SiloRef[S] 26 

The Problem with Closures Distributing Functions class MyCoolApp { val param = 42 val log = new Log(...) ... def work(silo: SiloRef[Int]) = { silo.apply(x => SiloRef.populate(currentHost, x + param) ).send() } } 27 

The Problem with Closures Distributing Functions class MyCoolApp { val param = 42 val log = new Log(...) ... def work(silo: SiloRef[Int]) = { silo.apply(x => SiloRef.populate(currentHost, x + this.param) ).send() } } Accidental capture of a non-serializable object. 28 

The Problem with Closures Distributing Functions class MyCoolApp { val param = 42 val log = new Log(...) ... def work(silo: SiloRef[Int]) = { silo.apply(x => SiloRef.populate(currentHost, x + this.param) ).send() } } Accidental capture of a non-serializable object. 29 

The Problem with Closures: Solution Distributing Functions class MyCoolApp { val param = 42 val log = new Log(...) ... def work(silo: SiloRef[Int]) = { silo.apply(spore { val localParam = this.param x => SiloRef.populate(currentHost, x + localParam) }).send() } } Spore header Spore body 30 See https://github.com/scalacenter/spores Copy just an Int into environment of spore 

More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... 31 

More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) adults 32 

More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(spore { val localVehicles = vehicles // spore header ps => localVehicles.apply(spore { val localps = ps // spore header vs => SiloRef.populate(currentHost, localps.flatMap(p => // list of (p, v) for a single person p vs.flatMap { v => if (v.owner.name == p.name) List((p, v)) else Nil } ) adults owners vehicles val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) 33 

More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(...) adults owners vehicles val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) 34 

More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(...) adults owners vehicles val sorted = adults.apply(spore { ps => SiloRef.populate(currentHost, ps.sortWith(p => p.age)) }) val labels = sorted.apply(spore { ps => SiloRef.populate(currentHost, ps.map(p => "Hi, " + p.name)) }) sorted labels val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) 35 

More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(...) adults owners vehicles sorted labels so far we just staged computation, we haven’t yet “kicked it off”. val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) val sorted = adults.apply(spore { ps => SiloRef.populate(currentHost, ps.sortWith(p => p.age)) }) val labels = sorted.apply(spore { ps => SiloRef.populate(currentHost, ps.map(p => "Hi, " + p.name)) }) 36 

More involved example Silo[List[Person]] Machine 1 SiloRef[List[Person]] Let’s make an interesting DAG! Machine 2 persons: val persons: SiloRef[List[Person]] = ... val vehicles: SiloRef[List[Vehicle]] = ... // adults that own a vehicle val owners = adults.apply(...) adults owners vehicles sorted labels λ List[Person]㱺List[String] Silo[List[String]] val adults = persons.apply(spore { ps => val res = ps.filter(p => p.age >= 18) SiloRef.populate(currentHost, res) }) val sorted = adults.apply(spore { ps => SiloRef.populate(currentHost, ps.sortWith(p => p.age)) }) val labels = sorted.apply(spore { ps => SiloRef.populate(currentHost, ps.map(p => "Hi, " + p.name)) }) labels.persist().send() 37 

A functional design for fault-tolerance Silos and SiloRefs relate to each other by means of lineages, persistent data structures. The lineage is the DAG of operations used to derive the data of each silo. Since the lineage is composed of spores, it is serializable. This means it can be persisted or transferred to other machines. Putting lineages to work 38 

Next: we formalize lineages, a concept from the database + systems communities, in the context of PL. Natural fit in context of functional programming! Intuition: Spores & SiloRefs are safe to serialize. Therefore, we can save entire DAGs, share them, and use them to restart computations. A functional design for fault-tolerance Putting lineages to work Formalization: typed, distributed core language with spores, silos, and futures. 39 

Mathematics of function passing 40 

41 Formal syntax of core language 

42 Lineages, formally 

Properties of function passing model Formalization Subject reduction theorem guarantees preservation of types under reduction, as well as preservation of lineage mobility. Progress theorem guarantees the finite materialization of remote, lineage-based data. First correctness results for a programming model for lineage-based distributed computation. 43 Both theorems (also) imply: "no ClassCastException" 

Building applications with function passing Built two miniaturized example systems inspired by popular big data frameworks. BabySpark MBrace Implemented Spark RDD operators in terms of the primitives of function passing: map, reduce, groupBy, and join Emulated MBrace using the primitives of function passing. (distributed collections) (F# async for distributing tasks) 44 See https://github.com/heathermiller/f-p/ 

Revisiting safety Preventing unsafe state Accessing global, mutable state from within silos is undefined and meaningless. Additional static checking required to prevent undefined accesses. Proposal: objects put into silos must conform to the object capability model. 45 

Object capability model Preventing unsafe state Methods (including constructors) only access parameters and this. Methods (including constructors) only instantiate "ocap" classes. Types of fields and method parameters are "ocap". 46 A class is "ocap" (conforms to the object capability model) if: Mark S. Miller. Robust Composition: Towards a Unified Approach to Access Control and Concurrency Control. PhD thesis, 2006 

Object capability model in Scala Empirical results LaCasa: Scala extension implementing the object capability model Empirical results show: many existing class definitions conform to the object capability model. Project #classes/traits #ocap (%) #dir. insec. (%) Scala stdlib 1,505 644 (43%) 212/861 (25%) Signal/Collect 236 159 (67%) 60/77 (78%) GeoTrellis -engine 190 40 (21%) 124/150 (83%) -raster 670 233 (35%) 325/437 (74%) -spark 326 101 (31%) 167/225 (74%) Total 2,927 1,177 (40%) 888/1,750 (51%) 47 See https://github.com/phaller/lacasa 

Find out more! References Spores: safe, serializable closures Function passing: Object capabilities and affine types in Scala: Haller, Miller, and Müller. A Programming Model and Foundation for Lineage-Based Distributed Computation. 2017. Draft: https://infoscience.epfl.ch/record/230304 Miller, Haller, and Odersky. Spores: a type-based foundation for closures in the age of concurrency and distribution. ECOOP 2014 Miller, Haller, Müller, and Boullier. Function passing: a model for typed, distributed functional programming. Onward! 2016 Haller and Loiko. LaCasa: lightweight affinity and object capabilities in Scala. OOPSLA 2016 48 

Conclusions Lessons learnt Let's reduce the granularity of reuseable building blocks of distributed systems! The function passing model is a first step towards first-class lineages. Mathematical model of function passing has desireable properties. First experience with an early prototype. 49 Slides: www.csc.kth.se/~phaller/ or https://speakerdeck.com/phaller/