Introduction to Building Distributed Systems with Scala and Akka

We explore the reasons for building distributed systems framed around the "fallacies of distributed computing". We contrast those fallacies against the requirements, constraints, and realities of building a modern system today, using Scala and Akka for all examples.

1. Building(Distributed(Systems with%the%Typesafe%Reac/ve%Pla3orm
2. Who$am$I? • Kevin'Webber,'Enterprise'Advocate,'Typesafe • Graduated'from'Georgian'College,'1999 • Java'developer'since'2001'(Java'1.1.8'→'Java'8),'Scala' developer'since'2012 • Previous'roles'include'developer,'team'lead,'and'handsKon' architect • Interviewed'>'100'developers
3. About&Typesafe • Founded(in(2011(in(an.cipa.on(of(a(genera.onal(shi7(to(cloud9 based(compu.ng(by • Mar$n&Odersky,(creator(of(Scala,(Java(Generics,(and(the( current(genera.on(of(javac • Jonas&Bonér,(creator(of(Akka
4. Who$is$Typesafe? • Typesafe)is)a)technology)company)providing)stewardship)for) open7source)so9ware)targe:ng)the)JVM • Scala)7)a)hybrid)object7func:onal)programming)language • Play)7)an)MVC)framework • Akka)7)a)concurrency)toolkit) • Apache,Spark*)7)a)fast)and)general)engine)for)big)data) processing)(streaming,)SQL,)machine)learning,)and)graph) processing)
6. Typesafe's)Customers
7. What%are%distributed%systems?
8. You$have$a$distributed$system$when$ the$crash$of$a$computer$you’ve$ never$heard$of$stops$you$from$ ge8ng$any$work$done. —"Leslie"Lamport:"A"Guide"to"Building"Dependable"Distributed"Systems
9. A"collec'on"of"independent" computers"that"appears"to"its"users" as"a"single"coherent"system. —"Tanenbaum"and"Steen:"Distributed"Systems,"Principles"and" Paradigms
10. Distributed*architecture
11. Why$build$distributed$systems?
12. Size%of%the%internet
13. Market'forces'leading'to'change Concurrent)connec*ons • "Internet(of(Things",(mobile(devices
14. Market'forces'leading'to'change Concurrent)connec*ons • "Internet(of(Things",(mobile(devices Big$data • Size&of&data&is&overwhelming&our&ability&to&manage&it
15. Market'forces'leading'to'change Concurrent)connec*ons • "Internet(of(Things",(mobile(devices Big$data • Size&of&data&is&overwhelming&our&ability&to&manage&it Response'(mes • Real&'me)results)(e.g,)analy'cs))with)sub&second)latencies
16. Physical)factors)leading)to)change • Expensive*hardware*→*cheap&hardware
17. Physical)factors)leading)to)change • Expensive*hardware*→*cheap&hardware • A*single*machine*→*clusters&of&machines
18. Physical)factors)leading)to)change • Expensive*hardware*→*cheap&hardware • A*single*machine*→*clusters&of&machines • A*single*core*→*mul3ple&cores
19. Physical)factors)leading)to)change • Expensive*hardware*→*cheap&hardware • A*single*machine*→*clusters&of&machines • A*single*core*→*mul3ple&cores • Slow*networks*→*fast&networks
20. Physical)factors)leading)to)change • Expensive*hardware*→*cheap&hardware • A*single*machine*→*clusters&of&machines • A*single*core*→*mul3ple&cores • Slow*networks*→*fast&networks • Small*data*snapshots*→*big&data,&streams&of&data
21. 2005$architecture
22. 2015%architecture
23. 2020#architecture
24. New$skills$demanded$by$industry • Local'→'distributed
25. New$skills$demanded$by$industry • Local'→'distributed • Impera.ve'→'func,onal
26. New$skills$demanded$by$industry • Local'→'distributed • Impera.ve'→'func,onal • Security'agnos.c'→'security1aware
27. New$skills$demanded$by$industry • Local'→'distributed • Impera.ve'→'func,onal • Security'agnos.c'→'security1aware • Single'database'→'reads/writes1across1many1data1stores
28. New$skills$demanded$by$industry • Local'→'distributed • Impera.ve'→'func,onal • Security'agnos.c'→'security1aware • Single'database'→'reads/writes1across1many1data1stores • Batch'processing'→'streams1of1data,1real6,me1analy,cs
29. New$skills$demanded$by$industry • Local'→'distributed • Impera.ve'→'func,onal • Security'agnos.c'→'security1aware • Single'database'→'reads/writes1across1many1data1stores • Batch'processing'→'streams1of1data,1real6,me1analy,cs • Ops'→'DevOps
30. Changing'how'we'think'about' building'so2ware
31. Thread'based* concurrency • Shared(mutable(state • Impera1ve,(sequen1al,(side7effec1ng • Threads(for(concurrency • Locks,(monitors,(etc • Implementa)ons( • Java(Enterprise(Edi1on,(Ruby(on(Rails
32. Message&driven, concurrency • Isolated*state • Asynchronous*message*passing* between*mailboxes • Process*isola:on*for*fault*tolerance • Actors*can*live*in*different*thread*pools,* JVMs,*nodes,*or*clusters • Implementa)ons* • Erlang,*Play,*Akka
33. Event&driven*concurrency • Thread(based+event+loops • Typically+single+threaded • Popular+among+JavaScript+frameworks • Can+lead+to+the+pyramid(of(doom • Implementa)ons+ • Node.js,+Twisted+Python,+ EventMachine+(Ruby)
34. Entry&level*Java*developer*perspec2ve
35. Reality
36. Building(so+ware(in(2015 Exploring*the*fallacies*of*distributed*compu6ng,*originally*wri:en* by*L.*Peter*Deutsch*in*1994*at*Sun*Microsystems
37. Fallacies(of(distributed(compu2ng • The$network$is$reliable"→"build"systems"that"func3on"even" when"integrated"systems"go"dark
38. Fallacies(of(distributed(compu2ng • The$network$is$reliable"→"build"systems"that"func3on"even" when"integrated"systems"go"dark • Latency$is$zero"→"there's"no"way"to"change"the"speed"of"light
39. Fallacies(of(distributed(compu2ng • The$network$is$reliable"→"build"systems"that"func3on"even" when"integrated"systems"go"dark • Latency$is$zero"→"there's"no"way"to"change"the"speed"of"light • Bandwidth$is$infinite"→"using"that"fancy"JS"framework"is" wonderful"un3l"it"blocks"page"loading"across"100,000"browsers
40. Fallacies(of(distributed(compu2ng • The$network$is$reliable"→"build"systems"that"func3on"even" when"integrated"systems"go"dark • Latency$is$zero"→"there's"no"way"to"change"the"speed"of"light • Bandwidth$is$infinite"→"using"that"fancy"JS"framework"is" wonderful"un3l"it"blocks"page"loading"across"100,000"browsers • The$network$is$secure"→"integra3on"points"can"introduce" vulnerabili3es
41. Fallacies(of(distributed(compu2ng • Topology'doesn’t'change"→"moving"applica.ons"to"the"cloud" gives"the"ability"to"change"topology"at"will;"add"new"app"servers," scale"on"demand,"etc
42. Fallacies(of(distributed(compu2ng • Topology'doesn’t'change"→"moving"applica.ons"to"the"cloud" gives"the"ability"to"change"topology"at"will;"add"new"app"servers," scale"on"demand,"etc • There'is'one'administrator"→"external"systems"have" performance,"availability,"or"security"issues"that"you"have"no" direct"influence"over
43. Fallacies(of(distributed(compu2ng • Topology'doesn’t'change"→"moving"applica.ons"to"the"cloud" gives"the"ability"to"change"topology"at"will;"add"new"app"servers," scale"on"demand,"etc • There'is'one'administrator"→"external"systems"have" performance,"availability,"or"security"issues"that"you"have"no" direct"influence"over • Transport'cost'is'zero"→"ask"anyone"doing"video"streaming" online"if"transport"costs"are"zero,"transport"is"priced"like"any" commodity
44. Why$func)onal$programming?
45. Side%effect)free)func.ons)enable)parallelism val lastNames = List("Smith","Jones","Frankenstein","Bach","Jackson","Rodin").par // lastNames: scala.collection.parallel.immutable.ParSeq[String] = // ParVector(Smith, Jones, Frankenstein, Bach, Jackson, Rodin) val res0 = lastNames.map(_.toUpperCase) // toUpperCase, higher-order function // res0: scala.collection.parallel.immutable.ParSeq[String] = // ParVector(SMITH, JONES, FRANKENSTEIN, BACH, JACKSON, RODIN) • Crea&ng)a)parallel)collec&on)is)simple • Parallel)opera&ons)are)implemented)with)divide)and)conquer) style)algorithms)that)parallelize)well
46. Example(of(a(chained(transforma3on(and(filter coll.map(_.toUpperCase).filter(predicate)
47. Enable'concurrency'in'a'transparent'way import scala.util.{Success, Failure} // obtain a future list of recent posts // executes on a different thread pool val f: Future[List[String]] = future { session.getRecentPosts } // success and failure callbacks f onComplete { case Success(posts) => for (post <- posts) println(post) case Failure(t) => println("An error has occured: " + t.getMessage) }
48. Message&passing&to&a&remote&worker val remoteGameActor = system.actorOf( GameActor.props.withDeploy(Deploy(scope = RemoteScope(address))), name = “gameActor”) val remoteGameActor = context.actorSelection( “akka.tcp://actorSystemName@10.0.0.1:2552/gameActor") remoteGameActor ! message
49. Loca%on'transparency
50. Easier'to'maintain'and'reason'about Scala%enables%DRY%code%with%the%following%features: • Higher(order+func0ons • Func0onal+composi0on+&+collec0ons+with+map,+filter,+reduce,+etc • Case+classes • Pa>ern+matching • Trait+composi0on+ • Etc...
51. As#of#April#7th,#2015,#29#jobs#are# posted#on#LinkedIn#for#Apple# requiring#Scala#or#other#funcAonal# programming#experience.
52. Web$Services$Engineer,$Apple Key$Qualifica,ons • Experience*in*developing*web2services*in*a*commercial*se6ng* and*familiar*with*HTTP,*REST,*JSON • Know*how*to*write*reliable,*resilient,*unit2tested,*performant,* baEle2tested*code • Deep*experience*in*any*one*of*Java,*Scala,*C*,*C#,*F# • Familiar*with*funcKonal*and*reacKve*programming*concepts
53. Do#I#expect#College#graduates#to#be#qualified#for#fully#qualified#to# build#distributed#systems#right#out#of#school? • No!%Building%distributed%systems%is%difficult%and%takes%years%of% experience%to%design%and%implement
54. Do#I#expect#College#graduates#to#be#qualified#for#fully#qualified#to# build#distributed#systems#right#out#of#school? • No!%Building%distributed%systems%is%difficult%and%takes%years%of% experience%to%design%and%implement • Building%blocks%can%be%taught%at%the%College%level,%posi?oning% students%for%future%on@the@job%learning,%e.g,%func?onal,%side@ effect@free%programming
55. Do#I#expect#College#graduates#to#be#qualified#for#fully#qualified#to# build#distributed#systems#right#out#of#school? • No!%Building%distributed%systems%is%difficult%and%takes%years%of% experience%to%design%and%implement • Building%blocks%can%be%taught%at%the%College%level,%posi?oning% students%for%future%on@the@job%learning,%e.g,%func?onal,%side@ effect@free%programming • Teaching%"mechanical%sympathy",%i.e,%to%understand%the%physical% limita?ons%of%machines/networks%and%how%soHware%is%deployed% and%executed
56. Interested(in(learning(more? • What%Is%Reac+ve%Programming?"#"h%p:/ /bit.ly/1Ay3lHR • Reac+veTO"#"h%p:/ /www.meetup.com/Reac;ve#TO • Func+onal%programming%in%Scala%(Coursera)"#"h%ps:/ / www.coursera.org/course/progfun • Typesafe%Ac+vator"#"h%ps:/ /typesafe.com/community/core# tools/ac;vator#and#sbt
57. Ques%ons?