Introduction to Building Distributed Systems with Scala and Akka

We explore the reasons for building distributed systems framed around the "fallacies of distributed computing". We contrast those fallacies against the requirements, constraints, and realities of building a modern system today, using Scala and Akka for all examples.

Transcript

1. Building(Distributed(Systems
   with%the%Typesafe%Reac/ve%Pla3orm
   

   View Slide

2. Who$am$I?
   • Kevin'Webber,'Enterprise'Advocate,'Typesafe
   • Graduated'from'Georgian'College,'1999
   • Java'developer'since'2001'(Java'1.1.8'→'Java'8),'Scala'
   developer'since'2012
   • Previous'roles'include'developer,'team'lead,'and'handsKon'
   architect
   • Interviewed'>'100'developers
   

   View Slide

3. About&Typesafe
   • Founded(in(2011(in(an.cipa.on(of(a(genera.onal(shi7(to(cloud9
   based(compu.ng(by
   • Mar$n&Odersky,(creator(of(Scala,(Java(Generics,(and(the(
   current(genera.on(of(javac
   • Jonas&Bonér,(creator(of(Akka
   

   View Slide

4. Who$is$Typesafe?
   • Typesafe)is)a)technology)company)providing)stewardship)for)
   open7source)so9ware)targe:ng)the)JVM
   • Scala)7)a)hybrid)object7func:onal)programming)language
   • Play)7)an)MVC)framework
   • Akka)7)a)concurrency)toolkit)
   • Apache,Spark*)7)a)fast)and)general)engine)for)big)data)
   processing)(streaming,)SQL,)machine)learning,)and)graph)
   processing)
   

   View Slide

5. View Slide

6. Typesafe's)Customers
   

   View Slide

7. What%are%distributed%systems?
   

   View Slide

8. You$have$a$distributed$system$when$
   the$crash$of$a$computer$you’ve$
   never$heard$of$stops$you$from$
   ge8ng$any$work$done.
   —"Leslie"Lamport:"A"Guide"to"Building"Dependable"Distributed"Systems
   

   View Slide

9. A"collec'on"of"independent"
   computers"that"appears"to"its"users"
   as"a"single"coherent"system.
   —"Tanenbaum"and"Steen:"Distributed"Systems,"Principles"and"
   Paradigms
   

   View Slide

10. Distributed*architecture
    

    View Slide

11. Why$build$distributed$systems?
    

    View Slide

12. Size%of%the%internet
    

    View Slide

13. Market'forces'leading'to'change
    Concurrent)connec*ons
    • "Internet(of(Things",(mobile(devices
    

    View Slide

14. Market'forces'leading'to'change
    Concurrent)connec*ons
    • "Internet(of(Things",(mobile(devices
    Big$data
    • Size&of&data&is&overwhelming&our&ability&to&manage&it
    

    View Slide

15. Market'forces'leading'to'change
    Concurrent)connec*ons
    • "Internet(of(Things",(mobile(devices
    Big$data
    • Size&of&data&is&overwhelming&our&ability&to&manage&it
    Response'(mes
    • Real&'me)results)(e.g,)analy'cs))with)sub&second)latencies
    

    View Slide

16. Physical)factors)leading)to)change
    • Expensive*hardware*→*cheap&hardware
    

    View Slide

17. Physical)factors)leading)to)change
    • Expensive*hardware*→*cheap&hardware
    • A*single*machine*→*clusters&of&machines
    

    View Slide

18. Physical)factors)leading)to)change
    • Expensive*hardware*→*cheap&hardware
    • A*single*machine*→*clusters&of&machines
    • A*single*core*→*mul3ple&cores
    

    View Slide

19. Physical)factors)leading)to)change
    • Expensive*hardware*→*cheap&hardware
    • A*single*machine*→*clusters&of&machines
    • A*single*core*→*mul3ple&cores
    • Slow*networks*→*fast&networks
    

    View Slide

20. Physical)factors)leading)to)change
    • Expensive*hardware*→*cheap&hardware
    • A*single*machine*→*clusters&of&machines
    • A*single*core*→*mul3ple&cores
    • Slow*networks*→*fast&networks
    • Small*data*snapshots*→*big&data,&streams&of&data
    

    View Slide

21. 2005$architecture
    

    View Slide

22. 2015%architecture
    

    View Slide

23. 2020#architecture
    

    View Slide

24. New$skills$demanded$by$industry
    • Local'→'distributed
    

    View Slide

25. New$skills$demanded$by$industry
    • Local'→'distributed
    • Impera.ve'→'func,onal
    

    View Slide

26. New$skills$demanded$by$industry
    • Local'→'distributed
    • Impera.ve'→'func,onal
    • Security'agnos.c'→'security1aware
    

    View Slide

27. New$skills$demanded$by$industry
    • Local'→'distributed
    • Impera.ve'→'func,onal
    • Security'agnos.c'→'security1aware
    • Single'database'→'reads/writes1across1many1data1stores
    

    View Slide

28. New$skills$demanded$by$industry
    • Local'→'distributed
    • Impera.ve'→'func,onal
    • Security'agnos.c'→'security1aware
    • Single'database'→'reads/writes1across1many1data1stores
    • Batch'processing'→'streams1of1data,1real6,me1analy,cs
    

    View Slide

29. New$skills$demanded$by$industry
    • Local'→'distributed
    • Impera.ve'→'func,onal
    • Security'agnos.c'→'security1aware
    • Single'database'→'reads/writes1across1many1data1stores
    • Batch'processing'→'streams1of1data,1real6,me1analy,cs
    • Ops'→'DevOps
    

    View Slide

30. Changing'how'we'think'about'
    building'so2ware
    

    View Slide

31. Thread'based*
    concurrency
    • Shared(mutable(state
    • Impera1ve,(sequen1al,(side7effec1ng
    • Threads(for(concurrency
    • Locks,(monitors,(etc
    • Implementa)ons(
    • Java(Enterprise(Edi1on,(Ruby(on(Rails
    

    View Slide

32. Message&driven,
    concurrency
    • Isolated*state
    • Asynchronous*message*passing*
    between*mailboxes
    • Process*isola:on*for*fault*tolerance
    • Actors*can*live*in*different*thread*pools,*
    JVMs,*nodes,*or*clusters
    • Implementa)ons*
    • Erlang,*Play,*Akka
    

    View Slide

33. Event&driven*concurrency
    • Thread(based+event+loops
    • Typically+single+threaded
    • Popular+among+JavaScript+frameworks
    • Can+lead+to+the+pyramid(of(doom
    • Implementa)ons+
    • Node.js,+Twisted+Python,+
    EventMachine+(Ruby)
    

    View Slide

34. Entry&level*Java*developer*perspec2ve
    

    View Slide

35. Reality
    

    View Slide

36. Building(so+ware(in(2015
    Exploring*the*fallacies*of*distributed*compu6ng,*originally*wri:en*
    by*L.*Peter*Deutsch*in*1994*at*Sun*Microsystems
    

    View Slide

37. Fallacies(of(distributed(compu2ng
    • The$network$is$reliable"→"build"systems"that"func3on"even"
    when"integrated"systems"go"dark
    

    View Slide

38. Fallacies(of(distributed(compu2ng
    • The$network$is$reliable"→"build"systems"that"func3on"even"
    when"integrated"systems"go"dark
    • Latency$is$zero"→"there's"no"way"to"change"the"speed"of"light
    

    View Slide

39. Fallacies(of(distributed(compu2ng
    • The$network$is$reliable"→"build"systems"that"func3on"even"
    when"integrated"systems"go"dark
    • Latency$is$zero"→"there's"no"way"to"change"the"speed"of"light
    • Bandwidth$is$infinite"→"using"that"fancy"JS"framework"is"
    wonderful"un3l"it"blocks"page"loading"across"100,000"browsers
    

    View Slide

40. Fallacies(of(distributed(compu2ng
    • The$network$is$reliable"→"build"systems"that"func3on"even"
    when"integrated"systems"go"dark
    • Latency$is$zero"→"there's"no"way"to"change"the"speed"of"light
    • Bandwidth$is$infinite"→"using"that"fancy"JS"framework"is"
    wonderful"un3l"it"blocks"page"loading"across"100,000"browsers
    • The$network$is$secure"→"integra3on"points"can"introduce"
    vulnerabili3es
    

    View Slide

41. Fallacies(of(distributed(compu2ng
    • Topology'doesn’t'change"→"moving"applica.ons"to"the"cloud"
    gives"the"ability"to"change"topology"at"will;"add"new"app"servers,"
    scale"on"demand,"etc
    

    View Slide

42. Fallacies(of(distributed(compu2ng
    • Topology'doesn’t'change"→"moving"applica.ons"to"the"cloud"
    gives"the"ability"to"change"topology"at"will;"add"new"app"servers,"
    scale"on"demand,"etc
    • There'is'one'administrator"→"external"systems"have"
    performance,"availability,"or"security"issues"that"you"have"no"
    direct"influence"over
    

    View Slide

43. Fallacies(of(distributed(compu2ng
    • Topology'doesn’t'change"→"moving"applica.ons"to"the"cloud"
    gives"the"ability"to"change"topology"at"will;"add"new"app"servers,"
    scale"on"demand,"etc
    • There'is'one'administrator"→"external"systems"have"
    performance,"availability,"or"security"issues"that"you"have"no"
    direct"influence"over
    • Transport'cost'is'zero"→"ask"anyone"doing"video"streaming"
    online"if"transport"costs"are"zero,"transport"is"priced"like"any"
    commodity
    

    View Slide

44. Why$func)onal$programming?
    

    View Slide

45. Side%effect)free)func.ons)enable)parallelism
    val lastNames = List("Smith","Jones","Frankenstein","Bach","Jackson","Rodin").par
    // lastNames: scala.collection.parallel.immutable.ParSeq[String] =
    // ParVector(Smith, Jones, Frankenstein, Bach, Jackson, Rodin)
    val res0 = lastNames.map(_.toUpperCase) // toUpperCase, higher-order function
    // res0: scala.collection.parallel.immutable.ParSeq[String] =
    // ParVector(SMITH, JONES, FRANKENSTEIN, BACH, JACKSON, RODIN)
    • Crea&ng)a)parallel)collec&on)is)simple
    • Parallel)opera&ons)are)implemented)with)divide)and)conquer)
    style)algorithms)that)parallelize)well
    

    View Slide

46. Example(of(a(chained(transforma3on(and(filter
    coll.map(_.toUpperCase).filter(predicate)
    

    View Slide

47. Enable'concurrency'in'a'transparent'way
    import scala.util.{Success, Failure}
    // obtain a future list of recent posts
    // executes on a different thread pool
    val f: Future[List[String]] = future {
    session.getRecentPosts
    }
    // success and failure callbacks
    f onComplete {
    case Success(posts) => for (post <- posts) println(post)
    case Failure(t) => println("An error has occured: " + t.getMessage)
    }
    

    View Slide

48. Message&passing&to&a&remote&worker
    val remoteGameActor = system.actorOf(
    GameActor.props.withDeploy(Deploy(scope = RemoteScope(address))),
    name = “gameActor”)
    val remoteGameActor = context.actorSelection(
    “akka.tcp://actorSystemName@10.0.0.1:2552/gameActor")
    remoteGameActor ! message
    

    View Slide

49. Loca%on'transparency
    

    View Slide

50. Easier'to'maintain'and'reason'about
    Scala%enables%DRY%code%with%the%following%features:
    • Higher(order+func0ons
    • Func0onal+composi0on+&+collec0ons+with+map,+filter,+reduce,+etc
    • Case+classes
    • Pa>ern+matching
    • Trait+composi0on+
    • Etc...
    

    View Slide

51. As#of#April#7th,#2015,#29#jobs#are#
    posted#on#LinkedIn#for#Apple#
    requiring#Scala#or#other#funcAonal#
    programming#experience.
    

    View Slide

52. Web$Services$Engineer,$Apple
    Key$Qualifica,ons
    • Experience*in*developing*web2services*in*a*commercial*se6ng*
    and*familiar*with*HTTP,*REST,*JSON
    • Know*how*to*write*reliable,*resilient,*unit2tested,*performant,*
    baEle2tested*code
    • Deep*experience*in*any*one*of*Java,*Scala,*C*,*C#,*F#
    • Familiar*with*funcKonal*and*reacKve*programming*concepts
    

    View Slide

53. Do#I#expect#College#graduates#to#be#qualified#for#fully#qualified#to#
    build#distributed#systems#right#out#of#school?
    • No!%Building%distributed%systems%is%difficult%and%takes%years%of%
    experience%to%design%and%implement
    

    View Slide

54. Do#I#expect#College#graduates#to#be#qualified#for#fully#qualified#to#
    build#distributed#systems#right#out#of#school?
    • No!%Building%distributed%systems%is%difficult%and%takes%years%of%
    experience%to%design%and%implement
    • Building%blocks%can%be%taught%at%the%College%level,%posi?oning%
    students%for%future%on@the@job%learning,%e.g,%func?onal,%side@
    effect@free%programming
    

    View Slide

55. Do#I#expect#College#graduates#to#be#qualified#for#fully#qualified#to#
    build#distributed#systems#right#out#of#school?
    • No!%Building%distributed%systems%is%difficult%and%takes%years%of%
    experience%to%design%and%implement
    • Building%blocks%can%be%taught%at%the%College%level,%posi?oning%
    students%for%future%on@the@job%learning,%e.g,%func?onal,%side@
    effect@free%programming
    • Teaching%"mechanical%sympathy",%i.e,%to%understand%the%physical%
    limita?ons%of%machines/networks%and%how%soHware%is%deployed%
    and%executed
    

    View Slide

56. Interested(in(learning(more?
    • What%Is%Reac+ve%Programming?"#"h%p:/
    /bit.ly/1Ay3lHR
    • Reac+veTO"#"h%p:/
    /www.meetup.com/Reac;ve#TO
    • Func+onal%programming%in%Scala%(Coursera)"#"h%ps:/
    /
    www.coursera.org/course/progfun
    • Typesafe%Ac+vator"#"h%ps:/
    /typesafe.com/community/core#
    tools/ac;vator#and#sbt
    

    View Slide

57. Ques%ons?
    

    View Slide