Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Building Distributed Systems with Scala and Akka

Introduction to Building Distributed Systems with Scala and Akka

We explore the reasons for building distributed systems framed around the "fallacies of distributed computing". We contrast those fallacies against the requirements, constraints, and realities of building a modern system today, using Scala and Akka for all examples.

Kevin Webber

April 09, 2015
Tweet

More Decks by Kevin Webber

Other Decks in Programming

Transcript

  1. Building(Distributed(Systems
    with%the%Typesafe%Reac/ve%Pla3orm

    View Slide

  2. Who$am$I?
    • Kevin'Webber,'Enterprise'Advocate,'Typesafe
    • Graduated'from'Georgian'College,'1999
    • Java'developer'since'2001'(Java'1.1.8'→'Java'8),'Scala'
    developer'since'2012
    • Previous'roles'include'developer,'team'lead,'and'handsKon'
    architect
    • Interviewed'>'100'developers

    View Slide

  3. About&Typesafe
    • Founded(in(2011(in(an.cipa.on(of(a(genera.onal(shi7(to(cloud9
    based(compu.ng(by
    • Mar$n&Odersky,(creator(of(Scala,(Java(Generics,(and(the(
    current(genera.on(of(javac
    • Jonas&Bonér,(creator(of(Akka

    View Slide

  4. Who$is$Typesafe?
    • Typesafe)is)a)technology)company)providing)stewardship)for)
    open7source)so9ware)targe:ng)the)JVM
    • Scala)7)a)hybrid)object7func:onal)programming)language
    • Play)7)an)MVC)framework
    • Akka)7)a)concurrency)toolkit)
    • Apache,Spark*)7)a)fast)and)general)engine)for)big)data)
    processing)(streaming,)SQL,)machine)learning,)and)graph)
    processing)

    View Slide

  5. View Slide

  6. Typesafe's)Customers

    View Slide

  7. What%are%distributed%systems?

    View Slide

  8. You$have$a$distributed$system$when$
    the$crash$of$a$computer$you’ve$
    never$heard$of$stops$you$from$
    ge8ng$any$work$done.
    —"Leslie"Lamport:"A"Guide"to"Building"Dependable"Distributed"Systems

    View Slide

  9. A"collec'on"of"independent"
    computers"that"appears"to"its"users"
    as"a"single"coherent"system.
    —"Tanenbaum"and"Steen:"Distributed"Systems,"Principles"and"
    Paradigms

    View Slide

  10. Distributed*architecture

    View Slide

  11. Why$build$distributed$systems?

    View Slide

  12. Size%of%the%internet

    View Slide

  13. Market'forces'leading'to'change
    Concurrent)connec*ons
    • "Internet(of(Things",(mobile(devices

    View Slide

  14. Market'forces'leading'to'change
    Concurrent)connec*ons
    • "Internet(of(Things",(mobile(devices
    Big$data
    • Size&of&data&is&overwhelming&our&ability&to&manage&it

    View Slide

  15. Market'forces'leading'to'change
    Concurrent)connec*ons
    • "Internet(of(Things",(mobile(devices
    Big$data
    • Size&of&data&is&overwhelming&our&ability&to&manage&it
    Response'(mes
    • Real&'me)results)(e.g,)analy'cs))with)sub&second)latencies

    View Slide

  16. Physical)factors)leading)to)change
    • Expensive*hardware*→*cheap&hardware

    View Slide

  17. Physical)factors)leading)to)change
    • Expensive*hardware*→*cheap&hardware
    • A*single*machine*→*clusters&of&machines

    View Slide

  18. Physical)factors)leading)to)change
    • Expensive*hardware*→*cheap&hardware
    • A*single*machine*→*clusters&of&machines
    • A*single*core*→*mul3ple&cores

    View Slide

  19. Physical)factors)leading)to)change
    • Expensive*hardware*→*cheap&hardware
    • A*single*machine*→*clusters&of&machines
    • A*single*core*→*mul3ple&cores
    • Slow*networks*→*fast&networks

    View Slide

  20. Physical)factors)leading)to)change
    • Expensive*hardware*→*cheap&hardware
    • A*single*machine*→*clusters&of&machines
    • A*single*core*→*mul3ple&cores
    • Slow*networks*→*fast&networks
    • Small*data*snapshots*→*big&data,&streams&of&data

    View Slide

  21. 2005$architecture

    View Slide

  22. 2015%architecture

    View Slide

  23. 2020#architecture

    View Slide

  24. New$skills$demanded$by$industry
    • Local'→'distributed

    View Slide

  25. New$skills$demanded$by$industry
    • Local'→'distributed
    • Impera.ve'→'func,onal

    View Slide

  26. New$skills$demanded$by$industry
    • Local'→'distributed
    • Impera.ve'→'func,onal
    • Security'agnos.c'→'security1aware

    View Slide

  27. New$skills$demanded$by$industry
    • Local'→'distributed
    • Impera.ve'→'func,onal
    • Security'agnos.c'→'security1aware
    • Single'database'→'reads/writes1across1many1data1stores

    View Slide

  28. New$skills$demanded$by$industry
    • Local'→'distributed
    • Impera.ve'→'func,onal
    • Security'agnos.c'→'security1aware
    • Single'database'→'reads/writes1across1many1data1stores
    • Batch'processing'→'streams1of1data,1real6,me1analy,cs

    View Slide

  29. New$skills$demanded$by$industry
    • Local'→'distributed
    • Impera.ve'→'func,onal
    • Security'agnos.c'→'security1aware
    • Single'database'→'reads/writes1across1many1data1stores
    • Batch'processing'→'streams1of1data,1real6,me1analy,cs
    • Ops'→'DevOps

    View Slide

  30. Changing'how'we'think'about'
    building'so2ware

    View Slide

  31. Thread'based*
    concurrency
    • Shared(mutable(state
    • Impera1ve,(sequen1al,(side7effec1ng
    • Threads(for(concurrency
    • Locks,(monitors,(etc
    • Implementa)ons(
    • Java(Enterprise(Edi1on,(Ruby(on(Rails

    View Slide

  32. Message&driven,
    concurrency
    • Isolated*state
    • Asynchronous*message*passing*
    between*mailboxes
    • Process*isola:on*for*fault*tolerance
    • Actors*can*live*in*different*thread*pools,*
    JVMs,*nodes,*or*clusters
    • Implementa)ons*
    • Erlang,*Play,*Akka

    View Slide

  33. Event&driven*concurrency
    • Thread(based+event+loops
    • Typically+single+threaded
    • Popular+among+JavaScript+frameworks
    • Can+lead+to+the+pyramid(of(doom
    • Implementa)ons+
    • Node.js,+Twisted+Python,+
    EventMachine+(Ruby)

    View Slide

  34. Entry&level*Java*developer*perspec2ve

    View Slide

  35. Reality

    View Slide

  36. Building(so+ware(in(2015
    Exploring*the*fallacies*of*distributed*compu6ng,*originally*wri:en*
    by*L.*Peter*Deutsch*in*1994*at*Sun*Microsystems

    View Slide

  37. Fallacies(of(distributed(compu2ng
    • The$network$is$reliable"→"build"systems"that"func3on"even"
    when"integrated"systems"go"dark

    View Slide

  38. Fallacies(of(distributed(compu2ng
    • The$network$is$reliable"→"build"systems"that"func3on"even"
    when"integrated"systems"go"dark
    • Latency$is$zero"→"there's"no"way"to"change"the"speed"of"light

    View Slide

  39. Fallacies(of(distributed(compu2ng
    • The$network$is$reliable"→"build"systems"that"func3on"even"
    when"integrated"systems"go"dark
    • Latency$is$zero"→"there's"no"way"to"change"the"speed"of"light
    • Bandwidth$is$infinite"→"using"that"fancy"JS"framework"is"
    wonderful"un3l"it"blocks"page"loading"across"100,000"browsers

    View Slide

  40. Fallacies(of(distributed(compu2ng
    • The$network$is$reliable"→"build"systems"that"func3on"even"
    when"integrated"systems"go"dark
    • Latency$is$zero"→"there's"no"way"to"change"the"speed"of"light
    • Bandwidth$is$infinite"→"using"that"fancy"JS"framework"is"
    wonderful"un3l"it"blocks"page"loading"across"100,000"browsers
    • The$network$is$secure"→"integra3on"points"can"introduce"
    vulnerabili3es

    View Slide

  41. Fallacies(of(distributed(compu2ng
    • Topology'doesn’t'change"→"moving"applica.ons"to"the"cloud"
    gives"the"ability"to"change"topology"at"will;"add"new"app"servers,"
    scale"on"demand,"etc

    View Slide

  42. Fallacies(of(distributed(compu2ng
    • Topology'doesn’t'change"→"moving"applica.ons"to"the"cloud"
    gives"the"ability"to"change"topology"at"will;"add"new"app"servers,"
    scale"on"demand,"etc
    • There'is'one'administrator"→"external"systems"have"
    performance,"availability,"or"security"issues"that"you"have"no"
    direct"influence"over

    View Slide

  43. Fallacies(of(distributed(compu2ng
    • Topology'doesn’t'change"→"moving"applica.ons"to"the"cloud"
    gives"the"ability"to"change"topology"at"will;"add"new"app"servers,"
    scale"on"demand,"etc
    • There'is'one'administrator"→"external"systems"have"
    performance,"availability,"or"security"issues"that"you"have"no"
    direct"influence"over
    • Transport'cost'is'zero"→"ask"anyone"doing"video"streaming"
    online"if"transport"costs"are"zero,"transport"is"priced"like"any"
    commodity

    View Slide

  44. Why$func)onal$programming?

    View Slide

  45. Side%effect)free)func.ons)enable)parallelism
    val lastNames = List("Smith","Jones","Frankenstein","Bach","Jackson","Rodin").par
    // lastNames: scala.collection.parallel.immutable.ParSeq[String] =
    // ParVector(Smith, Jones, Frankenstein, Bach, Jackson, Rodin)
    val res0 = lastNames.map(_.toUpperCase) // toUpperCase, higher-order function
    // res0: scala.collection.parallel.immutable.ParSeq[String] =
    // ParVector(SMITH, JONES, FRANKENSTEIN, BACH, JACKSON, RODIN)
    • Crea&ng)a)parallel)collec&on)is)simple
    • Parallel)opera&ons)are)implemented)with)divide)and)conquer)
    style)algorithms)that)parallelize)well

    View Slide

  46. Example(of(a(chained(transforma3on(and(filter
    coll.map(_.toUpperCase).filter(predicate)

    View Slide

  47. Enable'concurrency'in'a'transparent'way
    import scala.util.{Success, Failure}
    // obtain a future list of recent posts
    // executes on a different thread pool
    val f: Future[List[String]] = future {
    session.getRecentPosts
    }
    // success and failure callbacks
    f onComplete {
    case Success(posts) => for (post <- posts) println(post)
    case Failure(t) => println("An error has occured: " + t.getMessage)
    }

    View Slide

  48. Message&passing&to&a&remote&worker
    val remoteGameActor = system.actorOf(
    GameActor.props.withDeploy(Deploy(scope = RemoteScope(address))),
    name = “gameActor”)
    val remoteGameActor = context.actorSelection(
    “akka.tcp://[email protected]:2552/gameActor")
    remoteGameActor ! message

    View Slide

  49. Loca%on'transparency

    View Slide

  50. Easier'to'maintain'and'reason'about
    Scala%enables%DRY%code%with%the%following%features:
    • Higher(order+func0ons
    • Func0onal+composi0on+&+collec0ons+with+map,+filter,+reduce,+etc
    • Case+classes
    • Pa>ern+matching
    • Trait+composi0on+
    • Etc...

    View Slide

  51. As#of#April#7th,#2015,#29#jobs#are#
    posted#on#LinkedIn#for#Apple#
    requiring#Scala#or#other#funcAonal#
    programming#experience.

    View Slide

  52. Web$Services$Engineer,$Apple
    Key$Qualifica,ons
    • Experience*in*developing*web2services*in*a*commercial*se6ng*
    and*familiar*with*HTTP,*REST,*JSON
    • Know*how*to*write*reliable,*resilient,*unit2tested,*performant,*
    baEle2tested*code
    • Deep*experience*in*any*one*of*Java,*Scala,*C*,*C#,*F#
    • Familiar*with*funcKonal*and*reacKve*programming*concepts

    View Slide

  53. Do#I#expect#College#graduates#to#be#qualified#for#fully#qualified#to#
    build#distributed#systems#right#out#of#school?
    • No!%Building%distributed%systems%is%difficult%and%takes%years%of%
    experience%to%design%and%implement

    View Slide

  54. Do#I#expect#College#graduates#to#be#qualified#for#fully#qualified#to#
    build#distributed#systems#right#out#of#school?
    • No!%Building%distributed%systems%is%difficult%and%takes%years%of%
    experience%to%design%and%implement
    • Building%blocks%can%be%taught%at%the%College%level,%posi?oning%
    students%for%future%on@the@job%learning,%e.g,%func?onal,%side@
    effect@free%programming

    View Slide

  55. Do#I#expect#College#graduates#to#be#qualified#for#fully#qualified#to#
    build#distributed#systems#right#out#of#school?
    • No!%Building%distributed%systems%is%difficult%and%takes%years%of%
    experience%to%design%and%implement
    • Building%blocks%can%be%taught%at%the%College%level,%posi?oning%
    students%for%future%on@the@job%learning,%e.g,%func?onal,%side@
    effect@free%programming
    • Teaching%"mechanical%sympathy",%i.e,%to%understand%the%physical%
    limita?ons%of%machines/networks%and%how%soHware%is%deployed%
    and%executed

    View Slide

  56. Interested(in(learning(more?
    • What%Is%Reac+ve%Programming?"#"h%p:/
    /bit.ly/1Ay3lHR
    • Reac+veTO"#"h%p:/
    /www.meetup.com/Reac;ve#TO
    • Func+onal%programming%in%Scala%(Coursera)"#"h%ps:/
    /
    www.coursera.org/course/progfun
    • Typesafe%Ac+vator"#"h%ps:/
    /typesafe.com/community/core#
    tools/ac;vator#and#sbt

    View Slide

  57. Ques%ons?

    View Slide