Slide 1
Slide 1 text
SCALA - organizing control
flow (between imperative
and declarative approaches)
Ruslan Shevchenko
GoSave
!
[email protected]
@rssh1
https://github.com/rssh
Slide 2
Slide 2 text
Styles of control flow organizations:
!
• Future-s …
• async/await
• Actors
• Channels
• Reactive Collections
• DSLs for Computation Plans
Slide 3
Slide 3 text
Control Flow
val x = readX
val y = readY
val z = x*y
scala
X
Y
Z
Imperative = we explicitly set one
Slide 4
Slide 4 text
directed by evaluation strategy
z = x*y
where x = readX
y = readX
haskell
X Y
Z
let x = readX
y = readX
in x*y
Slide 5
Slide 5 text
Control flow: what we need ?
!
• Manage multi{core,machine} control-
flows.
!
• Optimize resource utilization
( see reactive manifesto
http://www.reactivemanifesto.org/ )
Slide 6
Slide 6 text
Reactivity
!
• Ugly situation in industry
!
• In ideal world - {operation system/
language VM} must care about resource
utilization, human - about logic
!
• Our world is far from ideal
Slide 7
Slide 7 text
Control Flow
val x = readX
val y = readY
val z = x*y
Can we readX and Y in parallel
X Y
Z
Slide 8
Slide 8 text
Low level
@volatile var x = _
val xThread = new Thread(
public void run() {
x = readX
}
).start();
!
@volatile var y = _
val yThread = new Thread(
public void run() {
y = readY
}
).start();
yThread.join()
xThread.join()
z = x+y
As GOTO from 60-s
Slide 9
Slide 9 text
Low level with thread pool
val xTask = new Task(
public X run() {
readX
}
);
pool.submit(xTask)
!
val yTask = new Task(
public void run() {
readY
}
)
pool.submit(yTask)
!
z = xTask.get()+yTask.get()
X Y
Z
Slide 10
Slide 10 text
Scala Future
X Y
Z
val x = Future{ readX }
val y = Future{ readY }
val z = Await.result(x, 1 minute) + Await.result(y, 1 minute)
Slide 11
Slide 11 text
Future
!
• Future[T] = pointer to the value in the
future
!
• isComplete: Boolean
!
• Await.result(feature, duration): T
!
• onComplete(Try[T] => U) : Unit
Slide 12
Slide 12 text
Future { readX }
object Future
{
!
def apply(body: =>T) : Future[T] =
…………
!
}
Call by name syntax = [call by function .. ]
by name - term from Algol 68
Own functions like control flow constructions
Slide 13
Slide 13 text
Future-s are composable.
!
• map: X=>Y Future[X] => Future[Y]
!
• map[Y](x:Future[X])(f:X=>Y): Future[Y]
!
• flatMap:
• X => Future[Y] Future[X]=>Future[Y]
!
• flatMap[Y](x:Future[X])(f:X=>Future[Y]): Future[Y]
Slide 14
Slide 14 text
Scala Future
val x = Future{ readX }
val y = Future{ readY }
val z = Future{ … after X and Y }
val z = Future{ readX } flatMap { x => readY map ( y=> x+y ) }
!
for{ x <- Future{ readY },
y <- Future{ readY }) yield x+y
!
Using monadic syntax:
Slide 15
Slide 15 text
Future
!
• Good for simple cases
!
• Hard when we want to implement
complex logic
!
Slide 16
Slide 16 text
SIP22 (Async/Await)
!
• when we want to implement complex
logic
!
• Ozz style -> (limited) F# ->(limited) C#
!
• in scala as library: https://github.com/
scala/async
!
Slide 17
Slide 17 text
SIP22 (Async/Await)
!
• async(body: T):Future[T]
!
• await(future: Future[T]): T
• can be used only inside async
!
• async macro rewrite body as state
machine.
!
Slide 18
Slide 18 text
Async/Await
val x = Future{ readX }
val y = Future{ readY }
val z = Future{ … after X and Y }
val z = async{
val x = Future{ readX }
val y = Future{ readY }
await(x) + await(y)
}
Slide 19
Slide 19 text
Async/Await
val z = async{
val x = Future{ readX }
val y = Future{ readY }
await(x) + await(y)
}
val z = {
val s1 = Future{ readX(); }
val s2 = Future{ readY(); }
val s3 = if (s1.isComplete)
if (s2.isComplete) {
Promise.success(s1.result + s2.result)
} else { s2 map s3 }
} else s1 map s3
s3
// Just show the idea, not actual
Slide 20
Slide 20 text
Akka http://www.akka.io
!
• Erlang-style concurrency
!
• Actor - active object
• send/receive message
• have local state
• available by name in akka cluster.
!
!
!
!
Slide 21
Slide 21 text
Actor
!
class EventsProcessor extends Actor
{
def receive = {
case Event(msg) => println(msg)
case Echo(msg) => sender ! Echo(msg)
case Ping => sender ! Pong
case Stop => context.stop(self)
}
!
}
Slide 22
Slide 22 text
Akka
!
• tell(x:Any)=>Unit — send and forget
actor ! x
!
!
• ack(x:Any)=>Future[Any] — send and
receive future to answer
actor ? x
!
!
Slide 23
Slide 23 text
Actor
Mailbox
Processor (with state) 1
Slide 24
Slide 24 text
Akka
!
• Scheduler
!
• Event Bus
!
• Common patterns
• Load balancing
• Throttling messages
• …..
!
!
Slide 25
Slide 25 text
Akka
!
• Scale on cluster
!
• Recovery
!
• Persistent Queue
!
!
Slide 26
Slide 26 text
Go-like channels
!
• Come from Go language
• http://golang.org/
!
• Implementation as library and async.
• (yet not ready) -
https://github.com/rssh/scala-gopher
branch “async/unsugared.”
Slide 27
Slide 27 text
Rx channels
!
• Reactive extensions
• http://rxscala.github.io/
!
• Rx collection call you.
!
Slide 28
Slide 28 text
Rx scala
!
trait Observer[E]
{
def onNext(e: E)
def onError(e: Throwable)
def onCompleted()
!
}
!
!
!
trait Observable[E]
{
!
def subscribe(o:Observer)
!
……….
!
map, flatMap, zip, filter …
}
!
Slide 29
Slide 29 text
Computation plan DSL
!
• Collections operations can be distributed
!
• Simple form: .par
for( x <- collection.par) yield x+1
!
!
• Same idea for hadoop map/reduce
!
Slide 30
Slide 30 text
Computation plan DSL for Hadoop
!
• Scalding
• https://github.com/twitter/scalding
!
• Scoobi
• http://nicta.github.io/scoobi/
!
• Scrunch
• http://crunch.apache.org/scrunch.html
Slide 31
Slide 31 text
Scoobi
!
val lines = fromTextFile("hdfs://in/...")
!
val counts = lines.mapFlatten(_.split(" "))
.map(word => (word, 1))
.groupByKey
.combine(Sum.int)
!
counts.toTextFile(\“hdfs://out/…",
overwrite=true).persist(ScoobiConfiguration())
map, groupByKey, combine => Map/Reduce tasks
Slide 32
Slide 32 text
Mahout: computation plan DSL for Spark
!
• https://mahout.apache.org/
!
• Scalable machine learning library.
• R-like matrix operations
• Optimizer for algebraic expression
Slide 33
Slide 33 text
Machout
!
// R-like operations (linear algebra)
!
val g = bt.t %*% bt - c - c.t + (s_q cross s_q) * (xi dot xi)
!
drmA.mapBlock(ncol = r) {
case (keys, blockA) =>
val blockY = blockA %*%
Matrices.symmetricUniformView(n, r, omegaSeed)
keys -> blockY
}
Match operations => computations plans on backend
Slide 34
Slide 34 text
Scala: organization of control flow
!
• Many styles. No one is better.
!
• Low-level: futures & callbacks
!
• Middle-level: actors, channels, streams
!
• Hight-level: declarative DSL
Slide 35
Slide 35 text
Scala: organization of control flow
!
• Thanks for attention.
!
• Questions (?)
!
• //Ruslan Shevchenko
!