Akka: A shiny new hammer Now show me some nails already! Dag Liodden VP Engineering Tapad North East Scala Symposium 2012

So what is Akka?! 2! “Akka is a toolkit and runtime for building highly concurrent, distributed, and fault tolerant event- driven applications on the JVM.” -  http://akka.io

3! ZOMG!

4! Node.js?

5! Automatic web scale?

So what is Akka?! 6! Akka is a library library and / or runtime for ❱  Parallelism. Increasing performance by splitting tasks and executing them on multiple cores simultaneously. ❱  Concurrency. Coordinating resources between multiple threads of control.

What does this mean?! 7! ❱  Akka is NOT a general purpose framework ❱  Focused feature set, excellent implementation ❱  Surprisingly broad applicability ❱  Not a golden hammer

Real-time Ad technology 101! Strictly Confidential"

Real-time audience buying! Strictly Confidential" 9! ❱  Before ­  “Sites about travel at a max price of $1 CPM”" ❱  Now ­  Real-time buying" ­  User-level targeting constraints" ­  Value prediction" "

Lots of traffic The RTB Eco-system! 10! Browser Website Ad exchange Buyer A Buyer B Buyer C

How we use Akka! Strictly Confidential" 11! ❱  Concurrent access to mutable state ❱  Circuit-breaking for graceful degradation of service ❱  Batching and buffering ❱  Resource balancing and throttling ❱  Scheduling of background tasks

How we use Akka! Strictly Confidential" 12! ❱  Concurrent access to mutable state ❱  Circuit-breaking for graceful degradation of service ❱  Batching and buffering ❱  Resource balancing and throttling ❱  Scheduling of background tasks

Actors for concurrent mutable state! Strictly Confidential"

The bidding decision! Strictly Confidential" 14! trait Bidder {! def bid(req: Offer) : Option[Bid]! def onWin(win: WinNotification) : Unit! }!

Bidder topology! 15! AuctionRunner AuctionRunner Auction runner threads Campaign A Campaign B Remote buyer A Workers Remote buyer B Workers

Why model bidders as actors?! Strictly Confidential" 16! ❱  Local actors have state ❱  Remote actors have unpredictable response times "

Local bidders: Should we bid at all?! Strictly Confidential" 17! ❱  Targeting constraints ­  Run on iPads visiting sites about Travel" ­  Stateless" ❱  Budget constraints, even pacing of spend ­  How much spent so far?" ­  How much estimated spend so far?" ­  Stateful, extreme concurrency"

A local bidder actor! Strictly Confidential" 18! trait Bidder {! def bid(req: Offer) : Option[Bid]! def onWin(win: WinNotification) : Unit! }! ! class LocalBidderActor(bidder: Bidder) extends Actor {! private[this] _bidder = bidder! def receive = { ! case req: Offer=> self.reply(target.bid(req))! case win: WinNotification => target.onWin(win)! case bidder: Bidder => _bidder = bidder // Hot swap! ...! }! }! ! // Is this just bidder.synchronize {} ?! ! !

The auction runner! 19! def runAuction(req: Offer, rules: AuctionRules) = {! val responseFutures = bidderActorList.map {! !_.!!![Option[Bid]](req, MAX_RESPONSE_TIME)! }! try {! Futures.awaitAll(responseFutures) // Block! } catch {! case e: FutureTimeoutException => // One or more timeouts! }! val timelyResponses = responseFutures.flatMap(_.result)! rules.determineResult(req, timelyResponses)! }!

Benefits over synchronized! Strictly Confidential" 20! ❱  A single access model for local and remote bidders ❱  Dead simple state management (no leaking) ❱  CPU Pinning ❱  Parallelism (when there are CPU cycles available)

Graceful degradation and circuit-breaking! Strictly Confidential"

Circuit-breaking! Strictly Confidential" 22! ❱  Relentless traffic with no back-off ❱  VM Warm-up ❱  K/V hick-ups ❱  Traffic spikes ❱  Degrade the service to catch up! "

How to degrade service? ! Strictly Confidential" 23! ❱  Accept increased latency? ❱  Refuse connections?

How to degrade service? (cont.)! 24! ❱  Domain / service specific options ­  Stop loading user data => Bid with less information" ­  Stop running predictive optimization => Bid conservative" ­  Start passing on offer altogether => Opportunity cost"

When to degrade service?! Strictly Confidential" 25! ❱  Response time > 10ms? ­  Actors with timestamped units of work" ❱  Mailbox size > 5000 elements? ­  Custom dispatcher"

Circuit-breaking actor! Strictly Confidential" 26! " ! protected def receive = {! case WorkUnit(submittedAt, offer) =>! if (System.currentTimeMillis > (submittedAt + MAX_TIME)) {! Stats.incr(PerformanceMetrics.AUCTION_RUNNER("dropped"))! self reply None! } else {! self reply executeAuction(offer)! }! }!

Issues with the CB in the actor?! Strictly Confidential" 27! ❱  Long pauses => Long queues => OOM ❱  Bounded mailbox => Passing on the most recent offers! ­  + a lot of exceptions"

Circuit-breaking dispatcher! Strictly Confidential" 28! " " val workerDispatcher = new ExecutorBasedEventDrivenDispatcher(" …" ) with CircuitBreakingDispatcherSemantics {" val policy = new LogAndDiscardCircuitBreakerPolicy(2000, AuctionRunner.this) {" override def replyToOverflow(overflowCount: Int, msg: Any) = {" Stats.incr(PerformanceMetrics.AUCTION_RUNNER("discarded"))" Right(None)" }" }" }"

Issues with CB in dispatchers?! Strictly Confidential" 29! ❱  Might mask performance issues ❱  What’s the correct max length? ❱  Hard to do in Akka 2.0

Batching and buffering! Strictly Confidential"

Batching storage actor! Strictly Confidential" 31! class OfferStorageActor(dao: OfferDao)! extends BatchingStorageActor[(Offer,AuctionResult)] with ActorLogging {! /**! * Flush every 2 seconds of 25000 requests. The former will almost always be the trigger.! */! final def batchSize = 25000! final def purgeInterval = 2000L! ! def save(xs: Seq[(BidRequest,AuctionResult)]) {! logger.debug("Storing {} bid requests.", xs.size)! bidRequestDao.save(xs)! logger.debug("Done storing bid requests.")! }! }!

Akka pain points?! Strictly Confidential" 32! ❱  Untyped “interfaces” ❱  Asynchronous indirection => Hard to find original invocation point ❱  Testing

Questions?! Strictly Confidential"

Resource throttling and balancing! Strictly Confidential"

Resource throttling and balancing! Strictly Confidential" 35! ❱  Using actors as an executor service ❱  Turn blocking APIs into non-blocking ❱  Tune connection pools for throughput / latency

Scheduling of background tasks! Strictly Confidential"

Background processing use cases! Strictly Confidential" 37! ❱  Asynchronous writes ❱  Batching / buffering writes ❱  Lazy caching ❱  “Cron jobs”