Akka: A Shiny New Hammer. Now Show Me Some Nails Already!

In this talk from the 2012 Northeast Scala Symposium, Dag Liodden, VP Engineering of Tapad, takes a pragmatic look at how Akka and Akka actors can be used in practice, looking at architecture and code examples from a real-time ad trading system.

Ever since it's first release, Akka has been receiving a lot of interest and praise from the Scala community. With a tagline promising to deliver "Event-driven, scalable and fault-tolerant architectures", who wouldn't want to use it? The framework does indeed deliver, but it's not a general purpose technology that can or should be applied anywhere. Dag starts off introducing us to Akka, then moves into real use cases, while addressing many of the strengths and potential pain points of the framework.

Transcript

1. Akka: A shiny new hammer Now show me some nails already! Dag Liodden VP Engineering Tapad North East Scala Symposium 2012
2. So what is Akka?! 2! “Akka is a toolkit and runtime for building highly concurrent, distributed, and fault tolerant event- driven applications on the JVM.” -  http://akka.io
3. 3! ZOMG!
4. 4! Node.js?
5. 5! Automatic web scale?
6. So what is Akka?! 6! Akka is a library library and / or runtime for ❱  Parallelism. Increasing performance by splitting tasks and executing them on multiple cores simultaneously. ❱  Concurrency. Coordinating resources between multiple threads of control.
7. What does this mean?! 7! ❱  Akka is NOT a general purpose framework ❱  Focused feature set, excellent implementation ❱  Surprisingly broad applicability ❱  Not a golden hammer
8. Real-time Ad technology 101! Strictly Confidential"
9. Real-time audience buying! Strictly Confidential" 9! ❱  Before ­  “Sites about travel at a max price of $1 CPM”" ❱  Now ­  Real-time buying" ­  User-level targeting constraints" ­  Value prediction" "
10. Lots of traffic The RTB Eco-system! 10! Browser Website Ad exchange Buyer A Buyer B Buyer C
11. How we use Akka! Strictly Confidential" 11! ❱  Concurrent access to mutable state ❱  Circuit-breaking for graceful degradation of service ❱  Batching and buffering ❱  Resource balancing and throttling ❱  Scheduling of background tasks
12. How we use Akka! Strictly Confidential" 12! ❱  Concurrent access to mutable state ❱  Circuit-breaking for graceful degradation of service ❱  Batching and buffering ❱  Resource balancing and throttling ❱  Scheduling of background tasks
13. Actors for concurrent mutable state! Strictly Confidential"
14. The bidding decision! Strictly Confidential" 14! trait Bidder {! def bid(req: Offer) : Option[Bid]! def onWin(win: WinNotification) : Unit! }!
15. Bidder topology! 15! AuctionRunner AuctionRunner Auction runner threads Campaign A Campaign B Remote buyer A Workers Remote buyer B Workers
16. Why model bidders as actors?! Strictly Confidential" 16! ❱  Local actors have state ❱  Remote actors have unpredictable response times "
17. Local bidders: Should we bid at all?! Strictly Confidential" 17! ❱  Targeting constraints ­  Run on iPads visiting sites about Travel" ­  Stateless" ❱  Budget constraints, even pacing of spend ­  How much spent so far?" ­  How much estimated spend so far?" ­  Stateful, extreme concurrency"
18. A local bidder actor! Strictly Confidential" 18! trait Bidder {! def bid(req: Offer) : Option[Bid]! def onWin(win: WinNotification) : Unit! }! ! class LocalBidderActor(bidder: Bidder) extends Actor {! private[this] _bidder = bidder! def receive = { ! case req: Offer=> self.reply(target.bid(req))! case win: WinNotification => target.onWin(win)! case bidder: Bidder => _bidder = bidder // Hot swap! ...! }! }! ! // Is this just bidder.synchronize {} ?! ! !
19. The auction runner! 19! def runAuction(req: Offer, rules: AuctionRules) = {! val responseFutures = bidderActorList.map {! !_.!!![Option[Bid]](req, MAX_RESPONSE_TIME)! }! try {! Futures.awaitAll(responseFutures) // Block! } catch {! case e: FutureTimeoutException => // One or more timeouts! }! val timelyResponses = responseFutures.flatMap(_.result)! rules.determineResult(req, timelyResponses)! }!
20. Benefits over synchronized! Strictly Confidential" 20! ❱  A single access model for local and remote bidders ❱  Dead simple state management (no leaking) ❱  CPU Pinning ❱  Parallelism (when there are CPU cycles available)
21. Graceful degradation and circuit-breaking! Strictly Confidential"
22. Circuit-breaking! Strictly Confidential" 22! ❱  Relentless traffic with no back-off ❱  VM Warm-up ❱  K/V hick-ups ❱  Traffic spikes ❱  Degrade the service to catch up! "
23. How to degrade service? ! Strictly Confidential" 23! ❱  Accept increased latency? ❱  Refuse connections?
24. How to degrade service? (cont.)! 24! ❱  Domain / service specific options ­  Stop loading user data => Bid with less information" ­  Stop running predictive optimization => Bid conservative" ­  Start passing on offer altogether => Opportunity cost"
25. When to degrade service?! Strictly Confidential" 25! ❱  Response time > 10ms? ­  Actors with timestamped units of work" ❱  Mailbox size > 5000 elements? ­  Custom dispatcher"
26. Circuit-breaking actor! Strictly Confidential" 26! " ! protected def receive = {! case WorkUnit(submittedAt, offer) =>! if (System.currentTimeMillis > (submittedAt + MAX_TIME)) {! Stats.incr(PerformanceMetrics.AUCTION_RUNNER("dropped"))! self reply None! } else {! self reply executeAuction(offer)! }! }!
27. Issues with the CB in the actor?! Strictly Confidential" 27! ❱  Long pauses => Long queues => OOM ❱  Bounded mailbox => Passing on the most recent offers! ­  + a lot of exceptions"
28. Circuit-breaking dispatcher! Strictly Confidential" 28! " " val workerDispatcher = new ExecutorBasedEventDrivenDispatcher(" …" ) with CircuitBreakingDispatcherSemantics {" val policy = new LogAndDiscardCircuitBreakerPolicy(2000, AuctionRunner.this) {" override def replyToOverflow(overflowCount: Int, msg: Any) = {" Stats.incr(PerformanceMetrics.AUCTION_RUNNER("discarded"))" Right(None)" }" }" }"
29. Issues with CB in dispatchers?! Strictly Confidential" 29! ❱  Might mask performance issues ❱  What’s the correct max length? ❱  Hard to do in Akka 2.0
30. Batching and buffering! Strictly Confidential"
31. Batching storage actor! Strictly Confidential" 31! class OfferStorageActor(dao: OfferDao)! extends BatchingStorageActor[(Offer,AuctionResult)] with ActorLogging {! /**! * Flush every 2 seconds of 25000 requests. The former will almost always be the trigger.! */! final def batchSize = 25000! final def purgeInterval = 2000L! ! def save(xs: Seq[(BidRequest,AuctionResult)]) {! logger.debug("Storing {} bid requests.", xs.size)! bidRequestDao.save(xs)! logger.debug("Done storing bid requests.")! }! }!
32. Akka pain points?! Strictly Confidential" 32! ❱  Untyped “interfaces” ❱  Asynchronous indirection => Hard to find original invocation point ❱  Testing
33. Questions?! Strictly Confidential"
34. Resource throttling and balancing! Strictly Confidential"
35. Resource throttling and balancing! Strictly Confidential" 35! ❱  Using actors as an executor service ❱  Turn blocking APIs into non-blocking ❱  Tune connection pools for throughput / latency
36. Scheduling of background tasks! Strictly Confidential"
37. Background processing use cases! Strictly Confidential" 37! ❱  Asynchronous writes ❱  Batching / buffering writes ❱  Lazy caching ❱  “Cron jobs”