Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Akka: A Shiny New Hammer. Now Show Me Some Nails Already!

marakana
April 04, 2012
2.3k

Akka: A Shiny New Hammer. Now Show Me Some Nails Already!

In this talk from the 2012 Northeast Scala Symposium, Dag Liodden, VP Engineering of Tapad, takes a pragmatic look at how Akka and Akka actors can be used in practice, looking at architecture and code examples from a real-time ad trading system.

Ever since it's first release, Akka has been receiving a lot of interest and praise from the Scala community. With a tagline promising to deliver "Event-driven, scalable and fault-tolerant architectures", who wouldn't want to use it? The framework does indeed deliver, but it's not a general purpose technology that can or should be applied anywhere. Dag starts off introducing us to Akka, then moves into real use cases, while addressing many of the strengths and potential pain points of the framework.

marakana

April 04, 2012
Tweet

Transcript

  1. Akka: A shiny new hammer
    Now show me some nails already!
    Dag Liodden
    VP Engineering Tapad
    North East Scala Symposium 2012

    View full-size slide

  2. So what is Akka?!
    2!

    “Akka is a toolkit and runtime for building highly
    concurrent, distributed, and fault tolerant event-
    driven applications on the JVM.”

    -  http://akka.io

    View full-size slide

  3. 5!




    Automatic web scale?

    View full-size slide

  4. So what is Akka?!
    6!
    Akka is a library library and / or runtime for
    ❱  Parallelism. Increasing performance by splitting tasks and
    executing them on multiple cores simultaneously.
    ❱  Concurrency. Coordinating resources between multiple
    threads of control.

    View full-size slide

  5. What does this mean?!
    7!

    ❱  Akka is NOT a general purpose framework
    ❱  Focused feature set, excellent implementation
    ❱  Surprisingly broad applicability
    ❱  Not a golden hammer



    View full-size slide

  6. Real-time Ad technology 101!
    Strictly Confidential"

    View full-size slide

  7. Real-time audience buying!
    Strictly Confidential" 9!
    ❱  Before
    ­  “Sites about travel at a max price of $1 CPM”"
    ❱  Now
    ­  Real-time buying"
    ­  User-level targeting constraints"
    ­  Value prediction"
    "

    View full-size slide

  8. Lots of traffic
    The RTB Eco-system!
    10!
    Browser Website
    Ad
    exchange
    Buyer A
    Buyer B
    Buyer C

    View full-size slide

  9. How we use Akka!
    Strictly Confidential" 11!
    ❱  Concurrent access to mutable state

    ❱  Circuit-breaking for graceful degradation of service
    ❱  Batching and buffering
    ❱  Resource balancing and throttling
    ❱  Scheduling of background tasks

    View full-size slide

  10. How we use Akka!
    Strictly Confidential" 12!
    ❱  Concurrent access to mutable state

    ❱  Circuit-breaking for graceful degradation of service
    ❱  Batching and buffering
    ❱  Resource balancing and throttling
    ❱  Scheduling of background tasks

    View full-size slide

  11. Actors for concurrent mutable state!
    Strictly Confidential"

    View full-size slide

  12. The bidding decision!
    Strictly Confidential" 14!


    trait Bidder {!
    def bid(req: Offer) : Option[Bid]!
    def onWin(win: WinNotification) : Unit!
    }!

    View full-size slide

  13. Bidder topology!
    15!
    AuctionRunner
    AuctionRunner
    Auction runner
    threads
    Campaign A
    Campaign B
    Remote buyer
    A
    Workers
    Remote buyer
    B
    Workers

    View full-size slide

  14. Why model bidders as actors?!
    Strictly Confidential" 16!
    ❱  Local actors have state
    ❱  Remote actors have unpredictable response times
    "


    View full-size slide

  15. Local bidders: Should we bid at all?!
    Strictly Confidential" 17!
    ❱  Targeting constraints
    ­  Run on iPads visiting sites about Travel"
    ­  Stateless"

    ❱  Budget constraints, even pacing of spend
    ­  How much spent so far?"
    ­  How much estimated spend so far?"
    ­  Stateful, extreme concurrency"

    View full-size slide

  16. A local bidder actor!
    Strictly Confidential" 18!
    trait Bidder {!
    def bid(req: Offer) : Option[Bid]!
    def onWin(win: WinNotification) : Unit!
    }!
    !
    class LocalBidderActor(bidder: Bidder) extends Actor {!
    private[this] _bidder = bidder!
    def receive = { !
    case req: Offer=> self.reply(target.bid(req))!
    case win: WinNotification => target.onWin(win)!
    case bidder: Bidder => _bidder = bidder // Hot swap!
    ...!
    }!
    }!
    !
    // Is this just bidder.synchronize {} ?!
    !
    !

    View full-size slide

  17. The auction runner!
    19!
    def runAuction(req: Offer, rules: AuctionRules) = {!
    val responseFutures = bidderActorList.map {!
    !_.!!![Option[Bid]](req, MAX_RESPONSE_TIME)!
    }!
    try {!
    Futures.awaitAll(responseFutures) // Block!
    } catch {!
    case e: FutureTimeoutException => // One or more timeouts!
    }!
    val timelyResponses = responseFutures.flatMap(_.result)!
    rules.determineResult(req, timelyResponses)!
    }!

    View full-size slide

  18. Benefits over synchronized!
    Strictly Confidential" 20!
    ❱  A single access model for local and remote bidders
    ❱  Dead simple state management (no leaking)
    ❱  CPU Pinning
    ❱  Parallelism (when there are CPU cycles available)

    View full-size slide

  19. Graceful degradation and circuit-breaking!
    Strictly Confidential"

    View full-size slide

  20. Circuit-breaking!
    Strictly Confidential" 22!
    ❱  Relentless traffic with no back-off
    ❱  VM Warm-up
    ❱  K/V hick-ups
    ❱  Traffic spikes
    ❱  Degrade the service to catch up!
    "

    View full-size slide

  21. How to degrade service? !
    Strictly Confidential" 23!
    ❱  Accept increased latency?
    ❱  Refuse connections?

    View full-size slide

  22. How to degrade service? (cont.)!
    24!
    ❱  Domain / service specific options
    ­  Stop loading user data => Bid with less information"
    ­  Stop running predictive optimization => Bid conservative"
    ­  Start passing on offer altogether => Opportunity cost"

    View full-size slide

  23. When to degrade service?!
    Strictly Confidential" 25!
    ❱  Response time > 10ms?
    ­  Actors with timestamped units of work"
    ❱  Mailbox size > 5000 elements?
    ­  Custom dispatcher"

    View full-size slide

  24. Circuit-breaking actor!
    Strictly Confidential" 26!
    "
    !
    protected def receive = {!
    case WorkUnit(submittedAt, offer) =>!
    if (System.currentTimeMillis > (submittedAt + MAX_TIME)) {!
    Stats.incr(PerformanceMetrics.AUCTION_RUNNER("dropped"))!
    self reply None!
    } else {!
    self reply executeAuction(offer)!
    }!
    }!

    View full-size slide

  25. Issues with the CB in the actor?!
    Strictly Confidential" 27!
    ❱  Long pauses => Long queues => OOM
    ❱  Bounded mailbox => Passing on the most recent offers!
    ­  + a lot of exceptions"

    View full-size slide

  26. Circuit-breaking dispatcher!
    Strictly Confidential" 28!
    "
    "
    val workerDispatcher = new ExecutorBasedEventDrivenDispatcher("
    …"
    ) with CircuitBreakingDispatcherSemantics {"
    val policy = new LogAndDiscardCircuitBreakerPolicy(2000, AuctionRunner.this) {"
    override def replyToOverflow(overflowCount: Int, msg: Any) = {"
    Stats.incr(PerformanceMetrics.AUCTION_RUNNER("discarded"))"
    Right(None)"
    }"
    }"
    }"

    View full-size slide

  27. Issues with CB in dispatchers?!
    Strictly Confidential" 29!

    ❱  Might mask performance issues

    ❱  What’s the correct max length?
    ❱  Hard to do in Akka 2.0

    View full-size slide

  28. Batching and buffering!
    Strictly Confidential"

    View full-size slide

  29. Batching storage actor!
    Strictly Confidential" 31!
    class OfferStorageActor(dao: OfferDao)!
    extends BatchingStorageActor[(Offer,AuctionResult)] with ActorLogging {!
    /**!
    * Flush every 2 seconds of 25000 requests. The former will almost always be the trigger.!
    */!
    final def batchSize = 25000!
    final def purgeInterval = 2000L!
    !
    def save(xs: Seq[(BidRequest,AuctionResult)]) {!
    logger.debug("Storing {} bid requests.", xs.size)!
    bidRequestDao.save(xs)!
    logger.debug("Done storing bid requests.")!
    }!
    }!

    View full-size slide

  30. Akka pain points?!
    Strictly Confidential" 32!
    ❱  Untyped “interfaces”
    ❱  Asynchronous indirection => Hard to find original invocation
    point
    ❱  Testing

    View full-size slide

  31. Questions?!
    Strictly Confidential"

    View full-size slide

  32. Resource throttling and balancing!
    Strictly Confidential"

    View full-size slide

  33. Resource throttling and balancing!
    Strictly Confidential" 35!
    ❱  Using actors as an executor service
    ❱  Turn blocking APIs into non-blocking
    ❱  Tune connection pools for throughput / latency

    View full-size slide

  34. Scheduling of background tasks!
    Strictly Confidential"

    View full-size slide

  35. Background processing use cases!
    Strictly Confidential" 37!
    ❱  Asynchronous writes
    ❱  Batching / buffering writes
    ❱  Lazy caching
    ❱  “Cron jobs”

    View full-size slide