The New Enterprise: RabbitMq, Akka and AWS at Getty Images

The New Enterprise: RabbitMq, Akka and AWS at Getty Images

Slide deck from a NYC Tech Talk Meetup in NYC on how Getty Images uses RabbitMq, Scala/Akka and AWS automation. The images are all available on Getty Images (http://www.gettyimages.com/Account/MediaBin/LightboxDetail.aspx?Id=13728186&MediaBinUserId=4670194)

3ca5501cb61a4251bd1e6f0a878bb8d4?s=128

Michael Hamrah

April 21, 2014
Tweet

Transcript

  1. THE NEW ENTERPRISE MIGRATING TO AKKA, RABBITMQ AND AWS /

    Michael Hamrah @mhamrah
  2. GETTY IMAGES We sell photos (and videos, music, illustrations) Photo:

    Pascal Le Segretain / Getty Images Entertainment
  3. 200,000 Contibutors Photo: Nisian Hughes/Stone

  4. 185 Countries Photo: Jeff J Mitchell / Getty Images News

  5. 300 Image Partners Photo: Martin Puddy/Stone

  6. Large Archive Photo: Bob Gomel/The LIFE Images Collection

  7. You maybe see a dozen of our images a day?

    Photo: Christopher Polk / Getty Images Entertainment
  8. Dev teams in Seattle, NYC, Calgary, Madison Scala, Ruby, Go,

    C#, Java, PHP
  9. ENTERPRISE WHAT? More complicated then they need to be Usually

    a mess Many different users A ton of use cases A common problem? Photo: Justin Lewis/Stone
  10. WHAT DO WE WANT? Loosely-coupled systems Easy evolution Leverage OSS

    Simplify Databases Easy infrastructure / HA Horizontal Scale Photo: Michael Hamrah/Moment Open
  11. BUT HOW? RabbitMq Scala + Akka CQRS/Event Sourcing

  12. RABBITMQ Messaging that just works Various messaging semantics We do

    Pub/Sub Publish to Exchange Consumers declare queues, bind to routing keys Ack/Nack messages Photo: 4FR / Vetta
  13. None
  14. MESSAGES Routing keys: *scope*.*entity*.*action* MessageId: Unique per message CorrelationId: Connect

    messages Header: Primary Key Header: Sender System Identifier Payload: Send Complete Entity (in Bytes) Photo: Michael Steele / Getty Images Sport
  15. ASSET DOMAIN Photo: Ilya Terentyev / Vetta

  16. HOW TO HANDLE? Get -> Validate -> Process -> Persist

    -> Publish -> Ack Idempotent Messaging == Your BFF Treat as forward-only transactions Photo: Oli Scarff / Getty Images Entertainment
  17. AKKA: LET IT CRASH Actor Model Lightweight Concurrent Resilient Photo:

    Vladimir Rys / Bongarts / Getty Images
  18. AKKA Runtime for objects Natural fit for sub-systems Photo: Harry

    How / Getty Images Sport
  19. ACTOR HIERARCHY Photo: Paul Taylor / Stone

  20. ACTOR DEPENDENCIES ActorRef = Existing Props = Create Great Test

    Framework Photo: 4FR/Vetta
  21. None
  22. PERSISTENCE: EVENT SOURCING Simplify Persistence Save events, not state Save

    events, not commands Slick: collection-like API Schema Changes Actor == Aggregate Cluster for Uniqueness Photo: Donald Miralle / Getty Images Sport
  23. CLEAN ARCHITECTURE / Robert "Uncle Bob" Martin Vimeo

  24. DEPLOY! AUTOMATE! TO THE CLOUD! Photo: Deborah Harrison / Photographer's

    Choice RF
  25. BUT WHAT ABOUT... Server Configs? Network topology? Databases, Queue Servers?

    Load Balancers? Fine-grain control? Photo: Tyler Olson / Hemera / 360
  26. WHAT IF YOU NEVER UPDATED A SERVER? Photo: Ronald Martinez

    / Getty Images Sport Classic
  27. ALWAYS BE LAUNCHING Define your topology Pre-bake a server Launch

    DIY PaaS Photo: Cameron Spencer / Getty Images Sport
  28. PACKER SUPERCHARGED VAGRANT Builds Amis, Docker VMware, VBox, more. Supports

    Chef/Puppet/Scripts Minimize startup times Predictable Photo: Scott Heavey / Getty Images Sport
  29. CLOUDFORMATION INFRASTRUCTURE AS CODE (JSON) Defines network, elb's, asg's, r53's,

    db's, everything Pass data via cloud-init Environment variables are helpful Launch multiple stacks, destroy at will Photo: Steve Allen/Digital Vision
  30. JENKINS DOES ALL Merge = AMI Packer via Puppet Passed

    to CFN template Update/Create Stack Roll in new AMIs Cutover DNS Enjoy Photo: Chris Hyde / Getty Images Sport
  31. THE END Photo: Denis O'Regan / Premium Archive