Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling infrastructure beyond containers

Scaling infrastructure beyond containers

This is the talk about using Apache Mesos to effectively manage resources in modern infrastructure.

7f1dfa02fd3771699d5bac40fc54a21c?s=128

Mateusz Gajewski

September 24, 2015
Tweet

Transcript

  1. @wendigo Mateusz „Serafin” Gajewski • AWS UG Meetup Scaling infrastructure

    beyond containers
  2. @wendigo Agenda • Evolution of infrastructure at Allegro, • Why

    Apache Mesos™? • Apache Mesos key concepts, • Future of datacenter and cloud computing?
  3. @wendigo History of scaling infrastructure @ Allegro

  4. @wendigo Infrastructure 1.0

  5. @wendigo Job allocation problem

  6. @wendigo Web Scale Resource management 100s dots 100s dots

  7. @wendigo Infrastructure 2.0 1000s dots another 1000s dots

  8. @wendigo Infrastructure 2.1 1000s dots another 1000s dots

  9. @wendigo Challenges • cloud not used as cloud ;), •

    high cost of virtualization, • effective resource utilization, • microservice architecture, • spread of new technologies, • heterogenous resources, • scalability, fault tolerance & HA, • performance isolation, • data processing at scale
  10. @wendigo Beyond cloud computing

  11. @wendigo Holly Grail of TCO

  12. @wendigo Infrastructure 3.0 A Platform for Fine-Grained Resource Sharing in

    the Data Center
  13. @wendigo Scheduling

  14. @wendigo Cluster scheduling

  15. @wendigo Mesos architecture

  16. @wendigo Mesos frameworks

  17. @wendigo Offers

  18. @wendigo Execution isolation

  19. @wendigo External Containerizers

  20. @wendigo Mesos HA • master election/failover with ZooKeeper, • master

    maintains soft-state, • framework state reconciliation, • slave checkpointing, • slave recovery, • framework checkpointing
  21. @wendigo Beyond offers • offer filters (constraints), • static (pre-startup)

    reservations, • dynamic (post-startup) reservations, • oversubscription, • persistent volumes, • pluggable allocator scheduling policy (fair, priority based)
  22. @wendigo Mesos frameworks

  23. None
  24. @wendigo Mesos recap • „programming against the datacenter", • distributed

    datacenter kernel, • two-level multi-resource scheduler, • scalable, highly-available & fault-tolerant, • performance isolation with containers, • exposes homogeneous resources, • elastic, dynamic partitioning, • high resource utilization
  25. @wendigo Future Datacenter

  26. @wendigo IaaC

  27. @wendigo Efficient utilization

  28. @wendigo Google’s Omega source: Omega: flexible, scalable schedulers for large

    compute clusters
  29. None
  30. Questions?

  31. http://meetup.com/allegro.tech http://allegro.tech @AllegroTechBlog Work with us