Scaling infrastructure beyond containers

@wendigo Mateusz „Serafin” Gajewski • AWS UG Meetup Scaling infrastructure
beyond containers

@wendigo Agenda • Evolution of infrastructure at Allegro, • Why
Apache Mesos™? • Apache Mesos key concepts, • Future of datacenter and cloud computing?

@wendigo History of scaling infrastructure @ Allegro

@wendigo Infrastructure 1.0

@wendigo Job allocation problem

@wendigo Web Scale Resource management 100s dots 100s dots

@wendigo Infrastructure 2.0 1000s dots another 1000s dots

@wendigo Infrastructure 2.1 1000s dots another 1000s dots

@wendigo Challenges • cloud not used as cloud ;), •
high cost of virtualization, • effective resource utilization, • microservice architecture, • spread of new technologies, • heterogenous resources, • scalability, fault tolerance & HA, • performance isolation, • data processing at scale

@wendigo Beyond cloud computing

@wendigo Holly Grail of TCO

@wendigo Infrastructure 3.0 A Platform for Fine-Grained Resource Sharing in
the Data Center

@wendigo Scheduling

@wendigo Cluster scheduling

@wendigo Mesos architecture

@wendigo Mesos frameworks

@wendigo Offers

@wendigo Execution isolation

@wendigo External Containerizers

@wendigo Mesos HA • master election/failover with ZooKeeper, • master
maintains soft-state, • framework state reconciliation, • slave checkpointing, • slave recovery, • framework checkpointing

@wendigo Beyond offers • offer filters (constraints), • static (pre-startup)
reservations, • dynamic (post-startup) reservations, • oversubscription, • persistent volumes, • pluggable allocator scheduling policy (fair, priority based)

@wendigo Mesos frameworks

@wendigo Mesos recap • „programming against the datacenter", • distributed
datacenter kernel, • two-level multi-resource scheduler, • scalable, highly-available & fault-tolerant, • performance isolation with containers, • exposes homogeneous resources, • elastic, dynamic partitioning, • high resource utilization

@wendigo Future Datacenter

@wendigo IaaC

@wendigo Efficient utilization

@wendigo Google’s Omega source: Omega: flexible, scalable schedulers for large
compute clusters

Questions?

http://meetup.com/allegro.tech http://allegro.tech @AllegroTechBlog Work with us

Scaling infrastructure beyond containers

Scaling infrastructure beyond containers

Mateusz Gajewski

More Decks by Mateusz Gajewski

Other Decks in Technology

Featured

Transcript

@wendigo Mateusz „Serafin” Gajewski • AWS UG Meetup Scaling infrastructure

@wendigo Agenda • Evolution of infrastructure at Allegro, • Why

@wendigo History of scaling infrastructure @ Allegro

@wendigo Infrastructure 1.0

@wendigo Job allocation problem

@wendigo Web Scale Resource management 100s dots 100s dots

@wendigo Infrastructure 2.0 1000s dots another 1000s dots

@wendigo Infrastructure 2.1 1000s dots another 1000s dots

@wendigo Challenges • cloud not used as cloud ;), •

@wendigo Beyond cloud computing

@wendigo Holly Grail of TCO

@wendigo Infrastructure 3.0 A Platform for Fine-Grained Resource Sharing in

@wendigo Scheduling

@wendigo Cluster scheduling

@wendigo Mesos architecture

@wendigo Mesos frameworks

@wendigo Offers

@wendigo Execution isolation

@wendigo External Containerizers

@wendigo Mesos HA • master election/failover with ZooKeeper, • master

@wendigo Beyond offers • offer filters (constraints), • static (pre-startup)

@wendigo Mesos frameworks

@wendigo Mesos recap • „programming against the datacenter", • distributed

@wendigo Future Datacenter

@wendigo IaaC

@wendigo Efficient utilization

@wendigo Google’s Omega source: Omega: flexible, scalable schedulers for large

Questions?

http://meetup.com/allegro.tech http://allegro.tech @AllegroTechBlog Work with us