System Integration: OpenStack's success story

OpenStack is a huge, open-source cloud provider. One of the main tenets of OpenStack is the (Shared Nothing Architecture) to which all modules stick very closely. In order to do that, services within OpenStack have adopted different strategies to integrate themselves and share data without sacrificing performance nor moving away from SNA.

This strategies are not applicable just to OpenStack but to any distributed system. Sharing data, regardless what that data is, is a must-have requirement of any successful cloud service.

This talk will present some of the existing integration strategies that are applicable to cloud infrastructures and enterprise services. The talk will be based on the strategies that have helped OpenStack to be successful and most importantly, scalable.

Transcript

1. @flaper87 System Integration OpenStack's success story
2. @flaper87 @flaper87
3. @flaper87 member of
4. @flaper87 • GsoC / OPW mentor • Rust language contributor • Member of MongoDB Masters • Physics and Philosophy Other things ...
5. @flaper87 What does integrating a system mean? Theory
6. @flaper87 Vertical Integration Star Integration Horizontal Integration Theory
7. @flaper87 Horizontal Integration from an application's perspective Theory
8. @flaper87 Methods FilesDatabases RPC Messaging
9. @flaper87 Speaking of files Probably one of the oldest method Good for few and very specific cases Try not to use it
10. @flaper87 Methods Databases RPC Messaging Files
11. @flaper87 Speaking of databases Asynchronous data-wise Not a message broker Probably the most common Great for storing states
12. @flaper87 Methods RPCMessaging Files Databases
13. @flaper87 Speaking of RPC Remote Procedure Calls Most used throughout OpenStack Message's channels may vary (database, broker, etc) Tightly coupled
14. @flaper87 Methods Messaging Files Databases RPC
15. @flaper87 Speaking of Messaging Loosely coupled Add more complexity Commonly used for notifications May depend on message routers, transformation, etc.
16. @flaper87 OpenStack's case Shared Nothing Architecture Databases (Inter-service) RPC (Inter-service) Messaging (Cross-service)
17. @flaper87 Brokers can be are a PITA ← Scaling brokers is hard →
18. @flaper87 Brokers can be are a PITA Brokers need lot of memory depending on your use-case
19. @flaper87 Brokers can be are a PITA Brokers need storage if you want messages to be durable
20. @flaper87 Brokers can be are a PITA Prefer federation over centralization AMQP 1.0 Message Router (qpid-dispatch)
21. @flaper87 Tips / Tricks Transmission protocol matters
22. @flaper87 Tips / Tricks Use versions for your wire protocol
23. @flaper87 Tips / Tricks Keep everything explicit
24. @flaper87 Tips / Tricks Design by contract
25. @flaper87 Tips / Tricks Keep services isolated... As much as possible.
26. @flaper87 Q&A Thanks we're hiring