The missing piece: when Docker networking unleashes software architecture 2.0

This is a talk I gave with Adrien Blind at Dockercon Barcelona

Transcript

1. The missing piece: when Docker networking unleashes software architecture 2.0 A. Blind DevOps coach Societe Generale @adrienblind L. Grangeau Solutions architect Finaxys @laurentgrangeau
2. Agenda 2 - Starters Docker networking & volume features discovered 4 - Dessert Taste-an-app 1 - Apetizer Back on current Docker paradigms 3 - Main course Application architecture shifts
3. Back on current docker paradigms
4. Back on Docker paradigms ‘’A universal, self-sufficient and standard artifact embedding an app module, and its subsequent infrastructure configuration’’ Immutable Versionned Light Portable Disposable Programatic Social Incremental  It’s mainly focused on enclosing computing capabilities: what about storage ? Network ?
5. Networking & volume features discovered
6. Docker networking
7. Docker networking
8. Docker networking The Container Network Model (CNM) A docker container Endpoint A docker container Endpoint A docker container Endpoint Endpoint Network sandbox Network sandbox Network sandbox Front network Back network
9. Docker networking $ docker network create mynetwork 5000dec7c180a63d87031de7e6bfcf2b25cf1e5daef6338f16fbd4 451210a938 $ docker network create –d overlay multihostnetwork e6537b859359843bc02392245ab226070f79dbf87be2d492969c84 3f89fb6de6
10. Docker networking $ docker network inspect mynetwork [ { "Name": "mynetwork", "Id": "5000dec7c180a63d87031de7e6bfcf2b25cf1e5daef6338f16fbd4451 210a938", "Scope": "local", "Driver": "bridge", "IPAM": { "Driver": "default", "Config": [ {} ] }, "Containers": {}, "Options": {} } ]
11. Docker networking Host Host Host Host SDNs SDN 1 SDN 2 SDN 3
12. Docker networking
13. Docker networking Docker Compose evolved to embrace new networking features $ docker-compose --x-networking --x-network-driver=overlay up $ docker-compose up
14. Docker volumes
15. Docker volumes Host file system Host file system ‘’Former data management locked in a host’’
16. Docker volumes Host file system Container Volume ‘’Containers mount a volume which may be backed externaly’’
17. Docker volumes $ docker volume create –d volplugin --name pool/name Cf872ca21d27843f6b6319ac1a34390dd38d94ed4649cd985456d5 23fb05d4cc $ docker run –d –p 8080:8080 –v pool/name:/var/jenkins_home jenkins 96aec6f4e45e050dfb4f75a1009e7f105bced5b406752e62d47061 5d07348b07
18. Docker volumes $ docker volume ls DRIVER VOLUME NAME local cf872ca21d27843f6b6319ac1a34390… local f19f50251f48c64a6b33a5c637c2330… $ docker volume inspect cf872ca21d27843f6b6319ac1a34390dd38d94… [ { "Name": "cf872ca21d27843f6b6319ac1a34390dd38d94…", "Driver": "local", "Mountpoint": "/mnt/sda1/var/lib/docker/volumes/[…]/_data" } ]
19. Take-away
20. Application Compute (Run containers) Docker building blocks
21. Application Compute (Run containers) Storage (Volumes) ‘’Immutability of containers, resiliency & scalability led to data externalization in separate objects’’
22. ‘’Fine-granularity of containers led to closely interconnect them’’ Application Compute (Run containers) Storage (Volumes) Transport (Network)
23. ‘’The whole topology can now be described’’ Application Compute (Run containers) Storage (Volumes) Transport (Network) Topology (Compose)
24. ‘’Docker finally shifted to object-oriented infra. architecture’’ Application Compute (Run containers) Storage (Volumes) Transport (Network) Topology (Compose) CaaS platform (Swarm, Machine...)
25. Application architecture shifts
26. Security paradigms shifts
27. Security paradigms shifts Your IT opens up • Externalization (housing, hosting) • Cloud (IaaS/PaaS/SaaS) Open up your IS • B2B, services exposition • Multi tenancy More & more breaches appears in your Great Wall of China!
28. Security paradigms shifts The necessary porosity of your IS requires to stick security closer to each application: sandbox your apps and expose protected interfaces! Network is part of application topology Security is an app topic, not just infra. concern Onboard security in feature teamSecDevOps
29. Network paradigms shifts VM VM VM VM VM VM VM VM VM Internet Internet DMZ Physical overview Logical overview Tenant #1 Tenant #2 LAN LAN DMZ1 DMZ2 Traditional networks relies a lot on low layers (L2, etc.) Application topologies are quite different from physical ones
30. Network paradigms shifts SDNs propose network solutions embracing cloud paradigms Massively multi-tenant Thousands tenants, massively scalable Easy & fast (de)provisioning Infra as code, API centric Infrastructure agnostic L3, does not stick with lower levels (physical designs, vlans & co) Decouple infrastructure & tenants lifecycles Cross technology, vendor agnostic
31. From Enterprise Services buses to full-mesh topologies ESB Service Service Service Service Service > Service Service Service Service Service Micro services
32. Fine-grained, highly decoupled and atomic purpose centric services Designed for failure Multi-versioned Scalable Micro services Stateless Share-nothing Immutable Continuously delivered Distributed
33. Service consumer Service provider Registry 2. Find 1. Publish 3. Bind Leverage on a Service registry to discover where are services located Micro services
34. Resilience & scalability: apps problem now! Vertical > horizontal Apps designed for failure & scalability Data to be externalized Dumber infrastructure  Structured: MongoDB, Hadoop, Cassandra, Elastic Search...  Binaries: object storage with Ceph, OpenStack Swift...  Helpful patterns: stateless, multi-versioning, loose coupling...  Infrastructure rationalization  Low-cost, poor-SLA commodity
35. « Organizations which design systems... are constrained to produce designs which are copies of the communication structures of these organizations ». - M. Conway, 1968 Consider shifting your organization if you wish to shift your architecture Forget about the central architects myth of organizing, integrating everything Consider changing your organization to expect changing the architecture!  promote feature teams Organization
36. Docker suits perfectly new applications challenges Create docker networks to isolate applications Docker container properties fits micro-services challenges Resilience & scalability is mostly about multiplying containers Expect to discuss roles shift in organization
37. Taste-an-app
38. Execution platform Registry Docker-machine Docker-swarm The registrator discovers new containers and feeds the registry
39. Application design Provider micro service Consumers The python app module exposes a REST service searching information in the MongoDB The NGINX reverse proxy forward app. requests on one of the python instance registered in Consul Find
40. Application topology & runtime The whole application topology is stored as: docker-compose yaml file docker-compose args (aka --x-networking & --x-network-driver) You can scale up or down the python instances of the micro-service using traditionnal docker-compose scale command
41. Network view Only the load balancer VIP is exposed externally A WAF instance could secure this entrypoint SDN « dockerconeu15 » Host network Provider micro service Consumers
42. Network view - advanced Provider micro service Consumers SDN « front » SDN « back » Host network Back Middle Front ‘’To enhance security you may decouple each application tier’’
43. Zoom on the registry usages Between micro-services, consumers asks the registry where a desired micro-service is located Inside a micro-service, NGINX is made aware of the backend API instances available, via the registry At container level, the registrator enable to registers any container instances, grouped per type At infrastructure level, the registry is used by swarm (internally) to be aware of the cluster’s participants Noticed the different usages of a registry ? You may consider using different registries for each usage : for example an internal registry for the micro service internal topology
44. Conclusions
45. Docker shifted from universal containers to object- oriented infrastructure Security is an app concern Software is eating the world: application architecture is the key, infrastructure is commodity
46. Thank you! A. Blind @adrienblind adrien.blind@sgcib.com L. Grangeau @laurentgrangeau laurent.grangeau@gmail.com