Distributed named pipes and other inter-services communication

Talk at DevOpsCon Berlin, see https://devopsconference.de/session/distributed-named-pipes-and-other-inter-services-communication/

Transcript

1. Distributed Named Pipes … and other inter-services communication Michael Hausenblas
 Developer Advocate Kubernetes & OpenShift
 2017-06-13, DevOpsCon Berlin
2. Hit me up on Twitter: @mhausenblas 2 admin SRE developer QA architect PM PHB
3. Hit me up on Twitter: @mhausenblas 3 What’s the problem we’re trying to solve?
4. Hit me up on Twitter: @mhausenblas 4 Motivation • multiple processes • dependencies • events & notifications • workflow/stages
5. Hit me up on Twitter: @mhausenblas 5 • local interprocess communication • read/write for multiple processes • FIFO • POSIX, Win* Named pipes 101
6. Hit me up on Twitter: @mhausenblas 6 Named pipes 101 $ mkfifo mypipe $ while true ; do \ printf 'NOW: ' > mypipe; date > mypipe ; \
 sleep 5 ; \
 done $ watch cat < mypipe
7. Hit me up on Twitter: @mhausenblas 7 What if processes run on different machines? • fallacies of distributed computing • The network is reliable • Latency is zero • Bandwidth is infinite • The network is secure • Topology doesn't change • There is one administrator • Transport cost is zero • The network is homogeneous • discovery
8. Hit me up on Twitter: @mhausenblas 8 Use cases • Work queues
 https://github.com/Adron/testing-aws-sqs-worker • Event dispatching in microservices
 https://blog.karmawifi.com/how-we-build-microservices-at-karma-71497a89bfb4 • Coordinate Function-as-a-Service executions
 https://cloudonaut.io/integrate-sqs-and-lambda-serverless-architecture-for-asynchronous-workloads/
9. Hit me up on Twitter: @mhausenblas 9 Introducing distributed named pipes • a distributed version of *nix named pipes • enabling IPC between distributed processes • more of a pattern than an implementation http://dnpip.es
10. Hit me up on Twitter: @mhausenblas 10
11. Hit me up on Twitter: @mhausenblas 11 • Donald Knuth, The Art of Computer Programming • Gang of four, Design Patterns: Elements of reusable object-oriented programming • Brendan Burns, Designing Distributed Systems • Sidecar • Adapter • Replicated load-balanced service dnpipes—a pattern
12. Hit me up on Twitter: @mhausenblas 12 dnpipes demo mkfifo xxx … create topic xxx in Kafka > xxx … produce message in xxx@broker < xxx … consume message from xxx@broker
13. Hit me up on Twitter: @mhausenblas 13 dnpipes demo
14. Hit me up on Twitter: @mhausenblas 14 dnpipes demo $ oc project devopscon $ oc new-app mjelen/zookeeper $ oc new-app mjelen/kafka $ kubectl exec $(oc get po | grep kafka | awk '{print $1}') -i -t -- bash # bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --partitions 1 \
 --replication-factor 1 --topic test # bin/kafka-topics.sh --list --zookeeper zookeeper:2181 # bin/kafka-console-producer.sh --broker-list kafka:9092 --topic test # bin/kafka-console-consumer.sh --bootstrap-server zookeeper:2181 \
 --topic test --from-beginning
15. Hit me up on Twitter: @mhausenblas 15 • DCOM/OLE • CORBA • AMQP • HTTP/REST-style • gRPC et al Prior art and related stuff
16. Hit me up on Twitter: @mhausenblas 16 • polling • hooks • ‘distributed pipes’ • https://github.com/lukasmartinelli/pipecat • http://www.dest-unreach.org/socat/ Prior art and related stuff
17. Hit me up on Twitter: @mhausenblas 17 What does the community think? https://twitter.com/jaykreps/status/808337482865463296
18. Hit me up on Twitter: @mhausenblas 18 • https://news.ycombinator.com/item?id=13151230 • https://lobste.rs/s/ymgqr5/distributed_named_pipes • https://hackernoon.com/a-quick-note-on-distributed-named-pipes-c0ec9cb1e909 Community feedback, criticism & limitations
19. Hit me up on Twitter: @mhausenblas 19 Community feedback, criticism & limitations • behavior/semantics compared to ‘local’ named pipes • payload • reference implementation
20. plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews openshift.com