Distributed named pipes and other inter-services communication

Distributed named pipes and other inter-services communication

5c3807aaaf0ffefe6c75e3dbbb8588b5?s=128

Michael Hausenblas

June 13, 2017
Tweet

Transcript

  1. Distributed Named Pipes … and other inter-services communication Michael Hausenblas


    Developer Advocate Kubernetes & OpenShift
 2017-06-13, DevOpsCon Berlin
  2. Hit me up on Twitter: @mhausenblas 2 admin SRE developer

    QA architect PM PHB
  3. Hit me up on Twitter: @mhausenblas 3 What’s the problem

    we’re trying to solve?
  4. Hit me up on Twitter: @mhausenblas 4 Motivation • multiple

    processes • dependencies • events & notifications • workflow/stages
  5. Hit me up on Twitter: @mhausenblas 5 • local interprocess

    communication • read/write for multiple processes • FIFO • POSIX, Win* Named pipes 101
  6. Hit me up on Twitter: @mhausenblas 6 Named pipes 101

    $ mkfifo mypipe $ while true ; do \ printf 'NOW: ' > mypipe; date > mypipe ; \
 sleep 5 ; \
 done $ watch cat < mypipe
  7. Hit me up on Twitter: @mhausenblas 7 What if processes

    run on different machines? • fallacies of distributed computing • The network is reliable • Latency is zero • Bandwidth is infinite • The network is secure • Topology doesn't change • There is one administrator • Transport cost is zero • The network is homogeneous • discovery
  8. Hit me up on Twitter: @mhausenblas 8 Use cases •

    Work queues
 https://github.com/Adron/testing-aws-sqs-worker • Event dispatching in microservices
 https://blog.karmawifi.com/how-we-build-microservices-at-karma-71497a89bfb4 • Coordinate Function-as-a-Service executions
 https://cloudonaut.io/integrate-sqs-and-lambda-serverless-architecture-for-asynchronous-workloads/
  9. Hit me up on Twitter: @mhausenblas 9 Introducing distributed named

    pipes • a distributed version of *nix named pipes • enabling IPC between distributed processes • more of a pattern than an implementation http://dnpip.es
  10. Hit me up on Twitter: @mhausenblas 10

  11. Hit me up on Twitter: @mhausenblas 11 • Donald Knuth,

    The Art of Computer Programming • Gang of four, Design Patterns: Elements of reusable object-oriented programming • Brendan Burns, Designing Distributed Systems • Sidecar • Adapter • Replicated load-balanced service dnpipes—a pattern
  12. Hit me up on Twitter: @mhausenblas 12 dnpipes demo mkfifo

    xxx … create topic xxx in Kafka > xxx … produce message in xxx@broker < xxx … consume message from xxx@broker
  13. Hit me up on Twitter: @mhausenblas 13 dnpipes demo

  14. Hit me up on Twitter: @mhausenblas 14 dnpipes demo $

    oc project devopscon $ oc new-app mjelen/zookeeper $ oc new-app mjelen/kafka $ kubectl exec $(oc get po | grep kafka | awk '{print $1}') -i -t -- bash # bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --partitions 1 \
 --replication-factor 1 --topic test # bin/kafka-topics.sh --list --zookeeper zookeeper:2181 # bin/kafka-console-producer.sh --broker-list kafka:9092 --topic test # bin/kafka-console-consumer.sh --bootstrap-server zookeeper:2181 \
 --topic test --from-beginning
  15. Hit me up on Twitter: @mhausenblas 15 • DCOM/OLE •

    CORBA • AMQP • HTTP/REST-style • gRPC et al Prior art and related stuff
  16. Hit me up on Twitter: @mhausenblas 16 • polling •

    hooks • ‘distributed pipes’ • https://github.com/lukasmartinelli/pipecat • http://www.dest-unreach.org/socat/ Prior art and related stuff
  17. Hit me up on Twitter: @mhausenblas 17 What does the

    community think? https://twitter.com/jaykreps/status/808337482865463296
  18. Hit me up on Twitter: @mhausenblas 18 • https://news.ycombinator.com/item?id=13151230 •

    https://lobste.rs/s/ymgqr5/distributed_named_pipes • https://hackernoon.com/a-quick-note-on-distributed-named-pipes-c0ec9cb1e909 Community feedback, criticism & limitations
  19. Hit me up on Twitter: @mhausenblas 19 Community feedback, criticism

    & limitations • behavior/semantics compared to ‘local’ named pipes • payload • reference implementation
  20. plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews openshift.com