Distributed named pipes and other inter-services communication

Talk at DevOpsCon Berlin, see https://devopsconference.de/session/distributed-named-pipes-and-other-inter-services-communication/

Transcript

1. Distributed Named Pipes
   … and other inter-services communication
   Michael Hausenblas

   Developer Advocate Kubernetes & OpenShift

   2017-06-13, DevOpsCon Berlin
   

   View Slide

2. Hit me up on Twitter: @mhausenblas
   2
   admin
   SRE
   developer
   QA
   architect
   PM
   PHB
   

   View Slide

3. Hit me up on Twitter: @mhausenblas
   3
   What’s the problem we’re trying to solve?
   

   View Slide

4. Hit me up on Twitter: @mhausenblas
   4
   Motivation
   • multiple processes
   • dependencies
   • events & notifications
   • workflow/stages
   

   View Slide

5. Hit me up on Twitter: @mhausenblas
   5
   • local interprocess communication
   • read/write for multiple processes
   • FIFO
   • POSIX, Win*
   Named pipes 101
   

   View Slide

6. Hit me up on Twitter: @mhausenblas
   6
   Named pipes 101
   $ mkfifo mypipe
   $ while true ; do \
   printf 'NOW: ' > mypipe; date > mypipe ; \

   sleep 5 ; \

   done
   $ watch cat < mypipe
   

   View Slide

7. Hit me up on Twitter: @mhausenblas
   7
   What if processes run on different machines?
   • fallacies of distributed computing
   • The network is reliable
   • Latency is zero
   • Bandwidth is infinite
   • The network is secure
   • Topology doesn't change
   • There is one administrator
   • Transport cost is zero
   • The network is homogeneous
   • discovery
   

   View Slide

8. Hit me up on Twitter: @mhausenblas
   8
   Use cases
   • Work queues

   https://github.com/Adron/testing-aws-sqs-worker
   • Event dispatching in microservices

   https://blog.karmawifi.com/how-we-build-microservices-at-karma-71497a89bfb4
   • Coordinate Function-as-a-Service executions

   https://cloudonaut.io/integrate-sqs-and-lambda-serverless-architecture-for-asynchronous-workloads/
   

   View Slide

9. Hit me up on Twitter: @mhausenblas
   9
   Introducing distributed named pipes
   • a distributed version of *nix named pipes
   • enabling IPC between distributed processes
   • more of a pattern than an implementation
   http://dnpip.es
   

   View Slide

10. Hit me up on Twitter: @mhausenblas
    10
    

    View Slide

11. Hit me up on Twitter: @mhausenblas
    11
    • Donald Knuth, The Art of Computer Programming
    • Gang of four, Design Patterns: Elements of
    reusable object-oriented programming
    • Brendan Burns, Designing Distributed Systems
    • Sidecar
    • Adapter
    • Replicated load-balanced service
    dnpipes—a pattern
    

    View Slide

12. Hit me up on Twitter: @mhausenblas
    12
    dnpipes demo
    mkfifo xxx … create topic xxx in Kafka
    > xxx … produce message in xxx@broker
    < xxx … consume message from xxx@broker
    

    View Slide

13. Hit me up on Twitter: @mhausenblas
    13
    dnpipes demo
    

    View Slide

14. Hit me up on Twitter: @mhausenblas
    14
    dnpipes demo
    $ oc project devopscon
    $ oc new-app mjelen/zookeeper
    $ oc new-app mjelen/kafka
    $ kubectl exec $(oc get po | grep kafka | awk '{print $1}') -i -t -- bash
    # bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --partitions 1 \

    --replication-factor 1 --topic test
    # bin/kafka-topics.sh --list --zookeeper zookeeper:2181
    # bin/kafka-console-producer.sh --broker-list kafka:9092 --topic test
    # bin/kafka-console-consumer.sh --bootstrap-server zookeeper:2181 \

    --topic test --from-beginning
    

    View Slide

15. Hit me up on Twitter: @mhausenblas
    15
    • DCOM/OLE
    • CORBA
    • AMQP
    • HTTP/REST-style
    • gRPC et al
    Prior art and related stuff
    

    View Slide

16. Hit me up on Twitter: @mhausenblas
    16
    • polling
    • hooks
    • ‘distributed pipes’
    • https://github.com/lukasmartinelli/pipecat
    • http://www.dest-unreach.org/socat/
    Prior art and related stuff
    

    View Slide

17. Hit me up on Twitter: @mhausenblas
    17
    What does the community think?
    https://twitter.com/jaykreps/status/808337482865463296
    

    View Slide

18. Hit me up on Twitter: @mhausenblas
    18
    • https://news.ycombinator.com/item?id=13151230
    • https://lobste.rs/s/ymgqr5/distributed_named_pipes
    • https://hackernoon.com/a-quick-note-on-distributed-named-pipes-c0ec9cb1e909
    Community feedback, criticism & limitations
    

    View Slide

19. Hit me up on Twitter: @mhausenblas
    19
    Community feedback, criticism & limitations
    • behavior/semantics compared to ‘local’ named pipes
    • payload
    • reference implementation
    

    View Slide

20. plus.google.com/+RedHat
    linkedin.com/company/red-hat
    youtube.com/user/RedHatVideos
    facebook.com/redhatinc
    twitter.com/RedHatNews
    openshift.com
    

    View Slide