Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Distributed named pipes and other inter-services communication

Distributed named pipes and other inter-services communication

Michael Hausenblas

June 13, 2017
Tweet

More Decks by Michael Hausenblas

Other Decks in Technology

Transcript

  1. Distributed Named Pipes
    … and other inter-services communication
    Michael Hausenblas

    Developer Advocate Kubernetes & OpenShift

    2017-06-13, DevOpsCon Berlin

    View Slide

  2. Hit me up on Twitter: @mhausenblas
    2
    admin
    SRE
    developer
    QA
    architect
    PM
    PHB

    View Slide

  3. Hit me up on Twitter: @mhausenblas
    3
    What’s the problem we’re trying to solve?

    View Slide

  4. Hit me up on Twitter: @mhausenblas
    4
    Motivation
    • multiple processes
    • dependencies
    • events & notifications
    • workflow/stages

    View Slide

  5. Hit me up on Twitter: @mhausenblas
    5
    • local interprocess communication
    • read/write for multiple processes
    • FIFO
    • POSIX, Win*
    Named pipes 101

    View Slide

  6. Hit me up on Twitter: @mhausenblas
    6
    Named pipes 101
    $ mkfifo mypipe
    $ while true ; do \
    printf 'NOW: ' > mypipe; date > mypipe ; \

    sleep 5 ; \

    done
    $ watch cat < mypipe

    View Slide

  7. Hit me up on Twitter: @mhausenblas
    7
    What if processes run on different machines?
    • fallacies of distributed computing
    • The network is reliable
    • Latency is zero
    • Bandwidth is infinite
    • The network is secure
    • Topology doesn't change
    • There is one administrator
    • Transport cost is zero
    • The network is homogeneous
    • discovery

    View Slide

  8. Hit me up on Twitter: @mhausenblas
    8
    Use cases
    • Work queues

    https://github.com/Adron/testing-aws-sqs-worker
    • Event dispatching in microservices

    https://blog.karmawifi.com/how-we-build-microservices-at-karma-71497a89bfb4
    • Coordinate Function-as-a-Service executions

    https://cloudonaut.io/integrate-sqs-and-lambda-serverless-architecture-for-asynchronous-workloads/

    View Slide

  9. Hit me up on Twitter: @mhausenblas
    9
    Introducing distributed named pipes
    • a distributed version of *nix named pipes
    • enabling IPC between distributed processes
    • more of a pattern than an implementation
    http://dnpip.es

    View Slide

  10. Hit me up on Twitter: @mhausenblas
    10

    View Slide

  11. Hit me up on Twitter: @mhausenblas
    11
    • Donald Knuth, The Art of Computer Programming
    • Gang of four, Design Patterns: Elements of
    reusable object-oriented programming
    • Brendan Burns, Designing Distributed Systems
    • Sidecar
    • Adapter
    • Replicated load-balanced service
    dnpipes—a pattern

    View Slide

  12. Hit me up on Twitter: @mhausenblas
    12
    dnpipes demo
    mkfifo xxx … create topic xxx in Kafka
    > xxx … produce message in [email protected]
    < xxx … consume message from [email protected]

    View Slide

  13. Hit me up on Twitter: @mhausenblas
    13
    dnpipes demo

    View Slide

  14. Hit me up on Twitter: @mhausenblas
    14
    dnpipes demo
    $ oc project devopscon
    $ oc new-app mjelen/zookeeper
    $ oc new-app mjelen/kafka
    $ kubectl exec $(oc get po | grep kafka | awk '{print $1}') -i -t -- bash
    # bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --partitions 1 \

    --replication-factor 1 --topic test
    # bin/kafka-topics.sh --list --zookeeper zookeeper:2181
    # bin/kafka-console-producer.sh --broker-list kafka:9092 --topic test
    # bin/kafka-console-consumer.sh --bootstrap-server zookeeper:2181 \

    --topic test --from-beginning

    View Slide

  15. Hit me up on Twitter: @mhausenblas
    15
    • DCOM/OLE
    • CORBA
    • AMQP
    • HTTP/REST-style
    • gRPC et al
    Prior art and related stuff

    View Slide

  16. Hit me up on Twitter: @mhausenblas
    16
    • polling
    • hooks
    • ‘distributed pipes’
    • https://github.com/lukasmartinelli/pipecat
    • http://www.dest-unreach.org/socat/
    Prior art and related stuff

    View Slide

  17. Hit me up on Twitter: @mhausenblas
    17
    What does the community think?
    https://twitter.com/jaykreps/status/808337482865463296

    View Slide

  18. Hit me up on Twitter: @mhausenblas
    18
    • https://news.ycombinator.com/item?id=13151230
    • https://lobste.rs/s/ymgqr5/distributed_named_pipes
    • https://hackernoon.com/a-quick-note-on-distributed-named-pipes-c0ec9cb1e909
    Community feedback, criticism & limitations

    View Slide

  19. Hit me up on Twitter: @mhausenblas
    19
    Community feedback, criticism & limitations
    • behavior/semantics compared to ‘local’ named pipes
    • payload
    • reference implementation

    View Slide

  20. plus.google.com/+RedHat
    linkedin.com/company/red-hat
    youtube.com/user/RedHatVideos
    facebook.com/redhatinc
    twitter.com/RedHatNews
    openshift.com

    View Slide