Upgrade to Pro — share decks privately, control downloads, hide ads and more …

UrbonBayesPere_BerlinBuzzwords18_ConnectingDataInfraWithDataFlow.pdf

Pere Urbón
June 13, 2018
120

 UrbonBayesPere_BerlinBuzzwords18_ConnectingDataInfraWithDataFlow.pdf

Pere Urbón

June 13, 2018
Tweet

Transcript

  1. @purbon
    Connecting the data
    infrastructure with the
    DataFlow

    View full-size slide

  2. @purbon
    Pere Urbon-Bayes
    Software Architect
    pere.urbon@{gmail.com, acm.org}

    View full-size slide

  3. Topics for Today
    • Integration patterns for the enterprise startup.
    • What is Apache NIFI.
    • Examples
    • NiFi on operation (best practises).

    View full-size slide

  4. @purbon
    Integrate all the
    things!

    View full-size slide

  5. @purbon
    Enterprise integration is the task of making
    separate applications work together to produce
    an unified set of functionality.
    The applications probably run on multiple
    computers, which may be geographically
    dispersed.

    View full-size slide

  6. @purbon
    Some application might need to be integrated
    even though they were not designed for
    integration and can not be changed.
    This issues, and others, are what makes
    application integration difficult.

    View full-size slide

  7. @purbon
    Each integration faces different needs and
    criteria, we can group them as
    Application coupling
    Integration simplicity
    Data formats and timeliness
    Data or functionality
    Communication

    View full-size slide

  8. @purbon
    There is only a limited set of integration
    options

    View full-size slide

  9. @purbon
    File transfer

    View full-size slide

  10. @purbon
    Shared database

    View full-size slide

  11. @purbon
    RPC invoke

    View full-size slide

  12. @purbon
    Messaging

    View full-size slide

  13. @purbon
    Enterprise Integration Patterns

    View full-size slide

  14. @purbon
    What is Apache NiFi?

    View full-size slide

  15. @purbon
    An easy to use, powerful, and reliable system to
    process and distribute data.
    Web-based interface
    Highly configurable
    Data Provenance
    Designed for extension
    Secure

    View full-size slide

  16. @purbon
    NiFi was build to automate the flow of data
    between systems.
    an automated and managed flow of information
    between systems.
    But what is Dataflow?

    View full-size slide

  17. @purbon
    How Apache NiFi look like

    View full-size slide

  18. @purbon
    Concepts behind Apache NiFi

    View full-size slide

  19. @purbon
    A Flow file

    View full-size slide

  20. @purbon
    The Flow file Processor

    View full-size slide

  21. @purbon
    A Connection

    View full-size slide

  22. @purbon
    A Process Group

    View full-size slide

  23. @purbon
    Apache NiFi Architecture
    Distributed using Apache Zookeper

    View full-size slide

  24. @purbon
    Let’s take a closer
    look…

    View full-size slide

  25. @purbon
    Apache NiFI
    Operations

    View full-size slide

  26. @purbon
    Maximum file handles
    hard nofile 50000
    soft nofile 50000
    /etc/security/limits.conf

    View full-size slide

  27. @purbon
    Maximum forked Procs
    hard nproc 10000
    soft nproc 10000
    /etc/security/limits.conf
    /etc/security/limits.d/90-nproc.conf

    View full-size slide

  28. @purbon
    Increase number of TCP sockets
    sudo sysctl -w net.ipv4.ip_local_port_range="10000 65000"

    View full-size slide

  29. @purbon
    Timeout sockets in TIMED_WAIT state
    sudo sysctl -w
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait="1"

    View full-size slide

  30. @purbon
    Never SWAP
    vm.swappiness = 0
    /etc/sysctl.conf
    /dev/sda7 /chroot ext2 defaults, noatime 1 2
    /etc/fstab

    View full-size slide

  31. @purbon
    Thanks a lot!
    Questions?
    disagreements? threads?
    Pere Urbon-Bayes
    Data Wrangler
    pere.urbon@{gmail.com, acm.org}

    View full-size slide