Spring XD Webinar - 02/17/2015

Developer Productivity for Data-driven Apps at Scale

Transcript

1. 1 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
   Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
   Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
   Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
   Spring XD
   Developer Productivity for
   Data-driven Apps at Scale
   Sabby Anandan, Mark Pollack, Mark Fisher
   

   View Slide

2. 2 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
   Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
   © Copyright 2014 Pivotal. All rights reserved.
   Your Presenters
   Mark Fisher
   Spring XD
   (co-lead)
   Mark Pollack
   Spring XD
   (co-lead)
   Sabby
   Anandan
   Product
   Manager
   

   View Slide

3. 3 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
   Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
   ●  Architecture
   ●  Functional Stream Processing
   ●  Spark Streaming and Spark Job
   ●  Kafka and Data Partitions
   Agenda
   

   View Slide

4. 4 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
   Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
   ●  Unified platform for stream and batch workloads
   ●  Evolve streams and jobs in isolation
   ●  Eliminates technical challenges with distributed computing, fault-
   tolerance, or high-availability
   ●  Open and Extensible runtime
   ●  Portable: Runs anywhere there is a JVM
   ●  Read/write Hadoop without integration headaches
   Purpose
   

   View Slide

5. 5 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
   Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
   Distributed Runtime
   

   View Slide

6. 6 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
   Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
   © Copyright 2014 Pivotal. All rights reserved.
   Hadoop without integration headaches..
   

   View Slide

7. 7 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
   Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
   Easy to Setup and Run
   

   View Slide

8. 8 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
   Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
   © Copyright 2014 Pivotal. All rights reserved.
   Functional Stream
   Processing
   

   View Slide

9. 9 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
   Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
   Reactor Stream API
   Source Reactor Processor Sink
   Bus Bus
   Stream API
   Input
   Stream
   Output
   Stream
   

   View Slide

10. 10 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    RxJava Observable API
    Source RxJava Processor Sink
    Bus Bus
    Observable
    API
    Input
    Stream
    Output
    Stream
    

    View Slide

11. 11 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    © Copyright 2014 Pivotal. All rights reserved.
    Spring XD + Spark
    

    View Slide

12. 12 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spark
    Fast and general engine for large-scale data processing.
    What is Spark in Spring XD?
    ●  Leverage OOTB source/sinks from XD and delegate computation
    logic to Spark cluster
    ●  Automatic failover capabilities in case of Spark Driver failure
    

    View Slide

13. 13 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spark Streaming API
    Source
    Spark Streaming
    Processor Sink
    Bus Bus
    Spark
    Streaming
    API
    Input
    Stream
    Output
    Stream
    

    View Slide

14. 14 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    © Copyright 2014 Pivotal. All rights reserved.
    Spring XD + Python
    

    View Slide

15. 15 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    What is Python in Spring XD?
    ●  OOTB ‘shell’ as processor and sink to execute external shell
    commands such as a Python script
    Python
    

    View Slide

16. 16 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    © Copyright 2014 Pivotal. All rights reserved.
    Spring XD + Kafka
    

    View Slide

17. 17 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Kafka
    Apache Kafka is publish-subscribe messaging rethought as
    a distributed commit log.
    What is Kafka in Spring XD?
    ●  As a message bus to send data from the output of one module to the
    input of the next module
    ●  As a source module to ingest data from Kafka topics
    ●  As a sink module to ingest data into Kafka topics
    

    View Slide

18. 18 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    © Copyright 2014 Pivotal. All rights reserved.
    Looking Ahead
    

    View Slide

19. 19 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    ●  Custom Modules and HDFS
    ●  UI Canvas:
    ○  create, visualize and deploy data pipelines
    ○  metrics and monitoring capabilities
    ●  Enrichment: OOTB modules/jobs
    ●  Apache Ambari Support
    ●  Security Enhancements
    ●  Spring XD on Pivotal Cloud Foundry (GA Release: May 2015)
    Roadmap
    

    View Slide

20. 20 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    ●  Project Site
    ○  GitHub
    ○  Wiki
    ●  Spring XD Samples
    ●  JIRA: Features and Issues
    Resources
    

    View Slide

21. 21 Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Learn More. Stay Connected.
    ●  Project Site
    ○  GitHub
    ○  Wiki
    ●  Spring XD Samples
    ●  JIRA: Features and Issues
    Twitter: twitter.com/springcentral
    YouTube: spring.io/video
    LinkedIn: spring.io/linkedin
    Google Plus: spring.io/gplus
    

    View Slide