Spring XD Webinar - 02/17/2015

7d5161a154a9f0958864fb12bb9ba966?s=47 Sabby Anandan
February 17, 2015

Spring XD Webinar - 02/17/2015

Developer Productivity for Data-driven Apps at Scale

7d5161a154a9f0958864fb12bb9ba966?s=128

Sabby Anandan

February 17, 2015
Tweet

Transcript

  1. 1 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring XD Developer Productivity for Data-driven Apps at Scale Sabby Anandan, Mark Pollack, Mark Fisher
  2. 2 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ © Copyright 2014 Pivotal. All rights reserved. Your Presenters Mark Fisher Spring XD (co-lead) Mark Pollack Spring XD (co-lead) Sabby Anandan Product Manager
  3. 3 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ •  Architecture •  Functional Stream Processing •  Spark Streaming and Spark Job •  Kafka and Data Partitions Agenda
  4. 4 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ •  Unified platform for stream and batch workloads •  Evolve streams and jobs in isolation •  Eliminates technical challenges with distributed computing, fault- tolerance, or high-availability •  Open and Extensible runtime •  Portable: Runs anywhere there is a JVM •  Read/write Hadoop without integration headaches Purpose
  5. 5 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Distributed Runtime
  6. 6 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ © Copyright 2014 Pivotal. All rights reserved. Hadoop without integration headaches..
  7. 7 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Easy to Setup and Run
  8. 8 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ © Copyright 2014 Pivotal. All rights reserved. Functional Stream Processing
  9. 9 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Reactor Stream API Source Reactor Processor Sink Bus Bus Stream API Input Stream Output Stream
  10. 10 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ RxJava Observable API Source RxJava Processor Sink Bus Bus Observable API Input Stream Output Stream
  11. 11 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ © Copyright 2014 Pivotal. All rights reserved. Spring XD + Spark
  12. 12 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spark Fast and general engine for large-scale data processing. What is Spark in Spring XD? •  Leverage OOTB source/sinks from XD and delegate computation logic to Spark cluster •  Automatic failover capabilities in case of Spark Driver failure
  13. 13 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spark Streaming API Source Spark Streaming Processor Sink Bus Bus Spark Streaming API Input Stream Output Stream
  14. 14 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ © Copyright 2014 Pivotal. All rights reserved. Spring XD + Python
  15. 15 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What is Python in Spring XD? •  OOTB ‘shell’ as processor and sink to execute external shell commands such as a Python script Python
  16. 16 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ © Copyright 2014 Pivotal. All rights reserved. Spring XD + Kafka
  17. 17 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Kafka Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. What is Kafka in Spring XD? •  As a message bus to send data from the output of one module to the input of the next module •  As a source module to ingest data from Kafka topics •  As a sink module to ingest data into Kafka topics
  18. 18 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ © Copyright 2014 Pivotal. All rights reserved. Looking Ahead
  19. 19 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ •  Custom Modules and HDFS •  UI Canvas: ◦  create, visualize and deploy data pipelines ◦  metrics and monitoring capabilities •  Enrichment: OOTB modules/jobs •  Apache Ambari Support •  Security Enhancements •  Spring XD on Pivotal Cloud Foundry (GA Release: May 2015) Roadmap
  20. 20 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ •  Project Site ◦  GitHub ◦  Wiki •  Spring XD Samples •  JIRA: Features and Issues Resources
  21. 21 Unless otherwise indicated, these slides are © 2013-2015 Pivotal

    Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Learn More. Stay Connected. •  Project Site ◦  GitHub ◦  Wiki •  Spring XD Samples •  JIRA: Features and Issues Twitter: twitter.com/springcentral YouTube: spring.io/video LinkedIn: spring.io/linkedin Google Plus: spring.io/gplus