Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building a Distributed Data Processing Pipeline Using Spring Cloud Data Flow

833d3a69650b8ac7e0d85c39650c505d?s=47 Kenny Bastani
November 02, 2015

Building a Distributed Data Processing Pipeline Using Spring Cloud Data Flow

In this talk I will introduce you to Spring Cloud Data Flow, a tool set for building cloud-native JVM applications as a distributed data processing pipeline. We will take a look at some of the common patterns for processing data streams using Spring Boot applications that consume and produce streams of messages.

We will then dive into a microservice example project of a cloud-native data processing application built using Spring Boot and Spring Cloud Data Flow. Using this example project, I'll show you how to use a container scheduler to spin up a microservice cluster on a development machine that processes a stream of tweets in a data flow pipeline.

833d3a69650b8ac7e0d85c39650c505d?s=128

Kenny Bastani

November 02, 2015
Tweet

Transcript

  1. © 2014 Pivotal Software, Inc. All rights reserved. Distributed Data

    Processing with Spring Cloud Data Flow Kenny Bastani Spring Developer Advocate 1
  2. © 2014 Pivotal Software, Inc. All rights reserved. Speaker Intro

    2
  3. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Kenny Bastani 3
  4. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Agenda 4 1 Agenda 2 Microservices 3 What is Spring Boot? 4 What is Spring Cloud? 5 Lattice - Cloud Native Platform 6 Spring Cloud Data Flow 7 Streaming Analytics Example (Twitter) 9. Demo
  5. © 2014 Pivotal Software, Inc. All rights reserved. Microservices “Kind

    of like a head ache, but more of them.” 5
  6. © 2015 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Quick Explanation of Microservices • Each team gets one database and one service • Shared caches are platform provided services that are shared for consistency 6
  7. © 2015 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani 7
  8. © 2014 Pivotal Software, Inc. All rights reserved. Cloud Native

    Microservices 8
  9. © 2015 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Cloud-native Microservice Deployment • Each microservice can be containerized with their application dependencies • Containers get scheduled on virtual machines with an allotted resource policy 9
  10. © 2015 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Auto-scaling • An elastic runtime handles auto-scaling of VMs with cloud providers • Microservices should be load balanced vertically and not horizontally 10
  11. © 2015 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Composition & Orchestration • Each microservice needs to communicate outside containers • Service discovery provides an automatic method for finding other service dependencies 11
  12. © 2014 Pivotal Software, Inc. All rights reserved. Spring Boot

    A JVM micro-framework for building microservices 12
  13. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani What is Spring Boot? 13
  14. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Spring Boot Roles 14
  15. © 2015 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Automatic Configuration • An application class is annotated with @SpringBootApplication • Additional annotations are added to indicate the role of the Spring Boot application 15
  16. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Spring Boot for Microservices 16
  17. © 2014 Pivotal Software, Inc. All rights reserved. Spring Cloud

    A toolset designed for building distributed systems 17
  18. © 2015 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani What is Spring Cloud? • Spring Cloud provides a way to turn Spring Boot microservices into distributed applications 18
  19. © 2015 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani What is Spring Cloud? 19 ✴ Service Discovery ✴ API Gateway ✴ Circuit Breakers ✴ Distributed Tracing
  20. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Service Discovery & Configuration Service 20
  21. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Configuration Service 21
  22. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Service Discovery 22
  23. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani API Gateway 23
  24. © 2014 Pivotal Software, Inc. All rights reserved. But what

    about deployments? 24
  25. © 2014 Pivotal Software, Inc. All rights reserved. Lattice A

    cloud-native platform for deploying and scaling containers in production 25
  26. © 2015 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Containers, containers, containers • Lattice helps you manage Docker container deployments on clusters of VMs • Choose the cloud provider you want, deploys containers from Docker hub 26
  27. © 2014 Pivotal Software, Inc. All rights reserved. Spring Cloud

    Data Flow Java microservices that process data in a pipeline 27
  28. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Spring Cloud Data Flow • What is Spring Cloud Data Flow? – Spring Cloud Data Flow is a data processing pipeline that uses Spring Boot microservices – Each Spring Boot microservice takes in a message and produces a message, containing the data that you're processing 28
  29. © 2014 Pivotal Software, Inc. All rights reserved. How do

    we design a data processing pipeline? 29
  30. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Start with understanding your source data • What do you want to measure? – Trending analytics in real time 30
  31. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Understand the function of your pipeline • What does an individual message contain? – A single tweet 31
  32. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Understand what you want to measure • What do you want to filter? – A set of hash tags in the body of a Tweet 32
  33. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Understand the outputs • What do I want to measure over time? – The velocity of hash tag counts from tweets every second 33
  34. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani What is the result of our measurements? • A graph that shows the velocity of each tweet and its hash tags over time • Real time streaming analytics so we can make fast informed decisions 34
  35. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Data Processing Pipeline Example 35 Source Process Filter Counter Data Data Filter messages Transformation Increment counters Ingest messages Database
  36. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Scalable distributed data processing 36 Source Process Filter Counter Data Data Filter messages Transformation Increment counters Ingest messages Source Filter Process Counter Database
  37. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Input and output channels • Each Spring Boot microservice has an input channel and an output channel 37 Spring Boot Service Input Channel Output Channel Microservice Message queue Message queue
  38. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Input and output channels 38 Spring Boot Service Input Channel Output Channel Microservice Message queue Message queue Message Message Messages Messages
  39. © 2015 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Building Real-time Analytics on Twitter hash tags 39
  40. © 2014 Pivotal Software, Inc. All rights reserved. Setting up

    a data processing pipeline 40
  41. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani We want to build something like this: 41 Source Process Filter Counter Data Data Filter messages Transformation Increment counters Ingest messages Source Filter Process Counter Database
  42. © 2014 Pivotal Software, Inc. All rights reserved. Tweet Source

    Module 42
  43. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Responsibility of a source module • Ingest data from multiple sources, such as a streaming REST API endpoint or HDFS • Transform a stream into discrete messages that are uniformly distributed, such as an individual tweet • Output those messages to an output channel for the next service to process 43
  44. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Ingesting tweets • We start by building a Spring Boot source module that imports tweets from Twitter 44 Data Tweet Ingest tweets Source Module
  45. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Visualize the tweet source module 45 Spring Boot Streaming Endpoint Output Channel Source Module Twitter API Twitter stream Message Message Tweets Twitter Stream Service Channel:
 twitter-stream
  46. © 2014 Pivotal Software, Inc. All rights reserved. Adding a

    filter module 46
  47. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Responsibility of a filter module • Filter messages from the source module • Filters noise to increase quality of measurements in down stream modules • Example: – I only want to measure tweets containing the hash tag #java2days 47
  48. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Visualizing the filter module 48 Spring Boot Input Channel Output Channel Only tweets containing:
 
 #java2days Message Message Tweets Filter Service Channel:
 twitter-stream Channel:
 processor-stream
  49. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Our pipeline now looks like: 49 Source Filter Data Tweets #java2days Twitter Stream API Twitter Stream Filter Tweets …
  50. © 2014 Pivotal Software, Inc. All rights reserved. Adding a

    processor module 50
  51. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Responsibility of a processor module • Take a filtered stream of messages and produce multiple output messages by transforming the payload into multiple dimensions of attributes • For example: – Take a #java2days tweet and parse the other hash tags and output one message per hash tag – #java2days -> (#Java, #SpringBoot, #JavaEE) 51
  52. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Processor module 52 Spring Boot Input Channel Output Channel “… #java2days …" 
 -> 
 #Java, #SpringBoot, #JavaEE… Message Message #java2days
 Tweets Processor Service Channel:
 processor-stream Channel:
 counter-stream
  53. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Our pipeline now looks like: 53 Source Filter Data Tweets #java2days Twitter Stream API Twitter Stream Filter Tweets Filter #Java, #SpringBoot… Process Hash tags …
  54. © 2014 Pivotal Software, Inc. All rights reserved. Adding a

    counter module 54
  55. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Responsibility of a counter module • Take messages from an input channel and output an increment to multiple buckets that count message attributes over time • Save the results to a sink, for example a Redis database • Use Spring Cloud Data Flow admin tool to measure tweets over time 55
  56. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Counter module 56 Spring Boot Input Channel Redis DB Increment counts for hashtags:
 
 #Java -> +1, 
 #SpringBoot -> +1 Message Message Hash tags Counter Service Channel:
 counter-stream #Java: 201 #SpringBoot: 120 #JavaEE: 111 Counter Metrics:
  57. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Our pipeline now looks like: 57 Source Filter Data Tweets #java2days Twitter Stream API Twitter Stream Filter Tweets Filter #Java, 
 #SpringBoot… Process Hash tags Filter #Java: 201
 #SpringBoot: 123 Increment Counter Metrics Redis Tweets Tweets Hash tags
  58. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Scaling our pipeline • Services in the pipeline can be scaled up and down automatically to handle the load and prevent bottlenecks 58
  59. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Auto-scaling 59 Source Process Filter Counter Filter Process 5,121 Filter 52 Filter 23 1 4 2 1 Scale up Scale down Instances: Instances: Instances: Instances:
  60. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Auto-scaling 60 Source Process Filter Counter Filter Process 123 Filter 412 121 1 3 3 1 Scale down Scale up Instances: Instances: Instances: Instances: Process
  61. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Goal of auto-scaling • Keep the data processing pipeline uniform to prevent bottlenecks • Optimize the instance count on cloud providers so that cost can be predicted and optimized 61
  62. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Sinking messages into counters • Each message in the pipeline has an opportunity to increase multiple counters • Counters are like buckets, and we can increment those buckets with a name and timestamp 62
  63. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Twitter Analytics Demo • We’re going to brave the demo gods • Wish me luck • First, let’s review the steps for the demo (in case it fails) 63
  64. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani What we will demo: 64 Source Filter Data Tweets All tweets Twitter Stream API Twitter Stream Filter Tweets Filter #Fun, 
 #Awesome… Process Hash tags Filter #Fun: 201
 #Awesome: 123 Increment Counter Metrics Redis Tweets Tweets Hash tags
  65. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Let’s go through the steps • Start a Redis server • Start the twitter streaming module • Start the filter module • Start the processor module • Start the counter module • Start Spring Cloud Data Flow Admin UI 65
  66. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Start Redis Server 66
  67. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Start Twitter Stream Module 67
  68. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Start the Filter Module 68
  69. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Start the Processor Module 69
  70. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Start the Counter Module 70
  71. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Start Spring Cloud Data Flow Admin UI 71
  72. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Navigate to the Admin UI 72
  73. © 2014 Pivotal Software, Inc. All rights reserved. Follow me

    on Twitter: @kennybastani Thanks! Questions? http://start.spring.io/ 73