Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Data Analysis with Kubernetes

Streaming Data Analysis with Kubernetes

Dealing with real-time, in-memory, streaming data is a unique challenge and with the advent of the smartphone and IoT (trillions of internet connected devices), we are witnessing an exponential growth in data at scale.

To be able to handle potential data growth, you want to process data in a cloud environment that can easily scale. In this space, Kubernetes offers great container orchestration and auto-scaling capabilities that are perfectly suited for streaming data use cases. When combined with Infinispan, an in-memory data grid from Red Hat, it empowers you with state of the art distributed data processing capabilities to tackle these challenges.

In this session, we will identify critical patterns and principles that will help you achieve greater scale and response speed. On top of that, you will witness how Infinispan follows these patterns and principles to tackle a Big Data situation via a live coding demonstration on top of a container platform orchestrated by Kubernetes.

Galder Zamarreño

October 19, 2017
Tweet

More Decks by Galder Zamarreño

Other Decks in Programming

Transcript

  1. #BaselOne17

    View Slide

  2. STREAMING DATA ANALYSIS WITH
    KUBERNETES
    Basel One
    Galder Zamarreño Arrizabalaga
    @galderz
    19th October 2017

    View Slide

  3. INSERT DESIGNATOR, IF NEEDED
    ENGINEER
    3
    Since 2006 Community Lead and
    Core Developer
    INFINISPAN
    CO-FOUNDER (2009)
    Since 2013
    Working from home
    LIVE A FEW MINUTES
    AWAY
    MOI

    View Slide

  4. BUILD STREAMING DATA ANALYSIS
    APPLICATION ON TOP OF A
    KUBERNETES-BASED PLATFORM

    View Slide

  5. INSERT DESIGNATOR, IF NEEDED
    THE PROBLEM
    5
    EXPONENTIAL DATA GROWTH
    YEAR ON YEAR
    Smartphones, IOT devices, trillions of internet
    connected devices...
    REAL-TIME STREAMING DATA
    PROCESSING IS CHALLENGING
    Delays can have a big impact

    View Slide

  6. INSERT DESIGNATOR, IF NEEDED
    ZZZ… NO!
    6

    View Slide

  7. INSERT DESIGNATOR, IF NEEDED
    7
    Platform-as-a-Service
    Based on Kubernetes
    Public or Private!
    Multi-language
    THE PLATFORM

    View Slide

  8. INSERT DESIGNATOR, IF NEEDED
    8
    Vert.x is a toolkit for building reactive apps
    On JVM, event-driven and non-blocking
    RxJava integrates with Vert.x
    Great at event transform and coordination
    Works best with many source of events
    (modern apps!)
    THE GLUE

    View Slide

  9. INSERT DESIGNATOR, IF NEEDED
    9
    IN-MEMORY DATA GRIDS
    Analytical
    Framework
    Custom
    Applications
    Mobile
    Applications
    Web Apps &
    Websites
    JBoss
    Middleware
    Fuse "memory" across machines into a unified data store
    Read-through, write-through, write-behind
    • Polyglot
    • Extreme Performance
    • Linear Scalability
    • Fault Tolerant
    • Event driven
    Infinispan

    View Slide

  10. INSERT DESIGNATOR, IF NEEDED
    10
    THE DATA
    http://transport.opendata.ch

    View Slide

  11. INSERT DESIGNATOR, IF NEEDED
    11
    THE DEMO

    View Slide

  12. DEMO

    View Slide

  13. INSERT DESIGNATOR, IF NEEDED
    13
    VERSATILITY OF INFINISPAN
    Expiration
    Listeners
    Eviction
    Embedded | Remote
    Transactions
    Persistence
    Querying
    Code execution
    Security
    Management and monitoring Cloud Integrations
    Distributed Cache Shared Memory Event Broker Analytics

    View Slide

  14. BUILD STREAMING DATA ANALYSIS
    APPLICATION ON TOP OF A
    KUBERNETES-BASED PLATFORM

    View Slide

  15. github.com/infinispan-demos/streaming-data-kubernetes
    infinispan.org
    redhat.com/en/technologies/jboss-middleware/data-grid
    openshift.com
    vertx.io
    THANK YOU

    View Slide