Distributed Data Processing with Infinispan and Java Streams

Infinispan is a distributed in-memory key/value data store capable accelerating data processing using Hadoop, Spark and home-grown Map/Reduce APIs. Starting with Infinispan 8, you can now also use the Java 8 Stream API to process, transform and analyse the data stored in the grid, without burdening the architecture with external platforms. Processing can be applied to keys and/or values and it uses Infinispan’s data partitioning logic to distribute operations to nodes where data lives so that they can be executed locally. In this talk you’ll learn about this new extension to Java 8’s Stream class to process data in Infinispan and how it compares with existing APIs.

Transcript

1. Distributed Data Processing with Infinispan and Java Streams Galder Zamarreño Arrizabalaga 11th March 2016
2. Moi • @ • Infinispan co-founder • JSR-107 • Scala developer since 2009 • Functional programming @galderz
3. What is infinispan?
4. Not just nosql key/value data store with optional schema, available under the ASL 2.0 license
5. From local... Store data from slow systems Store data hard to compute
6. ...Via Temporary... Store temporary data in for data that should survive , e.g.
7. ... to data grids Used as primary store for :
8. access mode Application and data live in same JVM
9. access mode Application and data separated by network
10. Clustering • Distribution mode • N copies of data in cluster • Data location defined by Consistent Hash
11. Store & Retrieve • ConcurrentMap • JSR-107 Cache • CDI • SpringCache • Experimental Functional API
12. Compute Extended Java 8 Stream API to data stored in
13. Java 8 stream List<Integer> numbers = Arrays.asList( 4, 74, 20, 97, 118, 50, 97, 34, 48); numbers.stream() .filter(i -> i > 70) // ^ Returns Stream<Integer> .map(n -> new String(Character.toChars(n))) // ^ Returns Stream<String> .reduce("", String::concat); Returns "Java"
14. Lazyness IntStream iterStream = IntStream.iterate(0, i -> i + 1); IntStream.iterate(0, i -> i + 1) .forEach(System.out::println); IntStream.iterate(0, i -> i + 1) .limit(10) // Returns IntStream .forEach(System.out::println); // Returns void Does nothing Prints 1 to 10 Runs forever :(
15. Distributed Streams map(λ) λ λ
16. Demo github.com/galderz/ distributed-streams
17. Topology changes • Streams processed without data loss when topology changes • Retries might happen... • Strive for idempotent lambdas • Idempotent forEach tricky...
18. Special Intermediate operations • distinct → origin + remote • limit → origin + remote • skip/peek → origin only • sorted → origin only Memory ++
19. "The Streams API will internally decompose your query to leverage the multiple cores on your computer." Raoul-Gabriel Urma
20. "Infinispan Distributed Streams API will internally decompose your query to leverage the computing power of multiple machines" Galder Zamarreño
21. Spark/Hadoop integration • Suits Spark/Hadoop users wanting different backend • Need to process data real- time, e.g. sliding windows • Remote access only
22. spark Demo $ docker run -it --name master -h master -e "SLAVES=1" gustavonalle/infinispan-server-domain $ docker run --name spark-master -ti gustavonalle/spark $ docker exec -it spark-master /usr/local/spark/bin/spark- shell --master spark://172.17.0.3:7077 --packages org.infinispan:infinispan-spark_2.10:0.2 --conf spark.io.compression.codec=lz4 Spark Shell Spark Infinispan
23. Spark DEMO link Code for demo: blog.infinispan.org/2015/08/ infinispan-spark-connector-01- released.html
24. which api? • Start → Java Stream API • Spark/Hadoop require management/configuration • Query API helps with deep understanding of values
25. Summary • Infinispan... • is a distributed K/V store • expands Java Streams to run in multi-node environments • offers more options for processing data: Spark/ Hadoop...etc
26. credits engineer by Wilson Joseph from the Noun Project panel by gira Park from the Noun Project Approve by Aha-Soft from the Noun Project Database sharing by YuguDesign from the Noun Project ram by Andrea Rizzato from the Noun Project Database Search by Nimal Raj from the Noun Project Cloud Analytics by Kevin Augustine LO from the Noun Project Broken Computer by Dan Hetteix from the Noun Project data search by Gregor Črešnar from the Noun Project Server by Creative Stall from the Noun Project Network by Creative Stall from the Noun Project transformation by Felipe Perucho from the Noun Project analytics by Roman Kovbasyuk from the Noun Project Server by Designify.me from the Noun Project
27. Thanks http://infinispan.org http://blog.infinispan.org @infinispan @galderz