Виктор Гамов — Всё, что вы хотели знать о In-Memory Data Grids, но боялись спросить

In-Memory Data Grids (или кратко, IMDG) исторически были исключительной прерогативой крупных инвестиционных банков и коммерческих продуктов, таких как Oracle Coherence, VMWare Pivotal Gemfire и Software AG Terracotta.

Hazelcast представляет собой альтернативу с открытым исходным кодом. Знакомая многим разработчикам программная модель (наследованная от java.util.* и java.util.concurrent.*) позволила завоевать популярность у разработчиков по всему миру.

IMDG часто используются с базами данных с целью повышения производительности приложений, распределения данных между серверами, кластерами и географическими регионами и управлять очень большими наборами данных или обеспечивать высокие скорости обработки.
Hazelcast предоставляет множество инструментов для организации распределенных вычислений.

После небольшого введения и обзора терминов, на основе примеров кода, Виктор покажет, как использовать Hazelcast для распределенной обработки и хранения данных.

Программа встречи состоит из трех частей:
Часть первая. Введение в распределенные данные в памяти
Часть вторая. Распределенные вычисления
Часть третья. Распределенная система обмена сообщениями

Transcript

2. • Senior Solutions Architect / Hazelcast • Conferences Speaker • Co-author O’Reilly’s «Enterprise Web Development» Viktor Gamov @gamussa
3. In-memory Data Intro This image cannot currently be displayed.
5. Tiered Storage JVM In-Memory Heap Store Local Off-Heap Store *** 2,000,000+ 1,000,000 100,000 2 1,000 10,000+ Speed (TPS) 1,000s Size (GB) External Data Source (e.g., Database) Disk Local Disk Store (Re-startable)
6. Network Database Databases Database Databases Database Databases Database Databases App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap J D B C J D B C J D B C Load Balancer Load Balancer Load Balancer Load Balancer J D B C
7. • JVM memory constraints • Heap size limitations • GC pauses and tuning • Handling peak loads • Swapping • CPU limitations • Limited processor power • Many applications sharing CPU cycles • CPU intensive processes
8. • Reliance on the database • All transactions go through the database • Expensive and complex to scale • Reliance on DBA(s) to maintain and tune • Network I/O • Bandwidth limitations • Latency • State-aware cluster • Session replication
9. Database Databases Database Databases Database Databases Database Databases Load Balancer Load Balancer Load Balancer Load Balancer Enterprise Java Dotnet Devices Phones Tablets Web HTML5 Javascript Rich Clients Next Big Trend? Universal Data Transfer Solution
10. Database Databases Database Databases Database Databases Database Databases Load Balancer Load Balancer Load Balancer Load Balancer Enterprise Java Dotnet Devices Phones Tablets Web HTML5 Javascript Rich Clients Next Big Trend? IMDG?
12. 01001 10101 01010 In Memory Data Computing In Memory Data Messaging + + In Memory Data Storage
13. Business Systems A B C RDBMS Mainframe MongoDB NoSQL REST Scale IMDG
15. IMDG Nodes IMDG Server JVM [Memory] A B C Business Logic Data Data Data CE = Compute Engine Result Business / Processing Logic Result TCP / IP Client Client
16. IMDG Distributed Topic Bus Topic Node 1 Node 2 Node 3 MSG Subscribes Delivers Subscribes Delivers
18. CACHES
20. Green Primary Green Backup Green Shard
21. Partition Groups D A B C Group 1 Group 2
23. Web Sessions (Tomcat/Jetty/Generic) Java C++ .NET Serialization (Serializable, Externalizable, DataSerializable, IdentifiedDataSerializable, Portable, Custom) Memcached REST Map Set Queue Executor Service Lock/ Sem. UUID Topic SQL Query Map / Reduce Aggregation Low-Level Services API Node Engine (Threads, Instances, Eventing, Wait/Notify, Invocation) Partition Management (Members, Lite Members, Master Partition, Replicas, Migrations, Partition Groups, Partition Aware) Cluster Management with Cloud Discovery SPI (Multicast, IP List, AWS, jclouds, Kubernetes, Consul) Networking (IPv4, IPv6) On-Heap Store Management Center (JMX/REST) Security Suite (Connection, Encryption, Authentication, Authorization, JAAS LoginModule) WAN Open Source Enterprise Edition Hazelcast 3.6 Architecture Predicate Entry Processor MultiMap High-Density Memory Store Continuous Query Cache Ring Buffer Open Client Network Protocol Hot Restart Store Hibernate 2nd Level Cache JCache java.util.concurrent Hazelcast 3.6 In-Memory Computing Platform Enterprise HD Edition HD Replicated Map Atomic Long HD HD HD HD Enabled Feature HD HD HD HD Near Cache HD
24. Great for early stages of rapid application development and iteration and for OEM. Necessary for scale-up or scale-out deployments – decouples upgrading of clients and cluster for long term TCO Embedded Mode Hazelcast Node 1 Applications Java API Client-Server Mode Hazelcast Node 3 Java API Applications .Net API Applications C++ API Applications Hazelcast Node 2 Hazelcast Node 1 Hazelcast Node 2 Applications Java API Hazelcast Node 3 Applications Java API
25. • Microsoft SignalR – ASP.NET library for real-time web functionality • Azure Cloud Discovery plugin • Hazelcast supported • Open Source • Common Hazelcast Feature Set • Hazelcast supported • Open Source • Hazelcast supported • Open Source • Client and Member APIs • Scala 2.11 support • Hazelcast Supported • Open Source • Community supported • Open Source • Hazelcast supported • Open Source • High-Density Memory Store (Enterprise HD edition) • Near Cache • Hazelcast Supported • Open Source
26. LET’S SEE SOME CODE
28. ExecutorService
29. public class MemberRunnable implements Runnable, HazelcastInstanceAware { transient HazelcastInstance hazelcastInstance; @Override public void run() { final Member localMember = hazelcastInstance.getCluster().getLocalMember(); System.out.println(localMember.getAddress()); } @Override public void setHazelcastInstance(HazelcastInstance hazelcastInstance) { this.hazelcastInstance = hazelcastInstance; } }
30. public class MemberRunnable implements Runnable, HazelcastInstanceAware { transient HazelcastInstance hazelcastInstance; @Override public void run() { final Member localMember = hazelcastInstance.getCluster().getLocalMember(); System.out.println(localMember.getAddress()); } @Override public void setHazelcastInstance(HazelcastInstance hazelcastInstance) { this.hazelcastInstance = hazelcastInstance; } }
31. public class MemberRunnable implements Runnable, HazelcastInstanceAware { transient HazelcastInstance hazelcastInstance; @Override public void run() { final Member localMember = hazelcastInstance.getCluster().getLocalMember(); System.out.println(localMember.getAddress()); } @Override public void setHazelcastInstance(HazelcastInstance hazelcastInstance) { this.hazelcastInstance = hazelcastInstance; } }
32. public interface IExecutorService extends ExecutorService, DistributedObject { void execute(Runnable command, MemberSelector memberSelector); void executeOnKeyOwner(Runnable command, Object key); void executeOnMember(Runnable command, Member member); void executeOnMembers(Runnable command, Collection<Member> members); void executeOnMembers(Runnable command, MemberSelector memberSelector); void executeOnAllMembers(Runnable command); }
33. public interface IExecutorService extends ExecutorService, DistributedObject { … <T> Future<T> submit(Callable<T> task, MemberSelector memberSelector); <T> Future<T> submitToKeyOwner(Callable<T> task, Object key); <T> Future<T> submitToMember(Callable<T> task, Member member); <T> Map<Member, Future<T>> submitToMembers(Callable<T> task, Collection<Member> members); <T> Map<Member, Future<T>> submitToMembers(Callable<T> task, MemberSelector memberSelector); <T> Map<Member, Future<T>> submitToAllMembers(Callable<T> task); … }
35. Queues, RingBuffers, Topics
36. http://www.creativebloq.com/3d/how-queue-tooled-bank-robbers-was-created-11410532 QUEUES
37. IQueue<Integer> queue = hz.getQueue("queue"); for (int i = 1; i < 100; i++) { queue.put(i); System.out.println("Producing: " + i); } queue.put(-1); System.out.println("Producer Finished!");
38. IQueue<Integer> queue = hz.getQueue("queue"); for (int i = 1; i < 100; i++) { queue.put(i); System.out.println("Producing: " + i); } queue.put(-1); System.out.println("Producer Finished!");
39. IQueue<Integer> queue = hz.getQueue("queue"); for (int i = 1; i < 100; i++) { queue.put(i); System.out.println("Producing: " + i); } queue.put(-1); System.out.println("Producer Finished!");
40. IQueue<Integer> queue = hz.getQueue("queue"); while (true) { int item = queue.take(); System.out.println("Consumed: " + item); if (item == -1) { queue.put(-1); break; } Thread.sleep(5000); } System.out.println("Consumer Finished!");
41. IQueue<Integer> queue = hz.getQueue("queue"); while (true) { int item = queue.take(); System.out.println("Consumed: " + item); if (item == -1) { queue.put(-1); break; } Thread.sleep(5000); } System.out.println("Consumer Finished!");
42. IQueue<Integer> queue = hz.getQueue("queue"); while (true) { int item = queue.take(); System.out.println("Consumed: " + item); if (item == -1) { queue.put(-1); break; } Thread.sleep(5000); } System.out.println("Consumer Finished!");
43. RING BUFFER
44. // retrieve a ring buffer Ringbuffer<Long> rb = hz.getRingbuffer("rb"); long sequence = rb.headSequence(); System.out.println("Start reading from: " + sequence); while (true) { long item = rb.readOne(sequence); sequence++; System.out.println("Read: " + item); }
45. Ringbuffer<Long> rb = hz.getRingbuffer("rb"); // we start from the oldest item. long sequence = rb.headSequence(); System.out.println("Start reading from: " + sequence); while (true) { long item = rb.readOne(sequence); sequence++; System.out.println("Read: " + item); }
46. Ringbuffer<Long> rb = hz.getRingbuffer("rb"); long sequence = rb.headSequence(); System.out.println("Start reading from: " + sequence); while (true) { long item = rb.readOne(sequence); sequence++; System.out.println("Read: " + item); }
47. Ringbuffer<Long> rb = hz.getRingbuffer("rb"); long i = 100; long result = rb.addAsync(i, FAIL).get(); long result = rb.addAsync(i, OVERWRITE).get(); import static com.hazelcast.ringbuffer.OverflowPolicy.*;
48. Ringbuffer<Long> rb = hz.getRingbuffer("rb"); long i = 100; long result = rb.addAsync(i, FAIL).get(); long result = rb.addAsync(i, OVERWRITE).get(); import static com.hazelcast.ringbuffer.OverflowPolicy.*;
49. ITopic<Date> topic = hz.getTopic("topic"); topic.publish(new Date());
50. ITopic<Date> topic = hz.getTopic("topic"); topic.addMessageListener(message -> System.out.println("Received: " + message.getMessageObject()) ); System.out.println("Subscribed");
52. http://hazelcast.org/learn