Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Виктор Гамов — Всё, что вы хотели знать о In-Memory Data Grids, но боялись спросить

Виктор Гамов — Всё, что вы хотели знать о In-Memory Data Grids, но боялись спросить

In-Memory Data Grids (или кратко, IMDG) исторически были исключительной прерогативой крупных инвестиционных банков и коммерческих продуктов, таких как Oracle Coherence, VMWare Pivotal Gemfire и Software AG Terracotta.

Hazelcast представляет собой альтернативу с открытым исходным кодом. Знакомая многим разработчикам программная модель (наследованная от java.util.* и java.util.concurrent.*) позволила завоевать популярность у разработчиков по всему миру.

IMDG часто используются с базами данных с целью повышения производительности приложений, распределения данных между серверами, кластерами и географическими регионами и управлять очень большими наборами данных или обеспечивать высокие скорости обработки.
Hazelcast предоставляет множество инструментов для организации распределенных вычислений.

После небольшого введения и обзора терминов, на основе примеров кода, Виктор покажет, как использовать Hazelcast для распределенной обработки и хранения данных.

Программа встречи состоит из трех частей:
Часть первая. Введение в распределенные данные в памяти
Часть вторая. Распределенные вычисления
Часть третья. Распределенная система обмена сообщениями

3fc5b5eb32bd3b48d7810fd67b37f9a1?s=128

Moscow JUG

May 04, 2016
Tweet

Transcript

  1. None
  2. • Senior Solutions Architect / Hazelcast • Conferences Speaker •

    Co-author O’Reilly’s «Enterprise Web Development» Viktor Gamov @gamussa
  3. In-memory Data Intro This image cannot currently be displayed.

  4. None
  5. Tiered Storage JVM In-Memory Heap Store Local Off-Heap Store ***

    2,000,000+ 1,000,000 100,000 2 1,000 10,000+ Speed (TPS) 1,000s Size (GB) External Data Source (e.g., Database) Disk Local Disk Store (Re-startable)
  6. Network Database Databases Database Databases Database Databases Database Databases App

    Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap App Server Application JVM Heap J D B C J D B C J D B C Load Balancer Load Balancer Load Balancer Load Balancer J D B C
  7. • JVM memory constraints • Heap size limitations • GC

    pauses and tuning • Handling peak loads • Swapping • CPU limitations • Limited processor power • Many applications sharing CPU cycles • CPU intensive processes
  8. • Reliance on the database • All transactions go through

    the database • Expensive and complex to scale • Reliance on DBA(s) to maintain and tune • Network I/O • Bandwidth limitations • Latency • State-aware cluster • Session replication
  9. Database Databases Database Databases Database Databases Database Databases Load Balancer

    Load Balancer Load Balancer Load Balancer Enterprise Java Dotnet Devices Phones Tablets Web HTML5 Javascript Rich Clients Next Big Trend? Universal Data Transfer Solution
  10. Database Databases Database Databases Database Databases Database Databases Load Balancer

    Load Balancer Load Balancer Load Balancer Enterprise Java Dotnet Devices Phones Tablets Web HTML5 Javascript Rich Clients Next Big Trend? IMDG?
  11. None
  12. 01001 10101 01010 In Memory Data Computing In Memory Data

    Messaging + + In Memory Data Storage
  13. Business Systems A B C RDBMS Mainframe MongoDB NoSQL REST

    Scale IMDG
  14. None
  15. IMDG Nodes IMDG Server JVM [Memory] A B C Business

    Logic Data Data Data CE = Compute Engine Result Business / Processing Logic Result TCP / IP Client Client
  16. IMDG Distributed Topic Bus Topic Node 1 Node 2 Node

    3 MSG Subscribes Delivers Subscribes Delivers
  17. None
  18. CACHES

  19. None
  20. Green Primary Green Backup Green Shard

  21. Partition Groups D A B C Group 1 Group 2

  22. None
  23. Web Sessions (Tomcat/Jetty/Generic) Java C++ .NET Serialization (Serializable, Externalizable, DataSerializable,

    IdentifiedDataSerializable, Portable, Custom) Memcached REST Map Set Queue Executor Service Lock/ Sem. UUID Topic SQL Query Map / Reduce Aggregation Low-Level Services API Node Engine (Threads, Instances, Eventing, Wait/Notify, Invocation) Partition Management (Members, Lite Members, Master Partition, Replicas, Migrations, Partition Groups, Partition Aware) Cluster Management with Cloud Discovery SPI (Multicast, IP List, AWS, jclouds, Kubernetes, Consul) Networking (IPv4, IPv6) On-Heap Store Management Center (JMX/REST) Security Suite (Connection, Encryption, Authentication, Authorization, JAAS LoginModule) WAN Open Source Enterprise Edition Hazelcast 3.6 Architecture Predicate Entry Processor MultiMap High-Density Memory Store Continuous Query Cache Ring Buffer Open Client Network Protocol Hot Restart Store Hibernate 2nd Level Cache JCache java.util.concurrent Hazelcast 3.6 In-Memory Computing Platform Enterprise HD Edition HD Replicated Map Atomic Long HD HD HD HD Enabled Feature HD HD HD HD Near Cache HD
  24. Great for early stages of rapid application development and iteration

    and for OEM. Necessary for scale-up or scale-out deployments – decouples upgrading of clients and cluster for long term TCO Embedded Mode Hazelcast Node 1 Applications Java API Client-Server Mode Hazelcast Node 3 Java API Applications .Net API Applications C++ API Applications Hazelcast Node 2 Hazelcast Node 1 Hazelcast Node 2 Applications Java API Hazelcast Node 3 Applications Java API
  25. • Microsoft SignalR – ASP.NET library for real-time web functionality

    • Azure Cloud Discovery plugin • Hazelcast supported • Open Source • Common Hazelcast Feature Set • Hazelcast supported • Open Source • Hazelcast supported • Open Source • Client and Member APIs • Scala 2.11 support • Hazelcast Supported • Open Source • Community supported • Open Source • Hazelcast supported • Open Source • High-Density Memory Store (Enterprise HD edition) • Near Cache • Hazelcast Supported • Open Source
  26. LET’S SEE SOME CODE

  27. None
  28. ExecutorService

  29. public class MemberRunnable implements Runnable, HazelcastInstanceAware { transient HazelcastInstance hazelcastInstance;

    @Override public void run() { final Member localMember = hazelcastInstance.getCluster().getLocalMember(); System.out.println(localMember.getAddress()); } @Override public void setHazelcastInstance(HazelcastInstance hazelcastInstance) { this.hazelcastInstance = hazelcastInstance; } }
  30. public class MemberRunnable implements Runnable, HazelcastInstanceAware { transient HazelcastInstance hazelcastInstance;

    @Override public void run() { final Member localMember = hazelcastInstance.getCluster().getLocalMember(); System.out.println(localMember.getAddress()); } @Override public void setHazelcastInstance(HazelcastInstance hazelcastInstance) { this.hazelcastInstance = hazelcastInstance; } }
  31. public class MemberRunnable implements Runnable, HazelcastInstanceAware { transient HazelcastInstance hazelcastInstance;

    @Override public void run() { final Member localMember = hazelcastInstance.getCluster().getLocalMember(); System.out.println(localMember.getAddress()); } @Override public void setHazelcastInstance(HazelcastInstance hazelcastInstance) { this.hazelcastInstance = hazelcastInstance; } }
  32. public interface IExecutorService extends ExecutorService, DistributedObject { void execute(Runnable command,

    MemberSelector memberSelector); void executeOnKeyOwner(Runnable command, Object key); void executeOnMember(Runnable command, Member member); void executeOnMembers(Runnable command, Collection<Member> members); void executeOnMembers(Runnable command, MemberSelector memberSelector); void executeOnAllMembers(Runnable command); }
  33. public interface IExecutorService extends ExecutorService, DistributedObject { … <T> Future<T>

    submit(Callable<T> task, MemberSelector memberSelector); <T> Future<T> submitToKeyOwner(Callable<T> task, Object key); <T> Future<T> submitToMember(Callable<T> task, Member member); <T> Map<Member, Future<T>> submitToMembers(Callable<T> task, Collection<Member> members); <T> Map<Member, Future<T>> submitToMembers(Callable<T> task, MemberSelector memberSelector); <T> Map<Member, Future<T>> submitToAllMembers(Callable<T> task); … }
  34. None
  35. Queues, RingBuffers, Topics

  36. http://www.creativebloq.com/3d/how-queue-tooled-bank-robbers-was-created-11410532 QUEUES

  37. IQueue<Integer> queue = hz.getQueue("queue"); for (int i = 1; i

    < 100; i++) { queue.put(i); System.out.println("Producing: " + i); } queue.put(-1); System.out.println("Producer Finished!");
  38. IQueue<Integer> queue = hz.getQueue("queue"); for (int i = 1; i

    < 100; i++) { queue.put(i); System.out.println("Producing: " + i); } queue.put(-1); System.out.println("Producer Finished!");
  39. IQueue<Integer> queue = hz.getQueue("queue"); for (int i = 1; i

    < 100; i++) { queue.put(i); System.out.println("Producing: " + i); } queue.put(-1); System.out.println("Producer Finished!");
  40. IQueue<Integer> queue = hz.getQueue("queue"); while (true) { int item =

    queue.take(); System.out.println("Consumed: " + item); if (item == -1) { queue.put(-1); break; } Thread.sleep(5000); } System.out.println("Consumer Finished!");
  41. IQueue<Integer> queue = hz.getQueue("queue"); while (true) { int item =

    queue.take(); System.out.println("Consumed: " + item); if (item == -1) { queue.put(-1); break; } Thread.sleep(5000); } System.out.println("Consumer Finished!");
  42. IQueue<Integer> queue = hz.getQueue("queue"); while (true) { int item =

    queue.take(); System.out.println("Consumed: " + item); if (item == -1) { queue.put(-1); break; } Thread.sleep(5000); } System.out.println("Consumer Finished!");
  43. RING BUFFER

  44. // retrieve a ring buffer Ringbuffer<Long> rb = hz.getRingbuffer("rb"); long

    sequence = rb.headSequence(); System.out.println("Start reading from: " + sequence); while (true) { long item = rb.readOne(sequence); sequence++; System.out.println("Read: " + item); }
  45. Ringbuffer<Long> rb = hz.getRingbuffer("rb"); // we start from the oldest

    item. long sequence = rb.headSequence(); System.out.println("Start reading from: " + sequence); while (true) { long item = rb.readOne(sequence); sequence++; System.out.println("Read: " + item); }
  46. Ringbuffer<Long> rb = hz.getRingbuffer("rb"); long sequence = rb.headSequence(); System.out.println("Start reading

    from: " + sequence); while (true) { long item = rb.readOne(sequence); sequence++; System.out.println("Read: " + item); }
  47. Ringbuffer<Long> rb = hz.getRingbuffer("rb"); long i = 100; long result

    = rb.addAsync(i, FAIL).get(); long result = rb.addAsync(i, OVERWRITE).get(); import static com.hazelcast.ringbuffer.OverflowPolicy.*;
  48. Ringbuffer<Long> rb = hz.getRingbuffer("rb"); long i = 100; long result

    = rb.addAsync(i, FAIL).get(); long result = rb.addAsync(i, OVERWRITE).get(); import static com.hazelcast.ringbuffer.OverflowPolicy.*;
  49. ITopic<Date> topic = hz.getTopic("topic"); topic.publish(new Date());

  50. ITopic<Date> topic = hz.getTopic("topic"); topic.addMessageListener(message -> System.out.println("Received: " + message.getMessageObject())

    ); System.out.println("Subscribed");
  51. None
  52. http://hazelcast.org/learn

  53. None