Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Все что вы хотели знать о In-Memory Data Grids, но боялись спросить - Moscow JUG

Все что вы хотели знать о In-Memory Data Grids, но боялись спросить - Moscow JUG

In-Memory Data Grids (или кратко, IMDG) исторически были исключительной прерогативой крупных инвестиционных банков и коммерческих продуктов, таких как Oracle Coherence, VMWare Pivotal Gemfire и Software AG Terracotta.

Hazelcast представляет собой альтернативу с открытым исходным кодом. Знакомая многим разработчикам программная модель (наследованная от java.util.* и java.util.concurrent.*) позволила завоевать популярность у разработчиков по всему миру.

IMDG часто используются с базами данных с целью повышения производительности приложений, распределения данных между серверами, кластерами и географическими регионами и управлять очень большими наборами данных или обеспечивать высокие скорости обработки.
Hazelcast предоставляет множество инструментов для организации распределенных вычислений.

После небольшого введения и обзора терминов, на основе примеров кода, Виктор покажет как использовать Hazelcast для распределенной обработки и хранения данных.

Программа встречи состоит из трех частей:

Часть первая. Введение в распределенные данные в памяти

Часть вторая. Распределенные вычисления

Часть третья. Распределенная система обмена сообщениями

Viktor Gamov

May 04, 2016
Tweet

More Decks by Viktor Gamov

Other Decks in Technology

Transcript

  1. • Senior Solutions Architect / Hazelcast
    • Conferences Speaker
    • Co-author O’Reilly’s
    «Enterprise Web Development»
    Viktor Gamov
    @gamussa

    View full-size slide

  2. In-memory Data Intro
    This image cannot currently be displayed.

    View full-size slide

  3. Tiered Storage
    JVM
    In-Memory
    Heap
    Store
    Local
    Off-Heap Store
    ***
    2,000,000+
    1,000,000
    100,000
    2
    1,000
    10,000+
    Speed (TPS)
    1,000s
    Size (GB)
    External Data
    Source
    (e.g., Database)
    Disk
    Local Disk
    Store
    (Re-startable)

    View full-size slide

  4. Network
    Database
    Databases
    Database
    Databases
    Database
    Databases
    Database
    Databases
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    App Server
    Application
    JVM Heap
    J
    D
    B
    C
    J
    D
    B
    C
    J
    D
    B
    C
    Load Balancer Load Balancer Load Balancer Load Balancer
    J
    D
    B
    C

    View full-size slide

  5. • JVM memory constraints
    • Heap size limitations
    • GC pauses and tuning
    • Handling peak loads
    • Swapping
    • CPU limitations
    • Limited processor power
    • Many applications sharing CPU cycles
    • CPU intensive processes

    View full-size slide

  6. • Reliance on the database
    • All transactions go through the database
    • Expensive and complex to scale
    • Reliance on DBA(s) to maintain and tune
    • Network I/O
    • Bandwidth limitations
    • Latency
    • State-aware cluster
    • Session replication

    View full-size slide

  7. Database
    Databases Database
    Databases Database
    Databases Database
    Databases
    Load Balancer Load Balancer Load Balancer Load Balancer
    Enterprise
    Java
    Dotnet
    Devices
    Phones
    Tablets
    Web
    HTML5
    Javascript
    Rich Clients
    Next Big
    Trend?
    Universal Data Transfer Solution

    View full-size slide

  8. Database
    Databases Database
    Databases Database
    Databases Database
    Databases
    Load Balancer Load Balancer Load Balancer Load Balancer
    Enterprise
    Java
    Dotnet
    Devices
    Phones
    Tablets
    Web
    HTML5
    Javascript
    Rich Clients
    Next Big
    Trend?
    IMDG?

    View full-size slide

  9. 01001
    10101
    01010
    In Memory
    Data Computing
    In Memory
    Data Messaging
    +
    +
    In Memory
    Data Storage

    View full-size slide

  10. Business Systems
    A B C
    RDBMS Mainframe
    MongoDB
    NoSQL
    REST
    Scale
    IMDG

    View full-size slide

  11. IMDG Nodes
    IMDG Server
    JVM [Memory]
    A B C
    Business Logic
    Data Data Data
    CE = Compute Engine
    Result
    Business / Processing Logic
    Result
    TCP / IP
    Client Client

    View full-size slide

  12. IMDG Distributed
    Topic Bus
    Topic
    Node 1 Node 2
    Node 3
    MSG
    Subscribes
    Delivers
    Subscribes
    Delivers

    View full-size slide

  13. Green
    Primary
    Green
    Backup
    Green
    Shard

    View full-size slide

  14. Partition Groups
    D
    A B C
    Group 1 Group 2

    View full-size slide

  15. Web Sessions
    (Tomcat/Jetty/Generic)
    Java
    C++ .NET
    Serialization
    (Serializable, Externalizable, DataSerializable, IdentifiedDataSerializable, Portable, Custom)
    Memcached REST
    Map Set Queue
    Executor Service
    Lock/
    Sem. UUID Topic
    SQL Query Map / Reduce Aggregation
    Low-Level Services API
    Node Engine
    (Threads, Instances, Eventing, Wait/Notify, Invocation)
    Partition Management
    (Members, Lite Members, Master Partition, Replicas, Migrations, Partition Groups, Partition Aware)
    Cluster Management with Cloud Discovery SPI
    (Multicast, IP List, AWS, jclouds, Kubernetes, Consul)
    Networking
    (IPv4, IPv6)
    On-Heap Store
    Management Center
    (JMX/REST)
    Security Suite
    (Connection, Encryption, Authentication, Authorization, JAAS LoginModule)
    WAN
    Open Source Enterprise Edition
    Hazelcast 3.6 Architecture
    Predicate Entry Processor
    MultiMap
    High-Density Memory Store
    Continuous
    Query Cache
    Ring
    Buffer
    Open Client Network Protocol
    Hot Restart Store
    Hibernate 2nd Level
    Cache JCache java.util.concurrent
    Hazelcast 3.6 In-Memory Computing Platform
    Enterprise HD Edition
    HD
    Replicated
    Map
    Atomic
    Long
    HD HD
    HD HD Enabled Feature
    HD HD
    HD
    HD
    Near Cache HD

    View full-size slide

  16. Great for early stages of rapid
    application development and iteration
    and for OEM.
    Necessary for scale-up or scale-out
    deployments
    – decouples upgrading of clients
    and cluster for long term TCO
    Embedded Mode
    Hazelcast
    Node 1
    Applications
    Java API
    Client-Server Mode
    Hazelcast
    Node 3
    Java API
    Applications
    .Net API
    Applications
    C++ API
    Applications
    Hazelcast
    Node 2
    Hazelcast
    Node 1
    Hazelcast
    Node 2
    Applications
    Java API
    Hazelcast
    Node 3
    Applications
    Java API

    View full-size slide

  17. • Microsoft SignalR – ASP.NET library for
    real-time web functionality
    • Azure Cloud Discovery plugin
    • Hazelcast supported
    • Open Source
    • Common Hazelcast Feature Set
    • Hazelcast supported
    • Open Source
    • Hazelcast supported
    • Open Source
    • Client and Member APIs
    • Scala 2.11 support
    • Hazelcast Supported
    • Open Source
    • Community supported
    • Open Source
    • Hazelcast supported
    • Open Source
    • High-Density Memory Store (Enterprise HD edition)
    • Near Cache
    • Hazelcast Supported
    • Open Source

    View full-size slide

  18. LET’S SEE SOME CODE

    View full-size slide

  19. ExecutorService

    View full-size slide

  20. public class MemberRunnable
    implements Runnable, HazelcastInstanceAware {
    transient HazelcastInstance hazelcastInstance;
    @Override public void run() {
    final Member localMember =
    hazelcastInstance.getCluster().getLocalMember();
    System.out.println(localMember.getAddress());
    }
    @Override public void
    setHazelcastInstance(HazelcastInstance hazelcastInstance) {
    this.hazelcastInstance = hazelcastInstance;
    }
    }

    View full-size slide

  21. public class MemberRunnable
    implements Runnable, HazelcastInstanceAware {
    transient HazelcastInstance hazelcastInstance;
    @Override public void run() {
    final Member localMember =
    hazelcastInstance.getCluster().getLocalMember();
    System.out.println(localMember.getAddress());
    }
    @Override public void
    setHazelcastInstance(HazelcastInstance hazelcastInstance) {
    this.hazelcastInstance = hazelcastInstance;
    }
    }

    View full-size slide

  22. public class MemberRunnable
    implements Runnable, HazelcastInstanceAware {
    transient HazelcastInstance hazelcastInstance;
    @Override public void run() {
    final Member localMember =
    hazelcastInstance.getCluster().getLocalMember();
    System.out.println(localMember.getAddress());
    }
    @Override public void
    setHazelcastInstance(HazelcastInstance hazelcastInstance) {
    this.hazelcastInstance = hazelcastInstance;
    }
    }

    View full-size slide

  23. public interface IExecutorService extends ExecutorService, DistributedObject {
    void execute(Runnable command, MemberSelector memberSelector);
    void executeOnKeyOwner(Runnable command, Object key);
    void executeOnMember(Runnable command, Member member);
    void executeOnMembers(Runnable command, Collection members);
    void executeOnMembers(Runnable command, MemberSelector memberSelector);
    void executeOnAllMembers(Runnable command);
    }

    View full-size slide

  24. public interface IExecutorService extends ExecutorService, DistributedObject {

    Future submit(Callable task, MemberSelector memberSelector);
    Future submitToKeyOwner(Callable task, Object key);
    Future submitToMember(Callable task, Member member);
    Map> submitToMembers(Callable task,
    Collection members);
    Map> submitToMembers(Callable task, MemberSelector
    memberSelector);
    Map> submitToAllMembers(Callable task);

    }

    View full-size slide

  25. Queues, RingBuffers, Topics

    View full-size slide

  26. http://www.creativebloq.com/3d/how-queue-tooled-bank-robbers-was-created-11410532
    QUEUES

    View full-size slide

  27. IQueue queue = hz.getQueue("queue");
    for (int i = 1; i < 100; i++) {
    queue.put(i);
    System.out.println("Producing: " + i);
    }
    queue.put(-1);
    System.out.println("Producer Finished!");

    View full-size slide

  28. IQueue queue = hz.getQueue("queue");
    for (int i = 1; i < 100; i++) {
    queue.put(i);
    System.out.println("Producing: " + i);
    }
    queue.put(-1);
    System.out.println("Producer Finished!");

    View full-size slide

  29. IQueue queue = hz.getQueue("queue");
    for (int i = 1; i < 100; i++) {
    queue.put(i);
    System.out.println("Producing: " + i);
    }
    queue.put(-1);
    System.out.println("Producer Finished!");

    View full-size slide

  30. IQueue queue = hz.getQueue("queue");
    while (true) {
    int item = queue.take();
    System.out.println("Consumed: " + item);
    if (item == -1) {
    queue.put(-1);
    break;
    }
    Thread.sleep(5000);
    }
    System.out.println("Consumer Finished!");

    View full-size slide

  31. IQueue queue = hz.getQueue("queue");
    while (true) {
    int item = queue.take();
    System.out.println("Consumed: " + item);
    if (item == -1) {
    queue.put(-1);
    break;
    }
    Thread.sleep(5000);
    }
    System.out.println("Consumer Finished!");

    View full-size slide

  32. IQueue queue = hz.getQueue("queue");
    while (true) {
    int item = queue.take();
    System.out.println("Consumed: " + item);
    if (item == -1) {
    queue.put(-1);
    break;
    }
    Thread.sleep(5000);
    }
    System.out.println("Consumer Finished!");

    View full-size slide

  33. // retrieve a ring buffer
    Ringbuffer rb = hz.getRingbuffer("rb");
    long sequence = rb.headSequence();
    System.out.println("Start reading from: " +
    sequence);
    while (true) {
    long item = rb.readOne(sequence);
    sequence++;
    System.out.println("Read: " + item);
    }

    View full-size slide

  34. Ringbuffer rb = hz.getRingbuffer("rb");
    // we start from the oldest item.
    long sequence = rb.headSequence();
    System.out.println("Start reading from: " +
    sequence);
    while (true) {
    long item = rb.readOne(sequence);
    sequence++;
    System.out.println("Read: " + item);
    }

    View full-size slide

  35. Ringbuffer rb = hz.getRingbuffer("rb");
    long sequence = rb.headSequence();
    System.out.println("Start reading from: " +
    sequence);
    while (true) {
    long item = rb.readOne(sequence);
    sequence++;
    System.out.println("Read: " + item);
    }

    View full-size slide

  36. Ringbuffer rb = hz.getRingbuffer("rb");
    long i = 100;
    long result = rb.addAsync(i, FAIL).get();
    long result = rb.addAsync(i, OVERWRITE).get();
    import static
    com.hazelcast.ringbuffer.OverflowPolicy.*;

    View full-size slide

  37. Ringbuffer rb = hz.getRingbuffer("rb");
    long i = 100;
    long result = rb.addAsync(i, FAIL).get();
    long result = rb.addAsync(i, OVERWRITE).get();
    import static
    com.hazelcast.ringbuffer.OverflowPolicy.*;

    View full-size slide

  38. ITopic topic = hz.getTopic("topic");
    topic.publish(new Date());

    View full-size slide

  39. ITopic topic = hz.getTopic("topic");
    topic.addMessageListener(message ->
    System.out.println("Received: " +
    message.getMessageObject())
    );
    System.out.println("Subscribed");

    View full-size slide

  40. http://hazelcast.org/learn

    View full-size slide