Coherence SIG: Database backed Coherence cache

Transcript

1. Database backed Coherence cache
   Tips, Tricks and Patterns
   Alexey Ragozin
   alexey.ragozin@gmail.com
   May 2012
   

   View Slide

2. Power of read-write-backing-map
   • Fetching data as needed
   • Separation of concerns
   • Gracefully handling concurrency
   • Write-behind – removing DB from critical path
   • Database operation bundling
   

   View Slide

3. … and challenges
    DB operations are order of magnitude slower
   • Less deterministic response time
   • Coherence thread pools issues
    How verify persistence with write behind?
    Data are written in DB in random order
    read-write-backing-map and expiry
   

   View Slide

4. TIPS
   

   View Slide

5. BinaryEntryStore, did you know?
   BinaryEntryStore – an alternative to
   CacheLoader / CacheStore interface.
   Works with BinaryEntry instead of objects.
    You can access binary key and value
   • Skip deserialization, if binary is enough
    You can access previous version of value
   • Distinguish inserts vs. updates
   • Find which fields were cached
    You cannot set entry TTL in cache loader 
   

   View Slide

6. When storeAll(…) is called?
    cache.getAll(…)
   • loadAll(…) will be called with partition granularity
   (since Coherence 3.7)
    cache.putAll(…)
   • write-behind scheme will use storeAll(…)
   • write-through scheme will use store(…)
   (this could be really slow)
   

   View Slide

7. When storeAll(…) is called?
    cache.invokeAll(…)/aggregate(…)
   • calling get() on entry will invoke load(…)
   (if entry is not cached yet)
   • calling set() on entry will invoke put(…)
   (in case of write-through)
   • you can check entry.isPresent() to avoid needless
   read-through
   • Coherence will never use bulk cache store
   operations for aggregators and entry processors
   

   View Slide

8. Warming up aggregator
   public static void preloadValuesViaReadThrough(Set entries) {
   CacheMap backingMap = null;
   Set keys = new HashSet();
   for (BinaryEntry entry : entries) {
   if (backingMap == null) {
   backingMap = (CacheMap) entry.getBackingMapContext().getBackingMap();
   }
   if (!entry.isPresent()) {
   keys.add(entry.getBinaryKey());
   }
   }
   backingMap.getAll(keys);
   }
   Code above will force all entries for working set to be
   preloaded using bulk loadAll(…).
   Call it before processing entries.
   

   View Slide

9. Why load(…) is called on write?
   Case:
   • Entry processor is called on set of entries which is
   not in cache and assigns values to them
   Question:
   • Why read-through is triggered?
   Answer:
   • BinaryEntry.setValue(Object) returns old value
   • Use BinaryEntry.setValue(Object, boolean)
   

   View Slide

10. Bulk put with write through
    You can use same trick for updates.
    1. Pack your values in entry processor.
    2. In entry processor obtain backing map reference.
    3. Call putAll(…) on backing map.
    Be careful !!!
    • You should only put key for partition entry processor
    was called for.
    • Backing map accepts serialized objects.
    

    View Slide

11. Using operation bundling
    Worker
    thread1
    Worker
    thread2
    Req 1
    Req 2
    Worker
    thread3
    storeAll()
    RWBM
    Cache
    Store
    Waiting
    Req 3
    Worker thread 3
    Req 1
    Req 2
    Req 3
    Worker
    thread1
    Worker
    thread2
    Worker
    thread3
    RWBM
    Cache
    Store
    

    View Slide

12. Using operation bundling
    storeAll(…) with N keys could be called if
     You have at least N concurrent operations
     You have at least N threads in worker pool
    
    
    
    store
    5
    4
    
    
    
    

    View Slide

13. Checking STORE decoration
     Configure cache as “write-behind”
     Put data
     Wait until, STORE decoration become TRUE
    (actually it will switch from FALSE to null)
    public class StoreFlagExtractor extends AbstractExtractor implements PortableObject {
    // ...
    private Object extractInternal(Binary binValue, BinaryEntry entry) {
    if (ExternalizableHelper.isDecorated(binValue)) {
    Binary store = ExternalizableHelper.getDecoration(binValue, ExternalizableHelper.DECO_STORE);
    if (store != null) {
    Object st = ExternalizableHelper.fromBinary(store, entry.getSerializer());
    return st;
    }
    }
    return Boolean.TRUE;
    }
    }
    

    View Slide

14. BEHIND SCENES
    

    View Slide

15. How it works?
    Distributed cache service
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    

    View Slide

16. How it works?
    Distributed cache service
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distribution and backup of data
    Storing cache data, expiry
    Interacting with
    persistent storage
    Caching cache loader misses
    Coordination
    

    View Slide

17. How it works?
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    get(key)
    Distributed cache service
    #1
    Cache service
    is receiving
    get(…)
    request.
    

    View Slide

18. How it works?
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    get(key)
    Distributed cache service
    get(key)
    #2
    Cache service
    is invoking
    get(…) on
    backing map.
    Partition
    transaction is
    open.
    

    View Slide

19. How it works?
    #3
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    get(key)
    get(key)
    Backing map
    checks internal
    map and miss
    cache if present.
    Key is not found.
    

    View Slide

20. How it works?
    #4
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    load(key)
    Distributed cache service
    get(key)
    get(key)
    Backing map is
    invoking
    load(…) on
    cache loader.
    

    View Slide

21. How it works?
    #5
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    query external data source
    get(key)
    get(key)
    load(key)
    Cache loader
    is retrieving
    value for
    external
    source
    

    View Slide

22. How it works?
    #6
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    get(key)
    get(key)
    load(key)
    query external data source
    Value is loaded
    

    View Slide

23. How it works?
    #7
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    get(key)
    get(key)
    load(key)
    query external data source
    Backing map is
    updating
    internal map
    

    View Slide

24. How it works?
    #8
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    map event
    Distributed cache service
    get(key)
    get(key)
    load(key)
    query external data source
    Internal map is
    observable and
    cache service is
    receiving event
    about new entry in
    internal map.
    

    View Slide

25. How it works?
    #9
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    map event
    Distributed cache service
    get(key)
    get(key)
    load(key)
    query external data source
    Call to backing
    map returns.
    Cache service
    is ready to
    commit
    partition
    transaction.
    

    View Slide

26. How it works?
    #10
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    update backup
    Distributed cache service
    get(key)
    get(key)
    load(key)
    map event
    query external data source
    Partition
    transaction is
    being
    committed.
    New value is
    being sent to
    backup node.
    

    View Slide

27. How it works?
    #11
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    get(key)
    get(key)
    load(key)
    map event
    update backup
    query external data source
    Response for
    get(…)
    request is sent
    back as backup
    has confirmed
    update.
    

    View Slide

28. How it works?
    #1
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    Cache service
    is receiving
    put(…)
    requiest.
    

    View Slide

29. How it works?
    #2
    put(k,v)
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    Cache service
    is invoking
    put(…) on
    backing map.
    Partition
    transaction is
    open.
    

    View Slide

30. How it works?
    #3
    put(k,v)
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    Value is
    immediately
    stored in
    internal map
    and put to
    write-behind
    queue.
    

    View Slide

31. How it works?
    #4
    put(k,v)
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    map event
    DECO_STORE=false
    Cache service is
    receiving event, but
    backing map is
    decorating value with
    DECO_STORE=false
    flag to mark that value
    is yet-to-stored.
    

    View Slide

32. How it works?
    #5
    put(k,v)
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    map event
    DECO_STORE=false
    Call to backing
    map return.
    

    View Slide

33. How it works?
    #6
    put(k,v)
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    update backup
    map event
    DECO_STORE=false
    Partition transaction
    is being committed.
    Backup will receive
    value decorated with
    DECO_STORE=false.
    

    View Slide

34. How it works?
    #7
    put(k,v)
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    map event
    DECO_STORE=false
    update backup
    Cache service is
    sending response
    back as soon as
    backup is
    confirmed.
    

    View Slide

35. How it works?
    #8
    put(k,v)
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    Eventually, cache
    store is called to
    persist value.
    It is done on
    separate thread.
    

    View Slide

36. How it works?
    #9
    put(k,v)
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    store to external storage
    Value is stored
    in external
    storage by
    cache store.
    

    View Slide

37. How it works?
    #10
    store to external storage
    put(k,v)
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    map event
    DECO_STORE=null
    Once call to cache
    store has returned
    successfully. Backing
    map is removing
    DECO_STORE
    decoration from value
    is internal map.
    Cache service is
    receiving map event
    

    View Slide

38. How it works?
    #11
    store to external storage
    put(k,v)
    put(k,v)
    read-write
    backing-map
    Cache store
    Internal map
    Miss cache
    Distributed cache service
    update backup
    map event
    DECO_STORE=null
    Map event was
    received by cache
    service outside of
    service thread. It will
    be put to OOB queue
    and eventually
    processed.
    Update to backup will
    be sent once event is
    processed.
    

    View Slide

39. THREADING
    

    View Slide

40. Requests and jobs
    putAll(…)
    Client side:
    One method call
    

    View Slide

41. Requests and jobs
    putAll(…)
    PutRequest
    PutRequest
    PutRequest
    Network:
    One request for
    each member
    

    View Slide

42. Requests and jobs
    putAll(…)
    PutRequest
    PutRequest
    PutRequest
    Job
    Job
    Job
    Storage node:
    One job per partition
    

    View Slide

43. Requests and jobs
    Problem
     Single API call may produce hundreds of jobs
    for worker threads in cluster (limited by
    partition count).
     Write-through and read-through jobs could be
    time consuming.
     While all threads are busy by time consuming
    jobs, cache is unresponsive.
    

    View Slide

44. Requests and jobs
    Workarounds
     Huge thread pools
     Request throttling
     By member (one network request at time)
     By partitions (one job at time)
     Priorities
     Applicable only to EP and aggregators
    

    View Slide

45. “UNBREAKABLE CACHE” PATTERN
    

    View Slide

46. “Canary” keys
     Canary keys – special keys (one per partitions)
    ignored by all cache operations.
     Canary key is inserted once “recovery”
    procedure have verified that partition data is
    complete.
     If partition is not yet loaded or lost due to
    disaster, canary key will be missing.
    

    View Slide

47. Recovery procedure
     Store object hash code in database
     Using hash you can query database for all keys
    belonging to partition
     Knowing all keys, can use read-through to pull
    data to a cache
     Cache is writable during recovery!
     Coherence internal concurrency control will
    ensure consistency
    

    View Slide

48. “Unbreakable cache”
    read/write-trough + canary keys + recovery
     Key based operations rely on read-through
     Filter based operations are checking “canary”
    keys (and activate recovery is needed)
     Preloading = recovery
     Cache is writable at all times
    

    View Slide

49. Checking “canary” keys
     Option 1
     check “canary” keys
     perform query
     Option 2
     perform query
     check “canary” keys
    

    View Slide

50. Checking “canary” keys
     Option 1
     check “canary” keys
     perform query
     Option 2
     perform query
     check “canary” keys
     Right way
     check “canaries” inside of query!
    

    View Slide

51. “Unbreakable cache”
    Motivation
     Incomplete data set would invalidate hundred of hours of
    number crunching
     100% complete data or exception
     Persistent DB is requirement anyway
    Summary
     Transparent recovery (+ preloading for free)
     Always writable (i.e. feeds are not waiting for recovery)
     Graceful degradation of service in case of “disastrous
    conditions”
    

    View Slide

52. Thank you
    Alexey Ragozin
    alexey.ragozin@gmail.com
    http://blog.ragozin.info
    - my articles
    http://code.google.com/p/gridkit
    - my open source code
    

    View Slide