Network Partition Handling in Geode

A look into how Apache Incubator Geode handles network failures

Transcript

1. Network Partition Handling in Geode Bruce Schuchardt
2. Quorum Based Loc1 P2 P1 P3 Loc2 [Loc1|1: Loc1, Loc2, P1, P2, P3] Locators weigh 3 Lead member 15 Others 10 This view has a weight of 41 (3+3+15+10+10)
3. Network Partition! Loc1 P2 P1 P3 Loc2 Loss of Loc2 detected Membership view: [Loc1|2: Loc1, P1, P2, P3] PREPARE_FOR_VIEW_CHANGE!
4. Network Partition! Loc1 P2 P1 P3 Loc2 P3 doesn’t respond! P3 health check fails Membership view: [Loc1|3: Loc1, P1, P2] Lost weight = 3 + 10 (Loc2 & P3) New view is prepared and installed
5. Network Partition! Loc1 P2 P1 P3 Loc2 Loc2 determines Loc1 is gone. Loc2 prepares new view [Loc2|2: Loc2, P1, P2, P3]
6. Network Partition! Loc1 P2 P1 P3 Loc2 P1 & P2 don’t respond! P1 & P2 health check fails New view: [Loc2|3: Loc2, P3] Lost weight = 3 + 15 + 10 (Loc1 & P1 & P2) 28 is > 50% of the original weight! Network Partition!
7. Network Partition! Loc1 P2 P1 P3 Loc2 Loc2 sends network partition message to P3. Loc2 & P3 disconnect and close caches Network-partition message
8. Quorum Checkers Loc1 P2 P1 P3 Loc2 Quorum Checkers periodically search for lost members, trying to reach a quorum
9. Quorum Checkers Loc1 P2 P1 P3 Loc2 When the network is healed quorum is re-established. Loc2 starts location services.
10. Quorum regained Loc1 P2 P1 P3 Loc2 [Loc1|4: Loc1, Loc2, P1, P2, P3] Loc2 and P3 rebuild their caches, recovering from other members and/or from disk
11. How to tell reconnect is happening "ReconnectThread" prio=10 tid=0x00007f2e28005800 nid=0x1f39 in Object.wait() [0x00007f2eb880e000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000ff708148> (a java.lang.Object) at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2777) at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2638) - locked <0x00000000ff708148> (a java.lang.Object) - locked <0x00000000ff85f638> (a java.lang.Class for com.gemstone.gemfire.internal.cache.GemFireCacheImpl) - locked <0x00000000ff78ac88> (a java.lang.Class for com.gemstone.gemfire.cache.CacheFactory) at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1053) at com.gemstone.gemfire.distributed.internal.DistributionManager$MyListener.membershipFailure(DistributionManager.java:4689) at "GemFire Auto-reconnect responder" daemon prio=10 tid=0x00007f2de802e800 nid=0x1f3c waiting on condition [0x00007f2eb981e000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.gemstone.gemfire.distributed.internal.membership.jgroup.QuorumCheckerImpl$1.run(QuorumCheckerImpl.java:254)
12. DEMO
13. Q & A Bruce Schuchardt: bschuchardt@apache.org