Distributed Systems at ok.ru

ok.ru is one of top 10 internet sites of the World, according to similarweb.com. Under the hood, it has several thousand servers. Each of those servers own only fraction of the data or business logic. Shared nothing architecture can be hardly applied to social network, due to its nature, so a lot of communication happens between these servers, diverse in kind and volume. This makes ok.ru one of the largest, complicated, yet highly loaded distributed systems in the world.

This talk is about our experience in building always available, resilient to failures distributed systems in Java, their basic and not so basic failure and recovery scenarios, methods of failure testing and diagnostics. We’ll also discuss on possible disasters and how to prevent or get over them.

Transcript

1. Distributed Systems @ OK.RU Oleg Anastasyev @m0nstermind oa@ok.ru
2. 1. Absolutely reliable network 2. with negligible Latency 3. and practically unlimited Bandwidth 4. It is homogenous 5. Nobody can break into our LAN 6. Topology changes are unnoticeable 7. All managed by single genius admin 8. So data transport cost is zero now 2 OK.ru has come to:
3. 1. Absolutely reliable network 2. with negligible Latency 3. and practically unlimited Bandwidth 4. It is homogenous (same HW and hop cnt to every server) 5. Nobody can break into our LAN 6. Topology changes are unnoticeable 7. All managed by single genius admin 8. So data transport cost is zero now 3 Fallacies of distributed computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing [Peter Deutsch, 1994; James Gosling 1997]
4. 4 4 Datacenters 150 distinct microservices 8000 iron servers OK.RU has come to:
5. 5 hardware engineers network engineers operations developers
6. 6 My friends page 1. Retrieve friends ids 2. Filter by friendship type 3. Apply black list 4. Resolve ids to profiles 5. Sort profiles 6. Retrieve stickers 7. Calculate summaries
7. 7 The Simple WayTM SELECT * FROM friendlist, users 
 WHERE userId=? AND f.kind=? AND u.name LIKE ? AND NOT EXISTS( SELECT * FROM blacklist …) …
8. • Friendships • 12 billions of edges, 300GB • 500 000 requests per sec 8 Simple ways don't work • User profiles • > 350 millions, • 3 500 000 requests/sec, 50 Gbit/sec
9. 9 How stuff works web frontend API frontend app server one-graph user-cache black-list microservices
10. 10 Micro-service dissected Remote interface Business logic, caches [ Local storage ] 1 JVM
11. 11 Micro-service dissected Remote interface https://github.com/odnoklassniki/one-nio interface GraphService extends RemoteService { @RemoteMethod long[] getFriendsByFilter(@Partition long vertexId, long relationMask); } interface UserCache {
 @RemoteMethod User getUserById(long id); }
12. 12 App Server code https://github.com/odnoklassniki/one-nio long []friendsIds = graphService.getFriendsByFilter(userId, mask); List<User> users = new ArrayList<Long>(friendsIds.length); for (long id : friendsIds) { if(blackList.isAllowed(userId,id)) { users.add(userCache.getUserById(id)); } } … return users;
13. • Partition by this parameter value • Using partitioning strategy • long id -> int partitionId(id) -> node1,node2,… • Strategies can be different • Cassandra ring, Voldemort partitions • or … 13 interface GraphService extends RemoteService { @RemoteMethod long[] getFriendsByFilter(@Partition long vertexId, long relationMask); }
14. 14 Weighted quadrant p = id % 16 p = 0 p = 15 p = 1 N01 N02 N03 . . . 019 020 W = 1 W = 100 N11 node = wrr(p) SET
15. 15 A coding issue https://github.com/odnoklassniki/one-nio long []friendsIds = graphService.getFriendsByFilter(userId, mask); List<User> users = new ArrayList<Long>(friendsIds.length); for (long id : friendsIds) { if(blackList.isAllowed(userId,id)) { users.add(userCache.getUserById(id)); } } … return users;
16. 16 latency 
 = 1.0ms * 2 reqs * 200 friends
 = 400 ms
 A roundtrip price 0.1-0.3 ms 0.7-1.0 ms remote datacenter * this price is tightly coupled with the specific infrastructure and frameworks 10k friends latency = 20 seconds
17. 17 Batch requests to the rescue public interface UserCache {
 @RemoteMethod( split = true ) Collection<User> getUsersByIds(long[] keys); } long []friendsIds = graphService.getFriendsByFilter(userId, mask); 
 friendsIds = blackList.filterAllowed(userId, friendsIds ); List<User> users = userCache.getUsersByIds(friendsIds); … return users;
18. 18 split & merge split ( ids by p ) -> ids0, ids1 p = 0 p = 1 N01 N02 N03 . . . N11 ids0 ids1 users = merge (users0, users1)
19. 19 1. Client crash 2. Server crash 3. Request omission 4. Response omission 5. Server timeout 6. Invalid value response 7. Arbitrary failure What could possibly fail ?
20. Failures Distributed systems at OK.RU
21. • We can not prevent failures - only mask them • If a Failure can occur it will occur • Redundancy is a must to mask failures • Information ( error correction codes ) • Hardware (replicas, substitute hardware) • Time (transactions, retries) 21 What to do with failures ?
22. 22 What happened to transaction ? Don’t give up! Must retry ! Must give up! 
 Don't retry ! ? ? Add Friend
23. • Client does not really know • What client can do ? • Don’t make any guarantees. • Never retry. At Most Once. • Always retry. At Least Once. 23 Was friendship succeeded ?
24. 1. Transaction in ACID database • single master, success is atomic (either yes or no) • atomic rollback is possible 2. Cache cluster refresh • many replicas, no master • no rollback, partial failures are possible 24 Making new friendship
25. • Operation can be reapplied multiple times with same result • e.g.: read, Set.add(), Math.max(x,y) • Atomic change with order and dup control
 25 Idempotence “Always retry” policy can be applied
 only on
 Idempotent Operations https://en.wikipedia.org/wiki/Idempotence
26. 26 Idempotence in ACID database Make friends wait; timeout Make friends (retry) Friendship, peace and bubble gum ! Already friends ? No, let’s make it ! Already friends ? Yes, NOP !
27. 27 Sequencing MakeFriends (OpId) Made friends! Is Dup (OpId) ? No, making changes OpId := Generate() Generate() examples: • OpId+=1 • OpId=currentTimeMillis() • OpId=TimeUUID http://johannburkard.de/software/uuid/
28. 1. Transaction in ACID database • single master, success is atomic (either yes or no) • atomic rollback is possible 2. Cache cluster refresh • many replicas, no master • no rollback, partial failures are possible 28 Making new friendship
29. 29 Cache cluster refresh add(Friend) p = 0 N01 N02 N03 . . . But replicas state will diverge otherwise Retries are meaningless
30. • Background data sync process • Reads updated records from ACID store
 
 SELECT * FROM users WHERE modified > ? • Applies them into its memory • Loads updates on node startup • Retry can be omitted then
 30 Syncing cache from DB
31. 31 Death by timeout GC Make Friends wait; timeout thread pool 
 exhausted
32. 1. Clients stop sending requests to server After X continuous failures for the last second 2. Clients monitor server availability In background, once a minute 3. And turn it back on 32 Server cut-off
33. 33 Death by slowing down Avg = 1.5ms Max = 1.5c 24 cpu cores Cap = 24,000 ops Choose 2.4ms timeout ? Cut it off from client if latency avg > 2.4ms ? Avg = 24ms Max = 1.5s 24 cpu cores Cap = 1,000 ops 10,000 ops
34. 34 Speculative retry Idemponent Op wait; timeout Retry Result Response
35. • Makes requests to replicas before timeout • Better 99%, even average latencies • More stable system • Not always applicable: • Idempotent ops, additional load, traffic (to consider) • Can be balanced: always, >avg, >99p 35 Speculative retry
36. More failures ! Distributed systems @ OK.RU
37. • Excessive load • Excessive paranoia • Bugs • Human error • Massive outages 37 All replicas failure
38. 38 Use of non-authoritative datasources, degrade consistency Use of incomplete data in UI, partial feature degradation
 Single feature full degradation Degrade (gracefully) !
39. 39 The code interface UserCache {
 @RemoteMethod Distributed<Collection<User>> getUsersByIds(long[] keys); } interface Distributed<D> { boolean isInconsistency(); D getData(); } class UserCacheStub implements UserCache { 
 Distributed<Collection<User>> getUsersByIds(long[] keys) { return Distributed.inconsistent(); } }
40. Resilience testing Distributed systems at OK.RU
41. 41 The product you make Operations in production env What to test for failure ? “Standard” products - with special care !
42. • What is does: • Detects network connections between servers • Disables them (iptables drop) • Runs auto tests • What we check • No crashes, nice UI messages are rendered • Server does start and can serve requests 42 The product we make : “Guerrilla”
43. Production diagnostics Distributed systems at OK.RU
44. • To know an accident exists. Fast. • To track down to the source of accident. Fast. • To prevent accidents before they happen. 44 Why
45. • Zabbix • Cacti • Operational metrics • Names od operations, e.g. “Graph.getFriendsByFilter” • Call count, their success or failure • Latency of calls 45 Is (will) there be accident ?
46. • Current metrics and trends • Aggregated call and failure counts • Aggregated latencies • Average, Max • Percentiles 50,75,98,99,99.9 46 What charts show to us
47. 47 More charts
48. 48 Anomaly detection
49. • The possibilities for failure in distributed systems are endless • Don't “prevent”, but mask failures through redundancy • Degrade gracefully on unmask-able failure • Test failures • Production diagnostics are key to failure detection and prevention 49 Short summary
50. 50 Distributed Systems at OK.RU slideshare.net/m0nstermind https://v.ok.ru/publishing.html http://www.cs.yale.edu/homes/aspnes/classes/465/notes.pdf Notes on Theory of Distributed Systems CS 465/565: 
 Spring 2014 James Aspnes Try these links for more