GeekcampSG - Introduction to Distributed Systems

Transcript

1. HOW TO STAY UP WHEN YOUR SERVER GOES DOWN (that includes AWS too)
2. WHY YOU NEED TO CARE?
4. •Customers don’t care
5. •Customers don’t care •Lost sales
6. •Customers don’t care •Lost sales •Bad rep
7. • A step by step magical formula that will ensure 100% uptime WHAT THIS TALK IS NOT
8. YOU CAN ONLY Reduce your chances of going down
9. FIND THE RIGHT BALANCE Between costs and how much you need to stay up.
10. THAT SAID There are cheap sources for low powered back up servers
11. WHAT THIS TALK REALLY IS
12. WHAT THIS TALK REALLY IS Introduction to distributed systems
13. DISTRIBUTED SYSTEMS
14. DISTRIBUTED SYSTEMS A collection of independent computers that appear to the users of the system as a single computer - Tanenbaum
15. CORE IDEAS
16. YUP It is one of those hardcore CS stuff that requires both practical ability and theoretical knowledge
17. DESIGN GOALS
18. DESIGN GOALS • Fault tolerant
19. DESIGN GOALS • Fault tolerant • Transparency
20. DESIGN GOALS • Fault tolerant • Transparency • Scalable
21. DESIGN GOALS • Fault tolerant • Transparency • Scalable • Open
22. DESIGN GOALS • Fault tolerant • Transparency • Scalable • Open • Secure
23. FAULT TOLERANT System should function even during failures of some parts Also known as making resources available
24. TRANSPARENCY Same response no matter where you access it from
25. SCALABLE Easily adaptable to increased load
26. OPENNESS Easy to extend and reimplement certain parts Set of rules to describe the standard behaviour
27. SECURE Network, authentication, access control
28. DISTRIBUTED SYSTEMS for your webapp
29. KEY CONCEPTS
30. REPLICATION
31. Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility. From Wikipedia [http://en.wikipedia.org/wiki/Replication_(computing)]
32. Human-speak : Storing your data multiple times in multiple locations and praying hard that all of them do not die at the same time
33. 1 2 3 4 Database Cluster
34. 1 2 3 4 Database Cluster Master Slave Slave Slave
35. 1 2 3 4 Database Cluster Down Slave Slave Slave
36. 1 2 3 4 Database Cluster Down Slave Master Slave
37. •Reads on any of the nodes •Writes only on master node •Works on the principle of eventually correct data
38. SHARDING
39. Horizontal partitioning is a database design principle whereby rows of a database table are held separately, rather than splitting by columns (which is what normalization and vertical partitioning do, to differing extents). Each partition forms part of a shard, which may in turn be located on a separate database server or physical location.
40. How MongoDB does it? Splitting of data into logical Chunks based on a predefined Shard Key.
41. Machine 1 Machine 2 Machine 3 Alabama → Arizona Colorado → Florida Arkansas → California Indiana → Kansas Idaho → Illinois Georgia → Hawaii Maryland → Michigan Kentucky → Maine Minnesota → Missouri Montana → Montana Nebraska → New Jersey Ohio → Pennsylvania New Mexico → North Dakota Rhode Island → South Dakota Tennessee → Utah Vermont → West Virgina Wisconsin → Wyoming TLDR VERSION
42. WHAT HAS THIS GOT TO DO WITH STAYING UP?
43. Machine 1 Machine 2 Machine 3 Alabama → Arizona Colorado → Florida Arkansas → California Indiana → Kansas Idaho → Illinois Georgia → Hawaii Maryland → Michigan Kentucky → Maine Minnesota → Missouri Montana → Montana Nebraska → New Jersey Ohio → Pennsylvania New Mexico → North Dakota Rhode Island → South Dakota Tennessee → Utah Vermont → West Virgina Wisconsin → Wyoming These users are still fine!
44. MERGING THE TWO
45. Machine 1 Machine 2 Machine 3 Alabama → Arizona Colorado → Florida Arkansas → California Indiana → Kansas Idaho → Illinois Georgia → Hawaii Maryland → Michigan Kentucky → Maine Minnesota → Missouri Montana → Montana Nebraska → New Jersey Ohio → Pennsylvania New Mexico → North Dakota Rhode Island → South Dakota Tennessee → Utah Vermont → West Virgina Wisconsin → Wyoming Machine 4 Machine 5 Machine 6 Alabama → Arizona Colorado → Florida Arkansas → California Indiana → Kansas Idaho → Illinois Georgia → Hawaii Maryland → Michigan Kentucky → Maine Minnesota → Missouri Montana → Montana Nebraska → New Jersey Ohio → Pennsylvania New Mexico → North Dakota Rhode Island → South Dakota Tennessee → Utah Vermont → West Virgina Wisconsin → Wyoming
46. Machine 1 Machine 2 Machine 3 Alabama → Arizona Colorado → Florida Arkansas → California Indiana → Kansas Idaho → Illinois Georgia → Hawaii Maryland → Michigan Kentucky → Maine Minnesota → Missouri Montana → Montana Nebraska → New Jersey Ohio → Pennsylvania New Mexico → North Dakota Rhode Island → South Dakota Tennessee → Utah Vermont → West Virgina Wisconsin → Wyoming Machine 4 Machine 5 Machine 6 Alabama → Arizona Colorado → Florida Arkansas → California Indiana → Kansas Idaho → Illinois Georgia → Hawaii Maryland → Michigan Kentucky → Maine Minnesota → Missouri Montana → Montana Nebraska → New Jersey Ohio → Pennsylvania New Mexico → North Dakota Rhode Island → South Dakota Tennessee → Utah Vermont → West Virgina Wisconsin → Wyoming
47. HOW SOME COMPANIES DO IT?
48. NETFLIX
49. SELF INFLICTED PAIN
51. Chaos Monkey Latency Monkey Conformity Monkey Doctor Monkey Janitor Monkey Security Monkey Chaos Gorilla NETFLIX SIMIAN ARMY
52. CHAOS MONKEY • Shuts down instances randomly • Allows the team to learn about weaknesses in the system • Gets the team to build auto recovery systems
53. LATENCY MONKEY • Introduces artificial delays in api client-server communication layer • Simulates server degradation • Useful for testing fault tolerance of new services
54. CONFORMITY MONKEY • Shuts down servers that don’t adhere to best practices • Forces the service owner to relaunch them properly
55. DOCTOR MONKEY • Runs checks on health of servers • Remove unhealthy servers from service and alerts service owner
56. SECURITY MONKEY • Scans for security violations or vulnerabilities • Ensures that SSL and DRM certs are not expiring
57. CHAOS GORILLA • Shuts down all instances within an availability zone • Protects themselves from electric storms
58. WHAT CAN YOU DO?
59. RUN YOUR OWN SIMIAN ARMY https://github.com/Netflix/SimianArmy
61. •Data storage is geo replicated •Built-in high availability features •Tech support hotline without paying for more
64. • mongod  -­‐-­‐fork  -­‐-­‐replSet  demo  -­‐-­‐logpath  /tmp/mongo.log
65. •rs.initiate() •rs.add('mongodb2.cloudapp.net:27017')  
66. DEMO
67. STUFF I HAVE RUNNING ON HA
70. Q&A
71. THE END Enjoy your rest of the day :D