Building a cloud service on a cloud infrastructure. Also, cloud.

Transcript

1. Building a cloud service on a cloud infrastructure at Building

   a cloud service on a cloud infrastructure at Also, cloud. Also, cloud. Mikhail Panchenko, Surge 2011

2. Who Am I? Who Am I? Pancakes Infrastructure Engineer at

   SimpleGeo Backend Engineer at Flickr before that Backend and Frontend Engineer at Yahoo! Ops/Tools before that Philosophy, Economics, and French major before that @mihasya [email protected]

3. Tools for mobile/geo developers Primarily focused on services, some data-

   oriented APIs PaaS, I guess? I've lost track a bit Availability, redundancy part of brand Our outage = your outage No pressure

4. Agenda Agenda Goals A little bit of theory Challenges in

   The Cloud General Architecture Implementation Details

5. Architectural Goals Architectural Goals High availability Linear scalability Elasticity/Flexibility Redundancy/Fault

   Tolerance

6. Read: don't wake me up, please Read: don't wake me

   up, please

7. Sound Familiar? Sound Familiar?

8. Some Theory, Food for Thought Some Theory, Food for Thought

9. The Internets as Complex Systems The Internets as Complex Systems

10. http://www.amazon.com/Normal-Accidents-Living-High-Risk-Technologies/dp/0691004129

11. "Complex interactions are those of unfamiliar sequences, or unplanned and

    unexpected sequences, and either not visible or not immediately comprehensible." Charles Perrow. Normal Accidents: Living with High-Risk Technologies (p. 78). Kindle Edition.

12. "The notion of baffling interactions is increasingly familiar to all

    of us. [...] As systems grow in size and in the number of diverse functions they serve, and are built to function in ever more hostile environments, increasing their ties to other systems, they experience more and more incomprehensible or unexpected interactions. They become more vulnerable to unavoidable system accidents." Charles Perrow. Normal Accidents: Living with High-Risk Technologies (p. 72). Kindle Edition.

13. Fortunately, Fortunately, This Is Only The Internet This Is Only

    The Internet

14. "The beauty of this is its simplicity. Once a plan

    gets too complex, everything can go wrong." Walter Sobchak, The Big Lebowski

15. Interactions Interactions Linear vs Complex Linear vs Complex

16. Coupling Coupling Tight vs Loose Tight vs Loose

17. None

18. Three Mile Island Three Mile Island "... they found that

    radioactive water was not traveling to the tank they intended, but because of complex flow and pressure interactions, was going to a different, wrong tank, which also overflowed, this time in the auxiliary building." Charles Perrow. Normal Accidents: Living with High-Risk Technologies (pp. 22-23). Kindle Edition.

19. Amazon Web Services Amazon Web Services "The traffic shift was

    executed incorrectly and rather than routing the traffic to the other router on the primary network, the traffic was routed onto the lower capacity redundant EBS network." "Summary of the Amazon EC2 and Amazon RDS Service Disruption in the US East Region" http://aws.amazon.com/message/65648/

20. Common Theme Common Theme Previously independent systems become coupled as

    a result of unanticipated interactions, leading to fundamentally surprising results

21. When pumping radioactive water into the wrong When pumping radioactive

    water into the wrong tank, the behavior of the program is undefined tank, the behavior of the program is undefined

22. But where does The Cloud come in?? But where does

    The Cloud come in??

23. The Trifle Analogy The Trifle Analogy Photo by mathematically_impossible

24. The Trifle Analogy The Trifle Analogy Photo by mathematically_impossible

25. A complex system consisting of complex subsystems A complex system

    consisting of complex subsystems

26. Photo by wwarby

27. The Trifle Analogy The Trifle Analogy Original photos by mathematically_impossible

    and miheco

28. Tightly coupled to a complex system over which you Tightly

    coupled to a complex system over which you have no control and into which you have no insight have no control and into which you have no insight

29. Photo by 20after4

30. Recall Recall "Baffling Interactions" "Baffling Interactions"

31. "The notion of baffling interactions is increasingly familiar to all

    of us. [...] As systems grow in size and in the number of diverse functions they serve, and are built to function in ever more hostile environments, increasing their ties to other systems, they experience more and more incomprehensible or unexpected interactions. They become more vulnerable to unavoidable system accidents." Charles Perrow. Normal Accidents: Living with High-Risk Technologies (p. 72). Kindle Edition.

32. DECOUPLE DECOUPLE DECOUPLE DECOUPLE DECOUPLE DECOUPLE ( also, simplify )

    ( also, simplify )

33. Photo by erikcharlton

34. Decouple Your Subsystems Decouple Your Subsystems Shared resources are the

    most common source of unexpected interaction Resist temptation to double up on roles Use queues, caches as buffers NOTE: those are complex subsystems of their own

35. Decouple Your Subsystems Decouple Your Subsystems Explicit Decoupling CPU Affinity

    Webserver on 1-7; SSH etc on 8 Crude, but gets the job done More robust solutions - containers

36. Decouple Your Functionality Decouple Your Functionality Service architecture Each service

    does one thing well Easier to measure, understand, and accommodate resource demands Reduce potential for interactions, cross-functional failure

37. Decouple from Your Environment with Configuration Decouple from Your Environment

    with Configuration Management Management Decouple from your platform (OS/kernel) Easy to test/bench potential candidates Easy to migrate if you find a winner This is especially important when dealing with cloud Automate as much of deploy/bootstrap process as possible Probably won't help much during a provider outage due to stampede BUT: DirectConnect You might not always be in the cloud..

38. Decouple Your Datacenters Decouple Your Datacenters Most robust redundancy mechanism

    Hot-hot keeps you on your toes Simplifies, not just for the cloud Yahoo! now foregoing datacenter features like HVAC "If it gets too hot in Washington, turn that DC off for a while" I'm sure they're not the only ones

39. Decouple Your Datacenters Decouple Your Datacenters "AZ" - Basic building

    block for EC2 This is the level they (theoretically) decouple at They are probably thinking along the same lines we are - must be able to turn off one AZ without impact in the other

40. ( there's a hidden interaction there ) ( there's a

    hidden interaction there )

41. Every datacenter as an independent microcosm of Every datacenter as

    an independent microcosm of your overall architecture your overall architecture

42. The Birds 'n' the Bees The Birds 'n' the Bees

43. Bird's Eye View Bird's Eye View

44. Photo by reschroederimages

45. Bird's Eye View Bird's Eye View

46. ( note the absence of specifics ) ( note the

    absence of specifics )

47. Bird's Eye View Bird's Eye View

48. Maintenance - Divide & Conquer Maintenance - Divide & Conquer

49. Local Degradation - Divide & Conquer Local Degradation - Divide

    & Conquer

50. Incompatible Upgrade - Guess! Incompatible Upgrade - Guess!

51. Incompatible Upgrade - Guess! Incompatible Upgrade - Guess!

52. Incompatible Upgrade - Yay! Incompatible Upgrade - Yay!

53. Baffling Single Node Failure Baffling Single Node Failure

54. 202 Accepted 202 Accepted

55. Spike in Write Traffic Spike in Write Traffic

56. Really simple operational steps for stressful tasks Really simple operational

    steps for stressful tasks & situations & situations

57. Temporally decouple the problem from the Temporally decouple the problem

    from the resolution resolution

58. Go back to sleep Go back to sleep Photo by

    joshme17

59. Now, how about those specifics? Now, how about those specifics?

60. Write Path Write Path

61. ELB ELB Dynamic Load Balancing Flexible virtual IP Easy to

    add/remove AZs Uses healthchecks to automatically evict nodes

62. Gate - "Layer 8 Proxy" Gate - "Layer 8 Proxy"

    Lightweight Node.js daemon OAuth Rate Limiting Basic routing to actual services

63. Recall Recall "Decouple Your Functionality" "Decouple Your Functionality"

64. Services - Pick Your Own Adventure Services - Pick Your

    Own Adventure Node.js and Python Some people just hate Node.js Can be anything, as long as Gate can talk to it ( another reason to decouple ) Highly specialized

65. RabbitMQ RabbitMQ A grenade for our knife-fight Very flexible -

    more than we need Simplification candidate New persistor in >= 1.3 - degradation over failure See talk at 1:30PM

66. Cassandra Cassandra A mostly-textbook DHT Homogenous distributed model Random load

    distribution Partition tolerance A perfect foundation for our architecture

67. Partition Tolerance Partition Tolerance It's not just for outages

68. Recall Recall "Divide & Conquer" "Divide & Conquer"

69. This too is a partition This too is a partition

70. Thank You! Thank You! @mihasya [email protected]