From Vertical to Horizontal

A presentation about the challenges of scalability in the cloud, given at LinuxWochen Wien 2017

Transcript

1. @pyr
   From Vertical to Horizontal
   The challenges of scalability in the cloud
   

   View Slide

2. @pyr
   Four-line bio
   ● CTO & co-founder at Exoscale
   ● Open Source Developer
   ● Monitoring & Distributed Systems Enthusiast
   ● Linux since 1997
   

   View Slide

3. @pyr
   Scalability
   “The ability of a system, network, or process to handle
   a growing amount of work in a capable manner or its
   ability to be enlarged to accommodate that growth”
   - Wikipedia
   

   View Slide

4. @pyr
   Scalability
   ● Culture
   ● Organization and Process
   ● Technical Architecture
   ● Operations
   

   View Slide

5. @pyr
   Scalability
   ● Culture
   ● Organization and Process
   ● Technical Architecture
   ● Operations
   

   View Slide

6. @pyr
   Scaling Geometry
   Recent History
   Enter the cloud
   Distributed Headaches
   Architecture Drivers
   Looking forward
   

   View Slide

7. Quick Notes
   ● “Cloud” an umbrella term
   ● Here conflated with public IAAS
   ● Oriented toward web application design
   

   View Slide

8. @pyr
   Scaling Geometry
   Vertical, Horizontal, and Diagonal
   

   View Slide

9. @pyr
   Vertical (scaling up)
   Adding resources to a single system
   

   View Slide

10. @pyr
    Vertical (scaling up)
    This is how you typically approach scaling MySQL
    

    View Slide

11. @pyr
    

    View Slide

12. @pyr
    Horizontal (scaling out)
    Accommodate growth by spreading workload over
    several systems
    

    View Slide

13. @pyr
    Horizontal (scaling out)
    Typical approach to scaling web servers
    

    View Slide

14. @pyr
    

    View Slide

15. @pyr
    

    View Slide

16. @pyr
    Diagonal
    Most common strategy: vertical first, and then
    horizontal
    

    View Slide

17. @pyr
    Recent History
    Leading up to IAAS
    

    View Slide

18. View Slide

19. @pyr
    Whenever possible, a great approach
    

    View Slide

20. @pyr
    So, why stop?
    

    View Slide

21. View Slide

22. @pyr
    Moore’s law
    “Over the history of computing, the number of
    transistors on integrated circuits doubles
    approximately every two years.”
    

    View Slide

23. @pyr
    Average core speed has been stable for several years
    Consistent increase in cores per node
    

    View Slide

24. @pyr
    “You mean I have to use threads?”
    

    View Slide

25. @pyr
    Vertical Scaling Challenges
    (424 pages)
    

    View Slide

26. @pyr
    Vertical Scaling Challenges
    Threads?
    

    View Slide

27. @pyr
    No more automatic vertical approach
    

    View Slide

28. @pyr
    Meanwhile...
    

    View Slide

29. @pyr
    “What if I put an API on it?”
    

    View Slide

30. @pyr
    Enter: the Cloud
    

    View Slide

31. @pyr
    ● IT as a utility
    ● Programmable resources
    ● Decoupling of storage from system resources
    ● Usage-based billing model
    

    View Slide

32. Upside
    

    View Slide

33. @pyr
    ● Much lower capacity planning overhead
    ● OPEX makes accounting department happy
    ● Nobody likes to change disks or rack servers
    

    View Slide

34. @pyr
    ● Switches? gone.
    ● VLANs? gone.
    ● IP allocation and translation? gone.
    ● OS partitioning? gone.
    ● OS RAID management? gone.
    

    View Slide

35. @pyr
    

    View Slide

36. @pyr
    provider "exoscale" {
    api_key = "${var.exoscale_api_key}"
    secret_key = "${var.exoscale_secret_key}"
    }
    resource "exoscale_instance" "web" {
    template = "ubuntu 17.04"
    disk_size = "50g"
    template = "ubuntu 17.04"
    profile = "medium"
    ssh_key = "production"
    }
    

    View Slide

37. Downside
    

    View Slide

38. @pyr
    “There is no cloud, there is just someone else’s computer”
    

    View Slide

39. @pyr
    ● It’s hard to break out of the big iron mental model
    ● It’s hard to change our trust model
    ○ “I want to be able to see my servers!”
    ● There is still an upper limit on node size
    ● Horizontal-first approach to building infrastructure
    

    View Slide

40. @pyr
    Distributed Headaches
    

    View Slide

41. @pyr
    Two nodes interacting imply a distributed system
    Reduces SPOF, increases amount of failure scenarios
    

    View Slide

42. @pyr
    Distributed systems are subject to Brewer/CAP
    Cannot enjoy three of Consistency, Availability, Partition tolerance
    

    View Slide

43. @pyr
    ● Consistency: Simultaneous requests see a consistent set of data
    ● Availability: Each incoming request is acknowledged and receives a success or failure
    response
    ● Partition Tolerance: The system will continue to process incoming requests in the face
    of failures
    

    View Slide

44. @pyr
    Architecture Drivers
    Reducing complexity to focus on higher order problems
    

    View Slide

45. @pyr
    Inspectable services
    Queues over RPC
    Degrade gracefully
    Prefer concerned citizens
    Configuration from a service registry
    Nodes as immutable data structures
    

    View Slide

46. @pyr
    Inspectable services
    

    View Slide

47. @pyr
    Build introspection within services
    Number of acknowledged, processed, failed requests
    Time actions to quickly identify hotspots
    

    View Slide

48. @pyr
    Avoid the monitor effect
    Small unobtrusive probes
    UDP is often sufficient
    

    View Slide

49. @pyr
    Leverage proven, existing tools
    Collectd, Syslog-NG, Statsd, Riemann
    

    View Slide

50. @pyr
    @wrap_riemann('activate-account')
    def activate_account(self, id):
    self.accounts.by_id(id).try_activate()
    

    View Slide

51. @pyr
    Queues over RPC
    

    View Slide

52. @pyr
    RPC couples systems
    Your service’s CAP properties are tied to the RPC provider
    

    View Slide

53. @pyr
    Take responsibility out of the callee as soon as
    possible
    Textbook example: SMTP
    

    View Slide

54. @pyr
    Queues promote statelessness
    {
    request_id: "97d4f7b3",
    host_id: "64e4-41b5",
    action: "mailout",
    recipients: [ "foo@example.com" ],
    content: "..."
    }
    

    View Slide

55. @pyr
    Queues help dynamically shape systems
    Queue backlog growing? Spin-up new workers!
    

    View Slide

56. @pyr
    Degrade Gracefully
    

    View Slide

57. @pyr
    Embrace failure
    Systems will fail. In ways you didn’t expect.
    

    View Slide

58. @pyr
    Avoid failure propagation
    Implement backpressure to avoid killing loaded systems. Queues make great pressure valves.
    

    View Slide

59. @pyr
    Don’t give up
    Use connection pooling and retry policies.
    Best in class: finagle, cassandra-driver
    

    View Slide

60. @pyr
    Keep systems up
    SQL down? No more account creations, still serving existing customers.
    

    View Slide

61. @pyr
    Prefer Concerned Citizens
    

    View Slide

62. @pyr
    All moving parts force new compromises
    This is true of internal and external components
    

    View Slide

63. @pyr
    Choose components accordingly
    

    View Slide

64. @pyr
    You probably want an AP queueing system
    So please avoid using MySQL as one!
    Candidates: Apache Kafka, RabbitMQ, Redis (to a lesser extent)
    

    View Slide

65. @pyr
    Cache locally
    Much higher aggregated cache capacity
    No Huge SPOF
    

    View Slide

66. @pyr
    Choose your storage compromises
    Object Storage, Distributed KV (eventual consistency), SQL (no P or A).
    

    View Slide

67. @pyr
    Configuration through service registries
    

    View Slide

68. @pyr
    Keep track of node volatility
    Reprovisioning of configuration on cluster topology changes
    Load-balancers make a great interaction point (concentrate changes there)
    

    View Slide

69. @pyr
    The service registry is critical
    Ideally needs to be a strongly consistent, distributed system.
    You already have an eventually consistent one: DNS!
    

    View Slide

70. @pyr
    Zookeeper and Etcd
    Current best in class. Promotes usage in-app as well as distributed locks, barriers, etc.
    

    View Slide

71. @pyr
    Immutable Infrastructure
    

    View Slide

72. @pyr
    No more fixing nodes
    Human intervention means configuration drift
    

    View Slide

73. @pyr
    ● Configuration Drift? Reprovision node.
    ● New version of software? Reprovision node.
    ● Configuration file change? Reprovision node.
    

    View Slide

74. @pyr
    Depart from using the machine as the base unit of
    reasoning
    All nodes in clusters should be equivalent
    

    View Slide

75. @pyr
    Looking Forward
    The cluster is the computer
    

    View Slide

76. @pyr
    A new layer of abstraction
    Virtual resources pooled and orchestrated
    

    View Slide

77. @pyr
    Generic platform abstractions
    PAAS solutions are a commodity (cf: OpenShift)
    Generic scheduling and failover frameworks (Mesos, Kubernetes Operators)
    

    View Slide

78. @pyr
    Thanks!
    Questions?
    

    View Slide