Billing the Cloud

A real world use case of stream processing. How we went from simlpe & naive cron jobs to Kafka stream processing to bill our customers.

Transcript

1. 1 Billing the cloud Real world stream processing
2. 2 . 1 @pyr Co-Founder, CTO at Exoscale Open source developer @marcaurele Senior Engineer @ Exoscale Ex-sport junky & world traveler
3. 3 . 1 Today Problem domain Scaling methodologies Our approach
4. 4 . 1
5. 4 . 1 5 . 1
6. 5 . 1 6 . 1
7. 6 . 1 7 . 1 Infrastructure isn't free!
8. 8 . 1 Business Model Provide cloud infrastructure ??? Pro t!
9. 9 . 1
10. 9 . 1 10 . 1
11. 10 . 1 11 . 1 10000 mile high view
12. 12 . 1
13. 12 . 1 13 . 1 Quantities Resources
14. 13 . 1 14 . 1 Quantities 10 megabytes have been sent from 159.100.251.251 over the last minute
15. 15 . 1 Resources Account kickass-company started instance foo with pro le large today at 12:00 Account kickass-company stopped instance foo today at 12:15
16. 16 . 1 A bit closer to reality {:type :usage :entity :vm :action :create :time #inst "2016-12-12T15:48:32.000-00:00" :template "ubuntu-16.04" :source :cloudstack :account "kickass-company" :uuid "7a070a3d-66ff-4658-ab08-fe3cecd7c70f" :version 1 :offering "medium"}
17. 17 . 1 A bit closer to reality message IPMeasure { /* Versioning */ required uint32 header = 1; required uint32 saddr = 2; required uint64 bytes = 3; /* Validity */ required uint64 start = 4; required uint64 end = 5; }
18. 18 . 1 Theory
19. 19 . 1 Quantities are simple
20. 20 . 1
21. 20 . 1 21 . 1 Resources are harder
22. 22 . 1
23. 22 . 1 23 . 1 This is per-account
24. 24 . 1
25. 24 . 1 25 . 1 Solving for all events resources = {} metering = [] def usage_metering(): for event in fetch_all_events(): uuid = event.uuid() time = event.time() if event.action() == 'start': resources[uuid] = time else: timespan = duration(resources[uuid], time) usage = Usage(uuid, timespan) metering.append(usage) return metering
26. 26 . 1 Practical matters This is a never-ending process Minute precision billing Only apply once an hour Avoid over billing at all cost Avoid under billing (we need to eat!)
27. 27 . 1 Practical matters Keep a small operational footprint
28. 28 . 1 A naive approach
29. 29 . 1 32 * * * * usage-metering >/dev/null 2>&1
30. 29 . 1 30 . 1
31. 30 . 1 31 . 1
32. 31 . 1 32 . 1 Advantages
33. 33 . 1 Low operational overhead Simple functional boundaries Easy to test
34. 33 . 1 34 . 1 Drawbacks High pressure on SQL server Hard to avoid overlapping jobs Overlaps result in longer metering intervals
35. 35 . 1 You are in a room full of overlapping cron jobs. You can hear the screams of a dying MySQL server. An Oracle vendor is here. To the West, a door is marked "Map/Reduce" To the East, a door is marked "Streaming"
36. 35 . 1 36 . 1 > Talk to Oracle
37. 37 . 1 You have been eaten by a grue.
38. 37 . 1 38 . 1 > Go West
39. 39 . 1
40. 39 . 1 40 . 1 Conceptually simple Spreads easily Data-locality aware processing
41. 40 . 1 41 . 1 ETL High latency High operational overhead
42. 41 . 1 42 . 1
43. 42 . 1 43 . 1 > Go East
44. 44 . 1
45. 44 . 1 45 . 1 Continuous computation on an unbounded stream
46. 45 . 1 46 . 1 Each event processed as it comes in Very low latency A never ending reduce
47. 46 . 1 47 . 1 (reductions + [1 2 3 4]) ;; => (1 3 6 10)
48. 47 . 1 48 . 1 Conceptually harder Where do we store intermediate results? How does data ow between computation steps?
49. 48 . 1 49 . 1
50. 49 . 1 50 . 1 Deciding factors
51. 51 . 1 Our shopping list
52. 52 . 1 Operational simplicity Integration through our whole stack Going beyond billing Room to grow
53. 52 . 1 53 . 1 Operational simplicity Experience matters Spark and Storm are intimidating Hbase & Hive discarded
54. 54 . 1 Integration HDFS would require simple integration Spark usually goes hand in hand with Cassandra Storm tends to prefer Kafka
55. 55 . 1 Room to grow A ton of logs A ton of metrics
56. 56 . 1 Thursday confessions Previously knew Kafka
57. 57 . 1
58. 57 . 1 58 . 1
59. 58 . 1 59 . 1 Publish & Subscribe Processing Store
60. 59 . 1 60 . 1 Publish & Subscribe Messages are produced to topics Topics have a prede ned number of partitions Messages have a key which determines its partition
61. 61 . 1 Consumers get assigned a set of partitions Consumers store their last consumed offset Brokers own partitions, handle replication
62. 61 . 1 62 . 1
63. 62 . 1 63 . 1 Stable consumer topology Memory desaggregation Can rely on in-memory storage
64. 63 . 1 64 . 1 Stream expiry
65. 65 . 1
66. 65 . 1 66 . 1
67. 66 . 1 67 . 1
68. 67 . 1 68 . 1
69. 68 . 1 69 . 1 Problem solved?
70. 70 . 1 Process crashes Undelivered message? Avoiding double billing Hourly credit change
71. 70 . 1 71 . 1 Process crashes Triggers a rebalance Loss of in-memory cache No initial state!
72. 72 . 1 Reconciliation Snapshot of full inventory Converges stored resource state if necessary Handles failed deliveries as well
73. 73 . 1 Avoiding double billing Reconciler acts as logical clock When supplying usage, attach a unique transaction ID Reject multiple transaction attempts on a single ID
74. 74 . 1 Looking back Things stay simple (roughly 600 LoC) Room to grow Stable and resilient DNS, Logs, Metrics, Event Sourcing
75. 75 . 1 What about batch Streaming doesn't work for everything Sometimes throughput matters more than latency Building models in batch, applying with stream processing
76. 76 . 1 Questions? Thanks!