PyConZA 2012: "Fractal Architectures" by Laurens Van Houtven

7b0645f018c0bddc8ce3900ccc3ba70c?s=47 Pycon ZA
October 05, 2012

PyConZA 2012: "Fractal Architectures" by Laurens Van Houtven

An alternative take on the traditional layered service architecture.

Traditional service architecture wisdom generally tells us to build services like this:

Load balancer in front
Web servers, preferably stateless
Database (with a caching layer)
That works great for a wide variety of use cases. The point of this talk isn't to deprecate that design, but to discuss a radically different one.

The design I will present in this talk is one consisting of recurring, identical components. It localizes state to individual application servers and persists it to durable stores later. It aims to be easier to scale horizontally: that is, enabling you to increase throughput by simply adding more machines to the homogenous cluster.

I will talk about its benefits, such as performance and how it fits in well with many cloud providers' services, but also its downsides, such as the inherent complexities of distributed systems. I analyze these qualities to come to a conclusion about which kinds of project this design is suitable or not suitable for.

In this talk I will discuss both the abstract concepts and the practical implementation that I have built using Twisted and Axiom (a simple object database on top of SQLite 3), which is currently running in production. Although I will touch on the practical implementation, the talk should still be useful for anyone wanting to implement a similar idea using different tools.

7b0645f018c0bddc8ce3900ccc3ba70c?s=128

Pycon ZA

October 05, 2012
Tweet

Transcript

  1. Fractal Architectures @lvh Friday, October 12, 2012

  2. $ whoami Friday, October 12, 2012

  3. @lvh, _@lvh.cc Friday, October 12, 2012

  4. Friday, October 12, 2012

  5. #python Friday, October 12, 2012

  6. Friday, October 12, 2012

  7. Friday, October 12, 2012

  8. Architectures Friday, October 12, 2012

  9. Load Balancing Application Servers Cache Database Services Friday, October 12,

    2012
  10. LB App DB App App App DB Friday, October 12,

    2012
  11. ! Friday, October 12, 2012

  12. DB Friday, October 12, 2012

  13. DB Friday, October 12, 2012

  14. DB Friday, October 12, 2012

  15. DB Friday, October 12, 2012

  16. DB $ $$ Friday, October 12, 2012

  17. App Friday, October 12, 2012

  18. + Queue Workers Friday, October 12, 2012

  19. LB App DB App App App DB Friday, October 12,

    2012
  20. Friday, October 12, 2012

  21. Friday, October 12, 2012

  22. Friday, October 12, 2012

  23. Friday, October 12, 2012

  24. Friday, October 12, 2012

  25. Friday, October 12, 2012

  26. Friday, October 12, 2012

  27. Friday, October 12, 2012

  28. app Friday, October 12, 2012

  29. user Friday, October 12, 2012

  30. thing Friday, October 12, 2012

  31. k n⋅k Friday, October 12, 2012

  32. Friday, October 12, 2012

  33. Friday, October 12, 2012

  34. Friday, October 12, 2012

  35. Friday, October 12, 2012

  36. Restrictions Creativity Friday, October 12, 2012

  37. Restrictions Strength Friday, October 12, 2012

  38. Friday, October 12, 2012

  39. ! Friday, October 12, 2012

  40. ! Friday, October 12, 2012

  41. Friday, October 12, 2012

  42. Friday, October 12, 2012

  43. Friday, October 12, 2012

  44. Friday, October 12, 2012

  45. Friday, October 12, 2012

  46. Friday, October 12, 2012

  47. Friday, October 12, 2012

  48. Friday, October 12, 2012

  49. Friday, October 12, 2012

  50. Friday, October 12, 2012

  51. Friday, October 12, 2012

  52. Friday, October 12, 2012

  53. SMTP Friday, October 12, 2012

  54. import this Friday, October 12, 2012

  55. Special cases aren't special enough to break the rules. Friday,

    October 12, 2012
  56. Although practicality beats purity. Friday, October 12, 2012

  57. Friday, October 12, 2012

  58. Friday, October 12, 2012

  59. Friday, October 12, 2012

  60. Friday, October 12, 2012

  61. Friday, October 12, 2012

  62. Friday, October 12, 2012

  63. Axiom Friday, October 12, 2012

  64. Simple Friday, October 12, 2012

  65. T O Friday, October 12, 2012

  66. Friday, October 12, 2012

  67. in-memory Friday, October 12, 2012

  68. Sophisticated Friday, October 12, 2012

  69. v1 (schema) v2 (schema) Friday, October 12, 2012

  70. v1 x (reference) v2 x (reference) y (integer) Friday, October

    12, 2012
  71. v1 x (reference) y (text) v2 x (reference) y (text)

    Friday, October 12, 2012
  72. A, vx ... A, vx+1 ... B, v1 ... Friday,

    October 12, 2012
  73. A, vx ... B, vy ... C, vz ... Friday,

    October 12, 2012
  74. ConstraintError: attribute [I.t = text()] must be (unicode string without

    NULL bytes); not 'str' Friday, October 12, 2012
  75. Friday, October 12, 2012

  76. + Friday, October 12, 2012

  77. Friday, October 12, 2012

  78. Friday, October 12, 2012

  79. Friday, October 12, 2012

  80. ! Friday, October 12, 2012

  81. Alternatives Friday, October 12, 2012

  82. Friday, October 12, 2012

  83. Friday, October 12, 2012

  84. Friday, October 12, 2012

  85. Longer term storage Friday, October 12, 2012

  86. Why? Friday, October 12, 2012

  87. Friday, October 12, 2012

  88. Friday, October 12, 2012

  89. Server A Server B Long Term Storage Friday, October 12,

    2012
  90. Server A Server B Long Term Storage Friday, October 12,

    2012
  91. Server A Server B Long Term Storage Friday, October 12,

    2012
  92. Options? Friday, October 12, 2012

  93. Friday, October 12, 2012

  94. Friday, October 12, 2012

  95. Friday, October 12, 2012

  96. Friday, October 12, 2012

  97. manhole Friday, October 12, 2012

  98. Anything will work Friday, October 12, 2012

  99. Friday, October 12, 2012

  100. Friday, October 12, 2012

  101. Friday, October 12, 2012

  102. Friday, October 12, 2012

  103. Friday, October 12, 2012

  104. Friday, October 12, 2012

  105. Friday, October 12, 2012

  106. Friday, October 12, 2012

  107. Two ways to better performance Friday, October 12, 2012

  108. Do less work or Do it faster Friday, October 12,

    2012
  109. 1E+00 1E+01 1E+02 1E+03 1E+04 1E+05 1E+06 1E+07 1E+08 1E+09

    L1 L2 RAM SSD LAN HD CA-NL Friday, October 12, 2012
  110. Local Low latency Friday, October 12, 2012

  111. Query latency vs. Access latency Friday, October 12, 2012

  112. (Almost) never Friday, October 12, 2012

  113. vs Remote Local Friday, October 12, 2012

  114. vs Remote Friday, October 12, 2012

  115. . . . Friday, October 12, 2012

  116. . . . { # Friday, October 12, 2012

  117. Generalizations are always wrong, but... Friday, October 12, 2012

  118. Computation is only rarely the bottleneck Friday, October 12, 2012

  119. Data in the right place at the right time is

    Friday, October 12, 2012
  120. pip install axiombench Friday, October 12, 2012

  121. Friday, October 12, 2012

  122. Some of my data doesn’t fit in :-( Friday, October

    12, 2012
  123. Friday, October 12, 2012

  124. Friday, October 12, 2012

  125. Friday, October 12, 2012

  126. Friday, October 12, 2012

  127. Friday, October 12, 2012

  128. Friday, October 12, 2012

  129. sum(xs) / len(xs) Friday, October 12, 2012

  130. users.aggregate, not user.aggregate Friday, October 12, 2012

  131. Friday, October 12, 2012

  132. Friday, October 12, 2012

  133. world.data, not user.data Friday, October 12, 2012

  134. Friday, October 12, 2012

  135. Friday, October 12, 2012

  136. duplication delegation separation Friday, October 12, 2012

  137. Friday, October 12, 2012

  138. Friday, October 12, 2012

  139. Everything local :-) Friday, October 12, 2012

  140. Simple :-) Friday, October 12, 2012

  141. Duplication :-( Friday, October 12, 2012

  142. Size? Friday, October 12, 2012

  143. Updates? Friday, October 12, 2012

  144. Limited data Limited mutation Friday, October 12, 2012

  145. Friday, October 12, 2012

  146. Friday, October 12, 2012

  147. No duplication :-) Friday, October 12, 2012

  148. Complex :-( Friday, October 12, 2012

  149. Sometimes local Friday, October 12, 2012

  150. Usage highly correlated with particular user Friday, October 12, 2012

  151. Example: sales data, sharded by area Friday, October 12, 2012

  152. Most queries local Friday, October 12, 2012

  153. Aggregate queries cross shards predictably Friday, October 12, 2012

  154. Tight coupling :-( Friday, October 12, 2012

  155. Friday, October 12, 2012

  156. Friday, October 12, 2012

  157. Friday, October 12, 2012

  158. Friday, October 12, 2012

  159. Friday, October 12, 2012

  160. Querying over all data is hard :-( Friday, October 12,

    2012
  161. Querying across stores is hard :-( Friday, October 12, 2012

  162. Yes. Friday, October 12, 2012

  163. Data analytics & Transactions Friday, October 12, 2012

  164. Data analytics Friday, October 12, 2012

  165. Most datasets Separation Friday, October 12, 2012

  166. Friday, October 12, 2012

  167. Big Data Specialized tools Friday, October 12, 2012

  168. Friday, October 12, 2012

  169. Friday, October 12, 2012

  170. Missing sweet spot? Friday, October 12, 2012

  171. Maybe. Friday, October 12, 2012

  172. Transactions Friday, October 12, 2012

  173. Cross-database transactions do exist Friday, October 12, 2012

  174. ATTACH DATABASE Friday, October 12, 2012

  175. dblink Friday, October 12, 2012

  176. Do it in Python! Friday, October 12, 2012

  177. Paxos Friday, October 12, 2012

  178. Friday, October 12, 2012

  179. Friday, October 12, 2012

  180. No existing tools or frameworks :-( Friday, October 12, 2012

  181. Friday, October 12, 2012

  182. Friday, October 12, 2012

  183. Friday, October 12, 2012

  184. Friday, October 12, 2012

  185. Be the change you want to see Friday, October 12,

    2012
  186. Divmod Mantissa Friday, October 12, 2012

  187. exponent, maxims Friday, October 12, 2012

  188. Friday, October 12, 2012