Scale-Independent Python

Scale-Independent Python

Have you ever written an application only to find yourself rewriting it later so it can scale? Many of us have. In this talk, you’ll learn scale-independent computing; what it is and how it can help you more easily scale your applications.

Scaling applications is hard. Our typical approaches all come with tradeoffs that we wish we didn’t have to make. There is a better way, scale-independent computing. Based on ideas from Pat Helland’s paper “Life Beyond Distributed Transactions,” scale-independent computing allows application developers to write their code without regard to scale and then run it on a scale-aware platform. The idea is relatively simple, the code you write should be able to run across any number of Python processes without you having to change any of your code. During this talk, you’ll learn the core ideas of scale-independence and see an example scale-independent application in action.

3c53e91d2a6ceb1b7f202d709f638b1b?s=128

Sean T Allen

October 06, 2018
Tweet

Transcript

  1. SCALE-INDEPENDENT PYTHON HOW TO SCALE YOUR PYTHON APPLICATION WITHOUT ANY

    CODE CHANGES
  2. SEAN T. ALLEN VP OF ENGINEERING AT WALLAROO LABS AUTHOR

    OF “STORM APPLIED” @SEANTALLEN @WALLAROOLABS
  3. THIS TALK IS ABOUT “SCALING”

  4. “SCALING” IS ABOUT REACTING TO CHANGE

  5. LET’S TALK ABOUT REACTING TO CHANGES IN DATA

  6. LET’S TALK ABOUT CHANGES IN APPLICATION STATE

  7. LET’S TALK ABOUT A VERY SPECIFIC THING

  8. LET’S TALK ABOUT HANDLING “MORE THINGS”

  9. MORE THINGS?

  10. MORE CUSTOMERS

  11. MORE ORDERS

  12. MORE VENDORS

  13. MORE THINGS

  14. THAT’S THE “SCALING” WE’RE TALKING ABOUT

  15. WE’RE TALKING ABOUT HORIZONTAL SCALING

  16. WE’RE TALKING ABOUT PAT HELLAND’S LIFE BEYOND DISTRIBUTED TRANSACTIONS

  17. SEVERAL BIG IDEAS

  18. LET’S TALK ABOUT ONE BIG IDEA

  19. TWO-LAYER ARCHITECTURE

  20. SCALE-INDEPENDENT LAYER

  21. OUR BUSINESS LOGIC LIVES IN THE SCALE-INDEPENDENT LAYER

  22. SCALE-AWARE LAYER

  23. CLUSTERING

  24. MESSAGE ROUTING

  25. GUARANTEED MESSAGE DELIVERY

  26. FAULT TOLERANCE

  27. AND AN API TO CONNECT YOUR SCALE-INDEPENDENT CODE TO THE

    SCALE-AWARE LAYER
  28. AND THAT IS WHAT WE’VE BUILT AT WALLAROO LABS

  29. WALLAROO SCALE-INDEPENDENT COMPUTING FOR PYTHON

  30. STATEFUL DATA PROCESSING ENGINE

  31. SCALE-INDEPENDENT API

  32. FLOW OF DATA

  33. TRANSFORMATIONS OF DATA

  34. ACCUMULATED APPLICATION STATE

  35. APPLICATION DEFINITION FLOW OF DATA FROM SOURCE TO SINK

  36. TWO DATA PIPELINES ORDERS

  37. TWO DATA PIPELINES MARKET DATA

  38. DEFINE OUR SOURCES 1 PER PIPELINE

  39. DEFINE OUR OPERATIONS IN THIS EXAMPLE, THERE’S 1 PER PIPELINE

  40. DEFINE OUR OPERATIONS CHECK ORDER AGAINST SYMBOL DATA

  41. DEFINE OUR OPERATIONS UPDATE SYMBOL DATA WITH LATEST MARKET DATA

  42. DEFINE OUR SINKS 1 PER PIPELINE

  43. DEFINE OUR SINKS ORDERS PIPELINE MIGHT HAVE OUTPUT

  44. DEFINE OUR SINKS MARKET DATA ONLY UPDATES SYMBOL DATA- NO

    OUTPUT
  45. SCALE-INDEPENDENT ONLY FLOW OF DATA AND OPERATIONS

  46. 100% SCALE-INDEPENDENT NOTHING ABOUT THE NUMBER OF PYTHON PROCESSES

  47. RUN WITH ANY NUMBER OF PYTHON PROCESSES

  48. WITHOUT CHANGING CODE

  49. SUCH IS THE GLORY OF SCALE-INDEPENDENCE

  50. THERE’S MORE BUT WE ARE PROBABLY CLOSE TO BEING OUT

    OF TIME
  51. QUICKLY, A FEW MORE WALLAROO FEATURES

  52. PLAIN OLD PYTHON FUNCTIONS

  53. PLAIN OLD PYTHON DATA OBJECTS

  54. USE ANY PYTHON LIBRARY

  55. LEARN MORE GITHUB.COM/SEANTALLEN/ SCALE-INDEPENDENT-PYTHON