Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scale-Independent Python

Scale-Independent Python

Have you ever written an application only to find yourself rewriting it later so it can scale? Many of us have. In this talk, you’ll learn scale-independent computing; what it is and how it can help you more easily scale your applications.

Scaling applications is hard. Our typical approaches all come with tradeoffs that we wish we didn’t have to make. There is a better way, scale-independent computing. Based on ideas from Pat Helland’s paper “Life Beyond Distributed Transactions,” scale-independent computing allows application developers to write their code without regard to scale and then run it on a scale-aware platform. The idea is relatively simple, the code you write should be able to run across any number of Python processes without you having to change any of your code. During this talk, you’ll learn the core ideas of scale-independence and see an example scale-independent application in action.

Sean T Allen

October 06, 2018
Tweet

More Decks by Sean T Allen

Other Decks in Technology

Transcript

  1. SCALE-INDEPENDENT PYTHON
    HOW TO SCALE YOUR PYTHON APPLICATION WITHOUT ANY CODE CHANGES

    View Slide

  2. SEAN T. ALLEN
    VP OF ENGINEERING AT WALLAROO LABS
    AUTHOR OF “STORM APPLIED”
    @SEANTALLEN
    @WALLAROOLABS

    View Slide

  3. THIS TALK IS ABOUT
    “SCALING”

    View Slide

  4. “SCALING”
    IS ABOUT REACTING TO CHANGE

    View Slide

  5. LET’S TALK ABOUT REACTING
    TO CHANGES IN DATA

    View Slide

  6. LET’S TALK ABOUT CHANGES IN
    APPLICATION STATE

    View Slide

  7. LET’S TALK ABOUT
    A VERY SPECIFIC THING

    View Slide

  8. LET’S TALK ABOUT HANDLING
    “MORE THINGS”

    View Slide

  9. MORE THINGS?

    View Slide

  10. MORE
    CUSTOMERS

    View Slide

  11. MORE
    ORDERS

    View Slide

  12. MORE
    VENDORS

    View Slide

  13. MORE
    THINGS

    View Slide

  14. THAT’S THE “SCALING”
    WE’RE TALKING ABOUT

    View Slide

  15. WE’RE TALKING ABOUT
    HORIZONTAL SCALING

    View Slide

  16. WE’RE TALKING ABOUT PAT HELLAND’S
    LIFE BEYOND DISTRIBUTED TRANSACTIONS

    View Slide

  17. SEVERAL BIG IDEAS

    View Slide

  18. LET’S TALK ABOUT ONE BIG IDEA

    View Slide

  19. TWO-LAYER ARCHITECTURE

    View Slide

  20. SCALE-INDEPENDENT LAYER

    View Slide

  21. OUR BUSINESS LOGIC LIVES IN THE
    SCALE-INDEPENDENT LAYER

    View Slide

  22. SCALE-AWARE LAYER

    View Slide

  23. CLUSTERING

    View Slide

  24. MESSAGE ROUTING

    View Slide

  25. GUARANTEED MESSAGE DELIVERY

    View Slide

  26. FAULT TOLERANCE

    View Slide

  27. AND AN API TO CONNECT
    YOUR SCALE-INDEPENDENT CODE
    TO THE SCALE-AWARE LAYER

    View Slide

  28. AND THAT IS WHAT WE’VE BUILT AT
    WALLAROO LABS

    View Slide

  29. WALLAROO
    SCALE-INDEPENDENT COMPUTING
    FOR PYTHON

    View Slide

  30. STATEFUL DATA PROCESSING ENGINE

    View Slide

  31. SCALE-INDEPENDENT API

    View Slide

  32. FLOW OF DATA

    View Slide

  33. TRANSFORMATIONS OF DATA

    View Slide

  34. ACCUMULATED APPLICATION STATE

    View Slide

  35. APPLICATION DEFINITION
    FLOW OF DATA FROM SOURCE TO SINK

    View Slide

  36. TWO DATA PIPELINES
    ORDERS

    View Slide

  37. TWO DATA PIPELINES
    MARKET DATA

    View Slide

  38. DEFINE OUR SOURCES
    1 PER PIPELINE

    View Slide

  39. DEFINE OUR OPERATIONS
    IN THIS EXAMPLE, THERE’S 1 PER PIPELINE

    View Slide

  40. DEFINE OUR OPERATIONS
    CHECK ORDER AGAINST SYMBOL DATA

    View Slide

  41. DEFINE OUR OPERATIONS
    UPDATE SYMBOL DATA WITH LATEST MARKET DATA

    View Slide

  42. DEFINE OUR SINKS
    1 PER PIPELINE

    View Slide

  43. DEFINE OUR SINKS
    ORDERS PIPELINE MIGHT HAVE OUTPUT

    View Slide

  44. DEFINE OUR SINKS
    MARKET DATA ONLY UPDATES SYMBOL DATA- NO OUTPUT

    View Slide

  45. SCALE-INDEPENDENT
    ONLY FLOW OF DATA AND OPERATIONS

    View Slide

  46. 100% SCALE-INDEPENDENT
    NOTHING ABOUT
    THE NUMBER OF PYTHON PROCESSES

    View Slide

  47. RUN WITH
    ANY NUMBER OF PYTHON PROCESSES

    View Slide

  48. WITHOUT CHANGING CODE

    View Slide

  49. SUCH IS THE GLORY OF
    SCALE-INDEPENDENCE

    View Slide

  50. THERE’S MORE
    BUT WE ARE PROBABLY CLOSE TO BEING
    OUT OF TIME

    View Slide

  51. QUICKLY,
    A FEW MORE WALLAROO FEATURES

    View Slide

  52. PLAIN OLD PYTHON FUNCTIONS

    View Slide

  53. PLAIN OLD PYTHON DATA OBJECTS

    View Slide

  54. USE ANY PYTHON LIBRARY

    View Slide

  55. LEARN MORE
    GITHUB.COM/SEANTALLEN/
    SCALE-INDEPENDENT-PYTHON

    View Slide