Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scale-Independent Python

Scale-Independent Python

Have you ever written an application only to find yourself rewriting it later so it can scale? Many of us have. In this talk, you’ll learn scale-independent computing; what it is and how it can help you more easily scale your applications.

Scaling applications is hard. Our typical approaches all come with tradeoffs that we wish we didn’t have to make. There is a better way, scale-independent computing. Based on ideas from Pat Helland’s paper “Life Beyond Distributed Transactions,” scale-independent computing allows application developers to write their code without regard to scale and then run it on a scale-aware platform. The idea is relatively simple, the code you write should be able to run across any number of Python processes without you having to change any of your code. During this talk, you’ll learn the core ideas of scale-independence and see an example scale-independent application in action.

Sean T Allen

October 06, 2018
Tweet

More Decks by Sean T Allen

Other Decks in Technology

Transcript

  1. SCALE-INDEPENDENT PYTHON
    HOW TO SCALE YOUR PYTHON APPLICATION WITHOUT ANY CODE CHANGES

    View full-size slide

  2. SEAN T. ALLEN
    VP OF ENGINEERING AT WALLAROO LABS
    AUTHOR OF “STORM APPLIED”
    @SEANTALLEN
    @WALLAROOLABS

    View full-size slide

  3. THIS TALK IS ABOUT
    “SCALING”

    View full-size slide

  4. “SCALING”
    IS ABOUT REACTING TO CHANGE

    View full-size slide

  5. LET’S TALK ABOUT REACTING
    TO CHANGES IN DATA

    View full-size slide

  6. LET’S TALK ABOUT CHANGES IN
    APPLICATION STATE

    View full-size slide

  7. LET’S TALK ABOUT
    A VERY SPECIFIC THING

    View full-size slide

  8. LET’S TALK ABOUT HANDLING
    “MORE THINGS”

    View full-size slide

  9. MORE THINGS?

    View full-size slide

  10. MORE
    CUSTOMERS

    View full-size slide

  11. THAT’S THE “SCALING”
    WE’RE TALKING ABOUT

    View full-size slide

  12. WE’RE TALKING ABOUT
    HORIZONTAL SCALING

    View full-size slide

  13. WE’RE TALKING ABOUT PAT HELLAND’S
    LIFE BEYOND DISTRIBUTED TRANSACTIONS

    View full-size slide

  14. SEVERAL BIG IDEAS

    View full-size slide

  15. LET’S TALK ABOUT ONE BIG IDEA

    View full-size slide

  16. TWO-LAYER ARCHITECTURE

    View full-size slide

  17. SCALE-INDEPENDENT LAYER

    View full-size slide

  18. OUR BUSINESS LOGIC LIVES IN THE
    SCALE-INDEPENDENT LAYER

    View full-size slide

  19. SCALE-AWARE LAYER

    View full-size slide

  20. MESSAGE ROUTING

    View full-size slide

  21. GUARANTEED MESSAGE DELIVERY

    View full-size slide

  22. FAULT TOLERANCE

    View full-size slide

  23. AND AN API TO CONNECT
    YOUR SCALE-INDEPENDENT CODE
    TO THE SCALE-AWARE LAYER

    View full-size slide

  24. AND THAT IS WHAT WE’VE BUILT AT
    WALLAROO LABS

    View full-size slide

  25. WALLAROO
    SCALE-INDEPENDENT COMPUTING
    FOR PYTHON

    View full-size slide

  26. STATEFUL DATA PROCESSING ENGINE

    View full-size slide

  27. SCALE-INDEPENDENT API

    View full-size slide

  28. FLOW OF DATA

    View full-size slide

  29. TRANSFORMATIONS OF DATA

    View full-size slide

  30. ACCUMULATED APPLICATION STATE

    View full-size slide

  31. APPLICATION DEFINITION
    FLOW OF DATA FROM SOURCE TO SINK

    View full-size slide

  32. TWO DATA PIPELINES
    ORDERS

    View full-size slide

  33. TWO DATA PIPELINES
    MARKET DATA

    View full-size slide

  34. DEFINE OUR SOURCES
    1 PER PIPELINE

    View full-size slide

  35. DEFINE OUR OPERATIONS
    IN THIS EXAMPLE, THERE’S 1 PER PIPELINE

    View full-size slide

  36. DEFINE OUR OPERATIONS
    CHECK ORDER AGAINST SYMBOL DATA

    View full-size slide

  37. DEFINE OUR OPERATIONS
    UPDATE SYMBOL DATA WITH LATEST MARKET DATA

    View full-size slide

  38. DEFINE OUR SINKS
    1 PER PIPELINE

    View full-size slide

  39. DEFINE OUR SINKS
    ORDERS PIPELINE MIGHT HAVE OUTPUT

    View full-size slide

  40. DEFINE OUR SINKS
    MARKET DATA ONLY UPDATES SYMBOL DATA- NO OUTPUT

    View full-size slide

  41. SCALE-INDEPENDENT
    ONLY FLOW OF DATA AND OPERATIONS

    View full-size slide

  42. 100% SCALE-INDEPENDENT
    NOTHING ABOUT
    THE NUMBER OF PYTHON PROCESSES

    View full-size slide

  43. RUN WITH
    ANY NUMBER OF PYTHON PROCESSES

    View full-size slide

  44. WITHOUT CHANGING CODE

    View full-size slide

  45. SUCH IS THE GLORY OF
    SCALE-INDEPENDENCE

    View full-size slide

  46. THERE’S MORE
    BUT WE ARE PROBABLY CLOSE TO BEING
    OUT OF TIME

    View full-size slide

  47. QUICKLY,
    A FEW MORE WALLAROO FEATURES

    View full-size slide

  48. PLAIN OLD PYTHON FUNCTIONS

    View full-size slide

  49. PLAIN OLD PYTHON DATA OBJECTS

    View full-size slide

  50. USE ANY PYTHON LIBRARY

    View full-size slide

  51. LEARN MORE
    GITHUB.COM/SEANTALLEN/
    SCALE-INDEPENDENT-PYTHON

    View full-size slide