$30 off During Our Annual Pro Sale. View Details »

Scale Agnostic Python

Scale Agnostic Python

Presented to the NYC Python meetup August 17, 2017

Sean T Allen

August 17, 2017
Tweet

More Decks by Sean T Allen

Other Decks in Programming

Transcript

  1. SCALE AGNOSTIC PYTHON
    HOW TO SCALE YOUR PYTHON APPLICATION WITHOUT ANY CODE CHANGES*

    View Slide

  2. SEAN T. ALLEN
    VP OF ENGINEERING AT SENDENCE
    AUTHOR OF “STORM APPLIED”
    @SEANTALLEN

    View Slide

  3. SCALE AGNOSTIC COMPUTING

    View Slide

  4. THE PROBLEM
    THE LITTLE APPLICATION
    THAT COULD…
    UNTIL IT COULDN’T.

    View Slide

  5. Scaling an application from “little data” to “a little
    more data” to “big data” is hard.

    View Slide

  6. OPTIONS

    View Slide

  7. Rewrite the application
    Rewrite to scale across
    multiple processes

    Example: “multiprocessing”
    Python library

    View Slide

  8. Run more versions
    Smart loader to load data in.
    Perhaps via a job queue like
    Celery.

    View Slide

  9. Go scale agnostic
    Write you application against
    a scale agnostic API that runs
    on a scale aware platform

    View Slide

  10. SENDENCE
    WALLAROO
    SCALE AGNOSTIC
    COMPUTING FOR PYTHON

    View Slide

  11. Sendence Wallaroo
    Simple Scale Agnostic API

    Managed in-memory state

    Guaranteed Message Processing

    View Slide

  12. “We handle the hard infrastructure plumbing so
    you don’t have to.”
    – The Wallaroo Team

    View Slide

  13. LEARN MORE…
    sendence.com
    Sendence Engineering on Twitter: @SendenceEng
    Pat Helland’s “Life Beyond Distributed Transactions”

    View Slide