EuRuKo 2017 - Data driven production apps

Ecf559571ebd26f6600a5a1881090e1b?s=47 Sai Warang
September 29, 2017

EuRuKo 2017 - Data driven production apps

A talk geared towards making a legacy Rails app smart and setting up the pillars for future success

Ecf559571ebd26f6600a5a1881090e1b?s=128

Sai Warang

September 29, 2017
Tweet

Transcript

  1. 2.
  2. 5.

    Turning back time Platform for merchants to sell online Fund

    merchants who are showing promise to help them do better! Use data we already have to predict how merchants will do in the future
  3. 6.

    The Setup Rails app that allows merchants to set up

    an online store, and that can process orders Following standard Rails design patterns to set up your application
  4. 8.

    Model Ruby class that encapsulates behaviour Has a persistence layer

    backed by a relational database Lastly, has historical data
  5. 9.
  6. 10.

    Controller Wrangle out the behaviour and data that lives in

    models Serve different consumers with this model data and logic
  7. 11.

    View Templates that describe how to present data Doesn’t know

    where the data comes from that gets filled into the views
  8. 14.

    Paradigm shifts Having an app that works well as the

    user expects is not enough Need to be able to predict sane choices for the user and automate repetitive actions in your apps Recommendations
  9. 15.

    .. specifically, we want to Predict which merchant is risky

    to fund and which merchant is not risky to fund Predict the amount that a merchant will reasonably be able to pay back Confidently automate underwriting
  10. 24.

  11. 25.

    Caveats Aggregations ± Cron background jobs for more complex calculations

    Performance issues in the app as it performs complex calculations leading to CPU resources getting tied up and there will be that one moment when you start dropping traffic because of DB locks
  12. 27.

  13. 28.

    More caveats Reporting views and dashboards Can your data be

    trusted? Does your app have test data in production? (Here’s looking at you past Sai )
  14. 35.

    Extract Transform Load ETL all your tables into a warehouse

    Timely snapshots of production database Sent over to an external system Clean your data
  15. 36.

    ActiveRecord::Base.connection.tables.each do |table| klass = table.classify if Time.now > @checkpoint[klass]

    warehouse_extract(klass.constantize.where(updated_at: @checkpoint[klass]..Time.now)) end end
  16. 37.

    ActiveRecord::Base.connection.tables.each do |table| klass = table.classify if Time.now > @checkpoint[klass]

    warehouse_extract(klass.constantize.where(updated_at: @checkpoint[klass]..Time.now, test: false, secret: false)) end end
  17. 38.

    Benefits of warehousing No more constraint of production environment Opens

    up the possibility of using many Machine Learning libraries No stress on production database
  18. 41.

    Main idea Not entity relationships Doesn’t follow the schema of

    production database tables Modelled around business domain
  19. 45.
  20. 46.
  21. 50.

    HTTP REST APIs Data warehouse tooling that generates insights HTTP

    REST API to deliver those insights to your app
  22. 51.

    Insight models Easily plug view layer Loose, can be freely

    modelled (JSON blobs, some fields) JOIN tables in normal flow of application logic Native Rails validations (data integrity) when inputted back into Rails
  23. 52.

    Data quality Complex analysis Real time predictions Rails app ⚠

    ETL + Dimensional Modelling ✅ ✅ Web tooling + New models ✅ ✅
  24. 57.

    Resiliency questionnaire How are we keeping APIs up to date?

    What happens when your ETL cron goes down? What happens when your dimension schemas become outdated? What is the end user experience?
  25. 59.