Upgrade to Pro — share decks privately, control downloads, hide ads and more …

POLO: Working With Real World Data In Development

POLO: Working With Real World Data In Development

Ivayr Farah Netto

September 25, 2015
Tweet

More Decks by Ivayr Farah Netto

Other Decks in Programming

Transcript

  1. POLO
    WORKING WITH REAL WORLD DATA IN
    DEVELOPMENT
    @nettofarah
    1 / 39

    View Slide

  2. nettofarah
    github.com/nettofarah
    @nettofarah
    http://bit.ly/polo-rb
    2 / 39

    View Slide

  3. Real World Production Data in
    Development
    3 / 39

    View Slide

  4. Because Managing Data is hard
    Especially across Different environments
    4 / 39

    View Slide

  5. Data in Rails.env.dev
    is...
    Ugly, Incomplete, Weird and Biased
    5 / 39

    View Slide

  6. Some Consequences of Inconsistent
    Data
    6 / 39

    View Slide

  7. !"
    !", #
    # misalignment, form
    validation, special characters $
    $
    7 / 39

    View Slide

  8. The Worst Consequence
    8 / 39

    View Slide

  9. Rails.env.dev
    will look %
    9 / 39

    View Slide

  10. But how do people solve these
    problems?
    10 / 39

    View Slide

  11. We can test it in PRODUCTION
    11 / 39

    View Slide

  12. Rails.env.staging ?
    12 / 39

    View Slide

  13. Custom db/migrations
    for data
    13 / 39

    View Slide

  14. rake db:seed
    14 / 39

    View Slide

  15. CSVs and Spreadsheets
    15 / 39

    View Slide

  16. 16 / 39

    View Slide

  17. Sample Database Snapshots
    17 / 39

    View Slide

  18. Your good ol'
    ActiveRecord::Associations
    to
    .sql
    18 / 39

    View Slide

  19. POLO lets you turn this...
    19 / 39

    View Slide

  20. class Chef < ActiveRecord::Base
    has_many :recipes
    has_many :ingredients, through: :recipes
    end
    class Recipe < ActiveRecord::Base
    has_many :recipes_ingredients
    has_many :ingredients, through: :recipes_ingredients
    end
    class Ingredient < ActiveRecord::Base
    end
    class RecipesIngredient < ActiveRecord::Base
    belongs_to :recipe
    belongs_to :ingredient
    end
    20 / 39

    View Slide

  21. Into this...
    21 / 39

    View Slide

  22. inserts = Polo.explore(Chef, 1)
    # Chef -> ActiveRecord::Base object
    # 1 -> Database ID. (Chef with ID 1)
    INSERT INTO `chefs` (`id`, `name`)
    VALUES (1, 'Netto')
    22 / 39

    View Slide

  23. It works with associations too
    23 / 39

    View Slide

  24. inserts = Polo.explore(Chef, 1, :recipes)
    # :recipes ->
    # ActiveRecord::Associations::HasManyAssociation
    #
    # a Chef has many Recipes
    INSERT INTO `chefs` (`id`, `name`)
    VALUES (1, 'Netto')
    INSERT INTO `recipes`
    (`id`, `title`, `num_steps`, `chef_id`)
    VALUES
    (1, 'Turkey Sandwich', NULL, 1)
    INSERT INTO `recipes`
    (`id`, `title`, `num_steps`, `chef_id`)
    VALUES
    (2, 'Cheese Burger', NULL, 1)
    24 / 39

    View Slide

  25. It also works with nested
    associations
    25 / 39

    View Slide

  26. inserts = Polo.explore(Chef, 1, { :recipes => :ingredients })
    # { :recipes => :ingredients } ->
    # load every recipe and ingredientes
    ...
    INSERT INTO `recipes` (`id`, `title`, `num_steps`, `chef_id`)
    VALUES (1, 'Turkey Sandwich', NULL, 1)
    INSERT INTO `recipes` (`id`, `title`, `num_steps`, `chef_id`)
    VALUES (2, 'Cheese Burger', NULL, 1)
    INSERT INTO `recipes_ingredients`
    (`id`, `recipe_id`, `ingredient_id`)
    VALUES (1, 1, 1)
    INSERT INTO `recipes_ingredients`
    (`id`, `recipe_id`, `ingredient_id`)
    VALUES (2, 1, 2)
    ...
    INSERT INTO `ingredients` (`id`, `name`, `quantity`)
    VALUES (1, 'Turkey', 'a lot')
    INSERT INTO `ingredients` (`id`, `name`, `quantity`)
    VALUES (2, 'Cheese', '1 slice')
    ...
    26 / 39

    View Slide

  27. &
    &
    Show me the magic!
    27 / 39

    View Slide

  28. Collect all the objects
    lib/polo/collector.rb#L16
    28 / 39

    View Slide

  29. asn = ActiveSupport::Notifications
    # Extracted so this can fit in a slide
    asn.subscribed(collector, 'sql.active_record') do
    # Set up ActiveRecord::Preloader
    base_finder = @base_class.
    includes(@dependency_tree)
    .where(id: @id)
    # Store SELECTS in some global storage
    collect_sql(@base_class, base_finder.to_sql)
    end
    29 / 39

    View Slide

  30. '
    '
    Transform them to SQL
    lib/polo/sql_translator.rb#51
    30 / 39

    View Slide

  31. attributes = record.attributes
    keys = attributes.keys.map do |key|
    "`#{key}`"
    end
    values = attributes.map do |key, value|
    column = record.column_for_attribute(key)
    attribute = cast_attribute(record, column, value)
    connection.quote(attribute)
    end
    joined_keys = keys.join(', ')
    joined_values = values.join(', ')
    table_name = record.class.table_name
    "INSERT INTO `#{table_name}`" +
    " (#{joined_keys}) VALUES (#{joined_values})"
    31 / 39

    View Slide

  32. PROFIT! (
    (
    32 / 39

    View Slide

  33. I need your help! )
    )
    33 / 39

    View Slide

  34. on Github
    http://github.com/IFTTT/polo
    34 / 39

    View Slide

  35. Pull Requests *
    *
    35 / 39

    View Slide

  36. PostgreSQL, SQLite and Oracle
    support
    (advanced features only, basic usage still
    works)
    36 / 39

    View Slide

  37. ActiveRecord < 3.2 support
    37 / 39

    View Slide

  38. Check out some other really cool
    Open Source projects
    http://ifttt.github.io
    38 / 39

    View Slide

  39. Questions?
    39 / 39

    View Slide