POLO: Working With Real World Data In Development

Transcript

1. POLO
   WORKING WITH REAL WORLD DATA IN
   DEVELOPMENT
   @nettofarah
   1 / 39
   

   View Slide

2. nettofarah
   github.com/nettofarah
   @nettofarah
   http://bit.ly/polo-rb
   2 / 39
   

   View Slide

3. Real World Production Data in
   Development
   3 / 39
   

   View Slide

4. Because Managing Data is hard
   Especially across Different environments
   4 / 39
   

   View Slide

5. Data in Rails.env.dev
   is...
   Ugly, Incomplete, Weird and Biased
   5 / 39
   

   View Slide

6. Some Consequences of Inconsistent
   Data
   6 / 39
   

   View Slide

7. !"
   !", #
   # misalignment, form
   validation, special characters $
   $
   7 / 39
   

   View Slide

8. The Worst Consequence
   8 / 39
   

   View Slide

9. Rails.env.dev
   will look %
   9 / 39
   

   View Slide

10. But how do people solve these
    problems?
    10 / 39
    

    View Slide

11. We can test it in PRODUCTION
    11 / 39
    

    View Slide

12. Rails.env.staging ?
    12 / 39
    

    View Slide

13. Custom db/migrations
    for data
    13 / 39
    

    View Slide

14. rake db:seed
    14 / 39
    

    View Slide

15. CSVs and Spreadsheets
    15 / 39
    

    View Slide

16. 16 / 39
    

    View Slide

17. Sample Database Snapshots
    17 / 39
    

    View Slide

18. Your good ol'
    ActiveRecord::Associations
    to
    .sql
    18 / 39
    

    View Slide

19. POLO lets you turn this...
    19 / 39
    

    View Slide

20. class Chef < ActiveRecord::Base
    has_many :recipes
    has_many :ingredients, through: :recipes
    end
    class Recipe < ActiveRecord::Base
    has_many :recipes_ingredients
    has_many :ingredients, through: :recipes_ingredients
    end
    class Ingredient < ActiveRecord::Base
    end
    class RecipesIngredient < ActiveRecord::Base
    belongs_to :recipe
    belongs_to :ingredient
    end
    20 / 39
    

    View Slide

21. Into this...
    21 / 39
    

    View Slide

22. inserts = Polo.explore(Chef, 1)
    # Chef -> ActiveRecord::Base object
    # 1 -> Database ID. (Chef with ID 1)
    INSERT INTO `chefs` (`id`, `name`)
    VALUES (1, 'Netto')
    22 / 39
    

    View Slide

23. It works with associations too
    23 / 39
    

    View Slide

24. inserts = Polo.explore(Chef, 1, :recipes)
    # :recipes ->
    # ActiveRecord::Associations::HasManyAssociation
    #
    # a Chef has many Recipes
    INSERT INTO `chefs` (`id`, `name`)
    VALUES (1, 'Netto')
    INSERT INTO `recipes`
    (`id`, `title`, `num_steps`, `chef_id`)
    VALUES
    (1, 'Turkey Sandwich', NULL, 1)
    INSERT INTO `recipes`
    (`id`, `title`, `num_steps`, `chef_id`)
    VALUES
    (2, 'Cheese Burger', NULL, 1)
    24 / 39
    

    View Slide

25. It also works with nested
    associations
    25 / 39
    

    View Slide

26. inserts = Polo.explore(Chef, 1, { :recipes => :ingredients })
    # { :recipes => :ingredients } ->
    # load every recipe and ingredientes
    ...
    INSERT INTO `recipes` (`id`, `title`, `num_steps`, `chef_id`)
    VALUES (1, 'Turkey Sandwich', NULL, 1)
    INSERT INTO `recipes` (`id`, `title`, `num_steps`, `chef_id`)
    VALUES (2, 'Cheese Burger', NULL, 1)
    INSERT INTO `recipes_ingredients`
    (`id`, `recipe_id`, `ingredient_id`)
    VALUES (1, 1, 1)
    INSERT INTO `recipes_ingredients`
    (`id`, `recipe_id`, `ingredient_id`)
    VALUES (2, 1, 2)
    ...
    INSERT INTO `ingredients` (`id`, `name`, `quantity`)
    VALUES (1, 'Turkey', 'a lot')
    INSERT INTO `ingredients` (`id`, `name`, `quantity`)
    VALUES (2, 'Cheese', '1 slice')
    ...
    26 / 39
    

    View Slide

27. &
    &
    Show me the magic!
    27 / 39
    

    View Slide

28. Collect all the objects
    lib/polo/collector.rb#L16
    28 / 39
    

    View Slide

29. asn = ActiveSupport::Notifications
    # Extracted so this can fit in a slide
    asn.subscribed(collector, 'sql.active_record') do
    # Set up ActiveRecord::Preloader
    base_finder = @base_class.
    includes(@dependency_tree)
    .where(id: @id)
    # Store SELECTS in some global storage
    collect_sql(@base_class, base_finder.to_sql)
    end
    29 / 39
    

    View Slide

30. '
    '
    Transform them to SQL
    lib/polo/sql_translator.rb#51
    30 / 39
    

    View Slide

31. attributes = record.attributes
    keys = attributes.keys.map do |key|
    "`#{key}`"
    end
    values = attributes.map do |key, value|
    column = record.column_for_attribute(key)
    attribute = cast_attribute(record, column, value)
    connection.quote(attribute)
    end
    joined_keys = keys.join(', ')
    joined_values = values.join(', ')
    table_name = record.class.table_name
    "INSERT INTO `#{table_name}`" +
    " (#{joined_keys}) VALUES (#{joined_values})"
    31 / 39
    

    View Slide

32. PROFIT! (
    (
    32 / 39
    

    View Slide

33. I need your help! )
    )
    33 / 39
    

    View Slide

34. on Github
    http://github.com/IFTTT/polo
    34 / 39
    

    View Slide

35. Pull Requests *
    *
    35 / 39
    

    View Slide

36. PostgreSQL, SQLite and Oracle
    support
    (advanced features only, basic usage still
    works)
    36 / 39
    

    View Slide

37. ActiveRecord < 3.2 support
    37 / 39
    

    View Slide

38. Check out some other really cool
    Open Source projects
    http://ifttt.github.io
    38 / 39
    

    View Slide

39. Questions?
    39 / 39
    

    View Slide