Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Developer happiness with DB Charmer

Lance Gleason
February 28, 2014

Developer happiness with DB Charmer

Presentation by Lance Gleason from Polyglot Programming Inc at the Atlanta Ruby Meetup and Jozi.rb in Johannesburg South Africa.

http://www.polyglotprogramminginc.com
http://www.polyglotprogramminginc.com/charming-large-databases-with-octopuses/

Lance Gleason

February 28, 2014
Tweet

More Decks by Lance Gleason

Other Decks in Programming

Transcript

  1. Introductions

    View full-size slide

  2. Twitter @lgleasain

    Github lgleasain

    !
    www.lancegleason.com

    www.polyglotprogrammincinc.com

    [email protected]

    !

    View full-size slide

  3. http://www.polyglotprogramminginc.com/purr-
    programming-2-0/

    View full-size slide

  4. www.purrprogramming.com

    View full-size slide

  5. Supporting More
    Users

    View full-size slide

  6. Optimized Queries

    View full-size slide

  7. How do you
    support this
    with
    ActiveRecord?

    View full-size slide

  8. production:  
       blah:  
           adapter:  mysql  
           username:  blah  
           host:  blah.local  
           database:  blah  
    !
       foo:  
           adapter:  mysql  
           username:  foo  
           host:  foo.local  
           database:  foo  

    View full-size slide

  9. class  Foo  <  ActiveRecord::Model;  end  
    !
    Foo.switch_connection_to(:blah)  
    Foo.switch_connection_to('foo')  
    Foo.switch_connection_to(Bar)  
    Foo.switch_connection_to(Baz.connectio
    n)  
    Foo.switch_connection_to(nil)  

    View full-size slide

  10. class  MultiDbTest  <  ActiveRecord::Migration  
         db_magic  :connection  =>  :second_db  
    !
         def  self.up  
             create_table  :test_table,  :force  =>  true  do  |t|
                 t.string  :test_string  
                 t.timestamps  
             end  
         end  
    !
         def  self.down  
             drop_table  :test_table  
         end  
     end  

    View full-size slide

  11. class  MultiDbTest  <  ActiveRecord::Migration  
       def  self.up  
           on_db  :second_db  do  
               create_table  :test_table,  :force  =>  true  do  
    |t|  
                   t.string  :test_string  
                   t.timestamps  
               end  
           end  
       end  
    !
       def  self.down  
           on_db  :second_db  {  drop_table  :test_table  }  
       end  
    end  

    View full-size slide

  12. class  MultiDbTest  <  ActiveRecord::Migration  
       db_magic  :connections  =>  [:second_db,  :default]  
    !
       def  self.up  
           create_table  :test_table,  :force  =>  true  do  |t|  
               t.string  :test_string  
               t.timestamps  
           end  
       end  
    !
       def  self.down  
           drop_table  :test_table  
       end  
    end  

    View full-size slide

  13. Auto Switch All
    Connections To Slaves

    View full-size slide

  14. class  Foo  <  ActiveRecord::Base  
       db_magic  :slave  =>  :slave01  
    end  
    !
    class  Bar  <  ActiveRecord::Base  
       db_magic  :slaves  =>  
    [  :slave01,  :slave02  ]  
    end  

    View full-size slide

  15. Default Master Slave
    Class Connections

    View full-size slide

  16. class  Foo  <  ActiveRecord::Base  
       db_magic  :connection  =>  :foo  
    end  
    class  Bar  <  ActiveRecord::Base  
       db_magic  :connection  =>  :bar,  :slave  
    =>  :bar_slave  
    end  

    View full-size slide

  17. Per Query Connection
    Management

    View full-size slide

  18. class  Foo  <  ActiveRecord::Model;  end  
    !
    Foo.switch_connection_to(:blah)  
    Foo.switch_connection_to('foo')  
    Foo.switch_connection_to(Bar)  
    Foo.switch_connection_to(Baz.connectio
    n)  
    Foo.switch_connection_to(nil)  

    View full-size slide

  19. User.on_master  do  
       user  =  User.find_by_login('foo')  
       user.update_attributes!(:activated  
    =>  true)  
    end  

    View full-size slide

  20. Comment.on_master.last(:limit  =>  5)  
    User.on_master.find_by_activation_code(code)  
    User.on_master.exists?(:login  =>  login,  :password  
    =>  password)  

    View full-size slide

  21. Comment.on_db(:olap).count  
    Post.on_db(:foo).find(:first)  

    View full-size slide

  22. Sharding
    • Range

    View full-size slide

  23. Sharding
    • Range
    • hash_map

    View full-size slide

  24. Sharding
    • Range
    • hash_map
    • db_block_map

    View full-size slide

  25. Sharding
    • Range
    • hash_map
    • db_block_map
    • db_block_group_map

    View full-size slide

  26. Custom Sharding Methods

    View full-size slide

  27. Is it thread safe?

    View full-size slide

  28. No transaction rollbacks

    View full-size slide

  29. •MySQL Only

    View full-size slide

  30. •MySQL Only
    •Doesn’t Support Rails 4

    View full-size slide

  31. •Supports Postgres and MySQL

    View full-size slide

  32. •Supports Postgres and MySQL
    •Includes migrations for moving
    data between shards

    View full-size slide

  33. octopus:  
       replicated:  true  
       production:  
           slave1:  
               adapter:  mysql  
               host:  localhost  
               database:  octopus_shard2  
           slave2:  
               adapter:  mysql  
               host:  localhost  
               database:  octopus_shard3  
           slave3:  
               adapter:  mysql  
               host:  localhost  
               database:  octopus_shard4  
           slave4:  
               adapter:  mysql  
               host:  localhost  
               database:  octopus_shard5  

    View full-size slide

  34. User.where(:name  =>  
    "Thiago").limit(3).using(:slave_one)  
    Octopus.using(:slave_two)  do  
       User.create(:name  =>  "Mike")  
    end  

    View full-size slide

  35. class  CreateUsersOnBothShards  <  
    ActiveRecord::Migration  
       using(:brazil,  :canada)  
    !
       def  self.up  
           User.create!(:name  =>  "Both")  
       end  
    !
       def  self.down  
           User.delete_all  
       end  
    end  

    View full-size slide

  36. class  ApplicationController  <  
    ActionController::Base  
       around_filter  :select_shard  
    !
       def  select_shard(&block)  
           Octopus.using(:brazil,  &block)  
       end  
    end  

    View full-size slide

  37. Alternatives

    View full-size slide

  38. Amazon Cloud
    Search

    View full-size slide

  39. Active Asari

    View full-size slide

  40. Twitter @lgleasain

    Github lgleasain

    !
    www.lancegleason.com

    www.polyglotprogrammincinc.com

    [email protected]

    !

    View full-size slide