Easy* Rewrites with ruby and @jesseplusplus GitHub Science!

Jesse Toth @jesseplusplus backend ! # $ % things for &

Easy* Rewrites with ruby and Science!

' * Rewrites are never easy

a rewrite ? ! ? !

' “Create a more flexible system to grant and revoke access to repositories, forks, issues, pull requests, teams, and organizations”

history

First, there was collaboration

Then, there was organization

! separate ways of granting permissions living side-by-side

! ! mysql> describe permissions; +---------------+---------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+---------+------+-----+---------+-------+ | user_id | int(11) | YES | MUL | NULL | | | repository_id | int(11) | YES | MUL | NULL | | +---------------+---------+------+-----+---------+-------+ 2 rows in set (0.00 sec)

! ! mysql> describe team_members; +-----------------+----------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-----------------+----------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | team_id | int(11) | YES | MUL | NULL | | | user_id | int(11) | YES | MUL | NULL | | | repository_id | int(11) | YES | MUL | NULL | | | created_at | datetime | YES | MUL | NULL | | | updated_at | datetime | YES | MUL | NULL | | +-----------------+----------+------+-----+---------+----------------+ 6 rows in set (0.01 sec)

lists of repositories, pull requests, teams, etc. # Find all repositories which this organization # controls the access to. ! def organization.controlled_repositories . . . end

lists of repositories, pull requests, teams, etc. # Find all pull requests to which this user # has access. Access to PRs follow the user’s # access rights to the PR’s repository. ! def user.accessible_pull_requests . . . end

! separate ways of granting permissions living side-by-side " tons of bugs and edge cases around transitional states

No content

! separate ways of granting permissions living side-by-side " tons of bugs and edge cases around transitional states # performance degredation

SELECT r.id FROM r, (( SELECT r.id as r_ids, 2 as perms from r WHERE r.owner_id = 99999 AND (r.x = 0) ) UNION ALL ( SELECT r.id as r_ids, 1 as perms from r INNER JOIN p ON r.id = p.r_id WHERE p.u_id = 99999 AND (r.x = 0) ) UNION ALL ( SELECT r.id as r_ids, 2 as perms from r INNER JOIN t ON r.o_id = t.o_id INNER JOIN t_m ON t.id = t_m.t_id WHERE t.name = 'X' AND t_m.u_id = 99999 AND (r.x = 0) ) UNION ALL ( SELECT r.id as r_ids, GROUP_CONCAT(distinct t.p) as perms from r INNER JOIN t_m r_t ON r.id = r_t.r_id INNER JOIN t ON r_t.t_id = t.id INNER JOIN t_m u_t ON t.id = u_t.t_id WHERE u_t.u_id = 99999 AND t.name != 'X' AND t.p in (2, 1, 0) AND (r.x = 0) GROUP BY r.id ) UNION ALL ( SELECT r.id as r_ids, 0 as perms from r JOIN u ON r.plan_owner_id = u.id JOIN t ON t.o_id = u.id JOIN t_m ON t.id = t_m.t_id WHERE u.type = 'XX' AND t.name = 'X' AND t_m.u_id = 99999 AND (r.x = 0) AND r.parent_id IS NOT NULL )) AS unioned WHERE r.id = r_ids;

No content

No content

rewrite !

No content

' Goals • Simple, flexible interface to grant and revoke a general permission • Fast — sub-second permission lookups • Easy to integrate and operate — MySQL

$% spike

No content

rewrite …and refactor

' tests weren’t modeling production data

No content

No content

class Repository def pullable_by?(user) science "ability.repository.pullable-by" do |e| e.context :user => (user.id if user), :repo => id ! e.use { pullable_by_legacy?(user) } e.try { pullable_by_abs?(user) } end end ! def pullable_by_abs?(user) user.can? :read, self end ! def pullable_by_legacy?(user) # original code for pullable_by? # collaborators || team || special case end end

class Repository def pullable_by?(user) science "ability.repository.pullable-by" do |e| e.context :user => (user.id if user), :repo => id ! e.use { pullable_by_legacy?(user) } e.try { pullable_by_abs?(user) } end end ! def pullable_by_abs?(user) user.can? :read, self end ! def pullable_by_legacy?(user) # original code for pullable_by? # collaborators || team || special case end end

class Repository def pullable_by?(user) science "ability.repository.pullable-by" do |e| e.context :user => (user.id if user), :repo => id ! e.use { pullable_by_legacy?(user) } e.try { pullable_by_abs?(user) } end end ! def pullable_by_abs?(user) user.can? :read, self end ! def pullable_by_legacy?(user) # original code for pullable_by? # collaborators || team || special case end end

class Repository def pullable_by?(user) science "ability.repository.pullable-by" do |e| e.context :user => (user.id if user), :repo => id ! e.use { pullable_by_legacy?(user) } e.try { pullable_by_abs?(user) } end end ! def pullable_by_abs?(user) user.can? :read, self end ! def pullable_by_legacy?(user) # original code for pullable_by? # collaborators || team || special case end end

class Repository def pullable_by?(user) science "ability.repository.pullable-by" do |e| e.context :user => (user.id if user), :repo => id ! e.use { pullable_by_legacy?(user) } e.try { pullable_by_abs?(user) } end end ! def pullable_by_abs?(user) user.can? :read, self end ! def pullable_by_legacy?(user) # original code for pullable_by? # collaborators || team || special case end end

class Repository def pullable_by?(user) science "ability.repository.pullable-by" do |e| e.context :user => (user.id if user), :repo => id ! e.use { pullable_by_legacy?(user) } e.try { pullable_by_abs?(user) } end end ! def pullable_by_abs?(user) user.can? :read, self end ! def pullable_by_legacy?(user) # original code for pullable_by? # collaborators || team || special case end end

class Scientist::Experiment def publish(event, payload) instrument "science.#{event}", payload end end

subscribe /^science\./ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.total" timing "#{experiment}.control", payload[:control][:duration] timing "#{experiment}.candidate", payload[:candidate][:duration] end ! subscribe /^science\.mismatch/ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.wrong" Redis.lpush "#{experiment}.mismatch", payload.to_json end

subscribe /^science\./ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.total" timing "#{experiment}.control", payload[:control][:duration] timing "#{experiment}.candidate", payload[:candidate][:duration] end ! subscribe /^science\.mismatch/ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.wrong" Redis.lpush "#{experiment}.mismatch", payload.to_json end

subscribe /^science\./ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.total" timing "#{experiment}.control", payload[:control][:duration] timing "#{experiment}.candidate", payload[:candidate][:duration] end ! subscribe /^science\.mismatch/ do |name, . . ., payload| name = payload[:experiment] experiment = "science.#{name}" ! increment "#{experiment}.wrong" Redis.lpush "#{experiment}.mismatch", payload.to_json end

subscribe /^science\./ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.total" timing "#{experiment}.control", payload[:control][:duration] timing "#{experiment}.candidate", payload[:candidate][:duration] end ! subscribe /^science\.mismatch/ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.wrong" Redis.lpush "#{experiment}.mismatch", payload.to_json end

subscribe /^science\./ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.total" timing "#{experiment}.control", payload[:control][:duration] timing "#{experiment}.candidate", payload[:candidate][:duration] end ! subscribe /^science\.mismatch/ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.wrong" Redis.lpush "#{experiment}.mismatch", payload.to_json end

subscribe /^science\./ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.total" timing "#{experiment}.control", payload[:control][:duration] timing "#{experiment}.candidate", payload[:candidate][:duration] end ! subscribe /^science\.mismatch/ do |name, . . ., payload| experiment = "science.#{payload[:experiment]}" ! increment "#{experiment}.wrong" Redis.lpush "#{experiment}.mismatch", payload.to_json end

' lessons learned

No content

user.can? :read, repository repository.grant user, :write repository.revoke user

+------+ | User |-------------- +------+ +------+ | Team |-------------- +------+ +------------+ | Repository |-------- +------------+

+------+ | User |---------------- +------+ +------+ 1 | Team |--------+------- +------+ | | +------------+ v | Repository |---------- +------------+

+------+ 2 | User |-----------+---- +------+ | | +------+ 1 v | Team |--------+------- +------+ | | +------------+ v | Repository |---------- +------------+

+------+ 2 3 | User |-----------+--+- +------+ | . | . +------+ 1 v . | Team |--------+-----.- +------+ | . | . +------------+ v v | Repository |---------- +------------+

& wrote the core of Abilities in a few months ' once Abilities was written, wrote migrators to generate Abilities data from legacy data % dark ship writing to Abilities and the legacy tables at the same time

science all the things

' ( examining the data

No content

No content

No content

–Johnny Appleseed “Type a quote here.” analyzing mismatches

irb(main):001> Redis.llen “science.ability.repository.pullable-by.mismatch” => 3283 ! irb(main):002> result = Redis.rpop “science.ability.repository.pullable- by.mismatch” => {"from"=>"GitHub::Jobs::ProcessEvent", "experiment"=>"ability.repository.pullable-by", "user"=>XXXXXXX, “repo"=>XXXXXXX, "timestamp"=>"2014-05-28T18:53:16-07:00", "candidate"=>{"duration"=>0.41368900000000003, "exception"=>nil, "value"=>false}, "control"=>{"duration"=>1.141963, "exception"=>nil, "value"=>true}, "first"=>"control"} !

irb(main):001> Redis.llen “science.ability.repository.pullable-by.mismatch” => 3283 ! irb(main):002> result = Redis.rpop “science.ability.repository.pullable- by.mismatch” => {"from"=>"GitHub::Jobs::ProcessEvent", "experiment"=>"ability.repository.pullable-by", "user"=>XXXXXXX, “repo"=>XXXXXXX, "timestamp"=>"2014-05-28T18:53:16-07:00", "candidate"=>{"duration"=>0.41368900000000003, "exception"=>nil, "value"=>false}, "control"=>{"duration"=>1.141963, "exception"=>nil, "value"=>true}, "first"=>"control"} !

irb(main):001> Redis.llen “science.ability.repository.pullable-by.mismatch” => 3283 ! irb(main):002> result = Redis.rpop “science.ability.repository.pullable- by.mismatch” => {"from"=>"GitHub::Jobs::ProcessEvent", "experiment"=>"ability.repository.pullable-by", "user"=>XXXXXXX, “repo"=>XXXXXXX, "timestamp"=>"2014-05-28T18:53:16-07:00", "candidate"=>{"duration"=>0.41368900000000003, "exception"=>nil, "value"=>false}, "control"=>{"duration"=>1.141963, "exception"=>nil, "value"=>true}, "first"=>"control"} !

irb(main):001> Redis.llen “science.ability.repository.pullable-by.mismatch” => 3283 ! irb(main):002> result = Redis.rpop “science.ability.repository.pullable- by.mismatch” => {"from"=>"GitHub::Jobs::ProcessEvent", "experiment"=>"ability.repository.pullable-by", "user"=>XXXXXXX, “repo"=>XXXXXXX, "timestamp"=>"2014-05-28T18:53:16-07:00", "candidate"=>{"duration"=>0.41368900000000003, "exception"=>nil, "value"=>false}, "control"=>{"duration"=>1.141963, "exception"=>nil, "value"=>true}, "first"=>"control"} !

" ' )*

' data quality

" ' )*

' ( performance problems

No content

No content

No content

rewrite …and refactor …and ) ) …and * # repair …and (

) progress organizations teams repositories

No content

) progress organizations teams repositories

No content

) progress organizations teams repositories

' One Last Data Quality Issue

No content

+ #

No content

No content

No content

No content

No content

) progress organizations teams repositories

–Johnny Appleseed “Type a quote here.” http://github.com/github/scientist + scientist

Thank you!