Basecamp Next: Code Spelunking

Heard about the big Basecamp launch this March? Wondering what's new, how it's shaping Rails, and the tech behind it? We're going to go over some the practices and patterns in the new Basecamp's code base and you can learn how to improve your app with them.

Transcript

1. code
   spelunking
   basecamp next
   @qrush
   

   View Slide

2. i’m @qrush
   

   View Slide

3. @
   i live here
   

   View Slide

4. i work at 37signals
   

   View Slide

5. old & new patterns
   neato gems
   how stuff works™
   

   View Slide

6. 50 days
   months
   commits
   code-test
   12
   14k
   1:1
   

   View Slide

7. architecture
   overview
   

   View Slide

8. rails
   services
   

   View Slide

9. rails
   mysql
   

   View Slide

10. rails
    mysql memcached
    

    View Slide

11. rails
    mysql
    redis memcached
    

    View Slide

12. rails
    mysql
    redis memcached
    elastic
    search
    

    View Slide

13. rails
    mysql
    redis memcached
    depot
    elastic
    search
    

    View Slide

14. rails
    mysql
    redis memcached
    portfolio
    depot
    elastic
    search
    

    View Slide

15. rails
    launchpad
    mysql
    redis memcached
    portfolio
    depot
    elastic
    search
    

    View Slide

16. rails
    launchpad
    mysql
    redis memcached
    queenbee portfolio
    depot
    elastic
    search
    

    View Slide

17. View Slide

18. View Slide

19. View Slide

20. rails
    frontends
    

    View Slide

21. poller rails
    

    View Slide

22. poller rails
    rainbows
    

    View Slide

23. poller rails
    rainbows
    nginx
    

    View Slide

24. poller rails
    unicorn
    rainbows
    nginx
    

    View Slide

25. poller rails
    unicorn
    nginx
    rainbows
    nginx
    

    View Slide

26. poller rails
    unicorn unicorn
    nginx nginx
    rainbows ...
    ...
    nginx
    rails
    

    View Slide

27. rails
    local
    

    View Slide

28. rails
    mysql
    

    View Slide

29. pow
    http://pow.cx
    

    View Slide

30. rails
    mysql
    pow
    

    View Slide

31. rails
    mysql
    pow
    launchpad
    

    View Slide

32. rails
    mysql
    pow
    launchpad
    pow
    

    View Slide

33. rails
    mysql
    pow
    launchpad portfolio
    pow
    

    View Slide

34. rails
    mysql
    pow
    launchpad portfolio
    pow pow
    

    View Slide

35. bcx.dev
    launchpad.dev
    37img.dev
    

    View Slide

36. testing
    

    View Slide

37. View Slide

38. Test::Unit
    mocha
    capybara
    m http://github.com/qrush/m
    testing
    

    View Slide

39. functional
    pow
    unit
    integration
    

    View Slide

40. functional
    pow
    unit
    integration
    api
    

    View Slide

41. functional
    pow
    unit
    integration
    api
    poller
    

    View Slide

42. functional
    pow
    unit
    integration
    api
    poller
    37id
    

    View Slide

43. functional
    pow
    unit
    integration
    api
    poller
    37id
    acceptance
    client
    

    View Slide

44. chain rake tasks
    

    View Slide

45. require 'rails/test_unit/sub_test_task'
    Rails::SubTestTask.new api: 'test:prepare' do |t|
    t.libs << 'test'
    t.pattern = 'test/api/**/*_test.rb'
    end
    

    View Slide

46. Rake::Task['test:run'].enhance do
    Rake::Task['test:api'].invoke
    end
    

    View Slide

47. development
    setup
    

    View Slide

48. View Slide

49. the reset button
    

    View Slide

50. one bash script
    

    View Slide

51. ./script/setup
    

    View Slide

52. ./script/setup;
    rake
    

    View Slide

53. 1. setup app
    2. reset database
    3. restart server
    4. custom wrangling
    

    View Slide

54. keep it quiet
    

    View Slide

55. echo "Installing libraries..."
    { (gem list -i bundler ||
    gem install bundler) &&
    rbenv rehash;
    bundle install && rbenv rehash
    } >&3 2>&1
    

    View Slide

56. use flags to speed up
    

    View Slide

57. if [ -z "$KEEPDB" ]; then
    echo "Reloading the database"
    {
    # lots o’ rake tasks
    } >&3 2>&1
    fi
    

    View Slide

58. concerns
    everywhere
    

    View Slide

59. sharing code is hard
    

    View Slide

60. use a concern!
    

    View Slide

61. module YourConcern
    extend ActiveSupport::Concern
    included do
    # class methods here
    end
    # instance methods here
    end
    

    View Slide

62. module Ajax
    extend ActiveSupport::Concern
    included do
    before_filter :set_ajax_cookies
    end
    private
    def set_ajax_cookies
    

    View Slide

63. models
    trashing
    searching
    buckets
    permissions
    

    View Slide

64. controllers
    rendering assets
    authorization
    authentication
    current person, account
    

    View Slide

65. group alike methods
    

    View Slide

66. class ApplicationController
    include CurrentAccount
    include CurrentPerson
    include ExceptionNotification
    include Ajax
    include Mobile
    include EnsureCompatibleBrowser
    include WrongContentTypeFallback
    

    View Slide

67. View Slide

68. modules != classes
    

    View Slide

69. still a ball of mud
    

    View Slide

70. assets
    lots of them
    

    View Slide

71. ~9700 ruby
    tests
    coffeescript
    ~10800
    ~9600
    lines of:
    

    View Slide

72. lots of SCSS too!
    

    View Slide

73. View Slide

74. vendor/assets
    jquery
    backbone
    underscore
    node’s EventEmitter
    wysihtml5
    eco
    

    View Slide

75. eco
    http://github.com/sstephenson/eco
    

    View Slide

76. wysihtml5
    http://github.com/xing/wysihtml5
    

    View Slide

77. jquery plugins
    scrollTo
    easing
    embedly
    cookies
    md5
    outsideEvents
    

    View Slide

78. javascript
    patterns
    

    View Slide

79. extend jQuery
    

    View Slide

80. $.fn.highlight = ->
    $(this).
    effect("highlight", {}, 1500)
    

    View Slide

81. $.fn.resetForm = ->
    @removeClass('submitting').
    get(0).
    reset()
    this
    

    View Slide

82. window.bcx
    

    View Slide

83. // application.js.coffee.erb
    window.bcx ?= {}
    _.extend bcx,
    models: {}
    collections: {}
    views: {}
    env:
    

    View Slide

84. data-behaviors
    

    View Slide

85. access-toggle
    account_name_cancel
    account_name_form
    account_name_header
    account_name_header_name
    account_name_link
    activate
    add_calendar
    add_to_all_calendars
    add_to_all_projects
    admin_permission
    alt_date_field
    assigned_to
    assignee_name
    assignee_options
    attachments_required
    autoresize
    autosave
    backbone_collection
    backbone_model
    bounce
    bounce_nav
    bucket_selector
    caching_classic_projects
    calendar_accesses
    calendar_alert
    calendar_event_drag_area
    calendar_invitees
    calendar_todo
    can_create_projects_permissi
    on
    collapse_on_click
    complete
    confirm
    confirm_action
    create_project
    date_entry
    date_picker
    delete
    dirty_tracking
    document_version_link
    due_date
    edit
    edit_bucket
    edit_calendar_event_form
    edit_calendar_todo
    edit_identity
    edit_project_header
    editable_field_prompt
    email_preferences
    enlargeable
    expand_exclusively
    expand_on_click
    expandable
    expandable
    expand_exclusively
    file_drop_target
    filter_and
    filter_due
    format_timeline
    grant-all
    has_hover_content
    input_change_emitter
    invite
    invite_notice
    invitees
    jump_to_month
    jump_to_new_calendar_event
    lazy_invitees
    lazy_load_subscribers
    link_container
    load_assignee_options
    load_completed_todos
    member
    members
    migrate_account
    move_action
    move_operation_form
    navigate
    new
    new_calendar_event
    new_group
    new_message
    new_project
    new_project_dialog
    new_subgroup
    new_todolist
    new_upload
    no_due_date
    no_reset
    notification_email
    nubbin
    pending_attachments
    project_palette
    project_star
    read_only
    refresh_timeline
    remove
    remove_duplicates
    remove_group
    remove_invitee
    remove_member
    rename_group
    resend_invitation
    resubscribe
    reveal_event_subsc
    revoke
    revoke-all
    save_document
    save_group
    scroll_content
    scroll_forward
    scroll_reverse
    scroll_view
    select_on_focus
    select_suggestion
    set_start_of_week
    show_all_todos
    show_link_if_acces
    slide
    slider
    sortable
    sortable_container
    sortable_handle
    

    View Slide

86. access-toggle
    account_name_cancel
    account_name_form
    account_name_header
    account_name_header_name
    account_name_link
    activate
    add_calendar
    add_to_all_calendars
    add_to_all_projects
    admin_permission
    alt_date_field
    assigned_to
    assignee_name
    assignee_options
    attachments_required
    autoresize
    autosave
    backbone_collection
    backbone_model
    bounce
    bounce_nav
    bucket_selector
    caching_classic_projects
    calendar_accesses
    calendar_alert
    calendar_event_drag_area
    calendar_invitees
    calendar_todo
    can_create_projects_permissi
    on
    collapse_on_click
    complete
    confirm
    confirm_action
    create_project
    date_entry
    date_picker
    delete
    dirty_tracking
    document_version_link
    due_date
    edit
    edit_bucket
    edit_calendar_event_form
    edit_calendar_todo
    edit_identity
    edit_project_header
    editable_field_prompt
    email_preferences
    enlargeable
    expand_exclusively
    expand_on_click
    expandable
    expandable
    expand_exclusively
    file_drop_target
    filter_and
    filter_due
    format_timeline
    grant-all
    has_hover_content
    input_change_emitter
    invite
    invite_notice
    invitees
    jump_to_month
    jump_to_new_calendar_event
    lazy_invitees
    lazy_load_subscribers
    link_container
    load_assignee_options
    load_completed_todos
    member
    members
    migrate_account
    move_action
    move_operation_form
    navigate
    new
    new_calendar_event
    new_group
    new_message
    new_project
    new_project_dialog
    new_subgroup
    new_todolist
    new_upload
    no_due_date
    no_reset
    notification_email
    nubbin
    pending_attachments
    project_palette
    project_star
    read_only
    refresh_timeline
    remove
    remove_duplicates
    remove_group
    remove_invitee
    remove_member
    rename_group
    resend_invitation
    resubscribe
    reveal_event_subsc
    revoke
    revoke-all
    save_document
    save_group
    scroll_content
    scroll_forward
    scroll_reverse
    scroll_view
    select_on_focus
    select_suggestion
    set_start_of_week
    show_all_todos
    show_link_if_acces
    slide
    slider
    sortable
    sortable_container
    sortable_handle
    

    View Slide

87. out the wazoo!
    keeps javascript logic out of views
    stops tying behavior to css classes
    still easy to query for
    

    View Slide

88. $('[data-behavior~=date_picker]').
    live 'focus', ->
    $(this).datepicker()
    

    View Slide

89. rails-behaviors
    http://josh.github.com/rails-behaviors/
    * not using this yet!
    

    View Slide

90. page events
    

    View Slide

91. use custom events!
    _.bind() is your new best friend
    trigger custom events for what you need
    keep namespacing consistent
    

    View Slide

92. View Slide

93. Stacker basics
    intercepts most clicks on
    requests the page via $.ajax
    renders the response as a new “page”
    smart about errors too!
    

    View Slide

94. history.js
    https://github.com/balupton/History.js/
    

    View Slide

95. bcx.on 'page:beforechange',
    'set bcx.currentBucket', ->
    bcx.on 'page:change',
    'lazily load invitees', ->
    bcx.on 'page:update',
    'install invitee field lists', ->
    

    View Slide

96. page:beforechange
    setting the current project/calendar
    apply classes to
    start performance timing
    

    View Slide

97. page:change
    fired when a new “page” is loaded
    install views
    hide/show elements based on roles
    lazily load data
    

    View Slide

98. page:update
    fired after change, on every ajaxSuccess
    install views, behavior
    moved loaded data into place
    stop “busy” animations
    

    View Slide

99. View Slide

100. View Slide

101. console
     mastery
     

     View Slide

102. console.count()
     +1
     

     View Slide

103. console.count("rendered a todo")
     1
     console.count("rendered a todo")
     2
     

     View Slide

104. console.count("rendered a todo")
     1
     console.count("rendered a todo")
     2
     

     View Slide

105. console.warn()
     console.error()
     make messages stand out
     

     View Slide

106. console.group()
     console.groupEnd()
     clump together logs
     

     View Slide

107. View Slide

108. View Slide

109. View Slide

110. console.profile()
     console.profileEnd()
     see what’s slow
     

     View Slide

111. View Slide

112. View Slide

113. View Slide

114. console.profile("something slow")
     console.profileEnd()
     

     View Slide

115. anywhere in your code:
     debugger
     when stopped:
     console.trace()
     

     View Slide

116. chrome.csi()
     

     View Slide

117. console.csi
     

     View Slide

118. chrome.csi()
     see how long it’s been since page load
     

     View Slide

119. View Slide

120. View Slide

121. api building
     with love
     

     View Slide

122. Repo for API docs
     http://github.com/37signals/bcx-api
     

     View Slide

123. jbuilder
     http://github.com/rails/jbuilder
     

     View Slide

124. json.(@document, :id, :title, :content,
     :created_at, :updated_at)
     json.last_updater
     @document.last_updater, :id, :name
     json.partial! "api/comments/comments",
     comments: @document.comments
     

     View Slide

125. {
     "id": 963979453,
     "title": "By Jove!",
     "content": "I’ve figured it out!",
     "created_at": "2012-03-27T13:19:29-05:0
     "updated_at": "2012-03-27T13:53:24-05:0
     "last_updater": {
     "id": 149087659,
     "name": "Sherlock Holmes"
     },
     "comments": [
     ]
     }
     

     View Slide

126. why not to_json?
     keep view data in the views
     use partials!
     public vs private JSON
     

     View Slide

127. strong_parameters
     http://github.com/rails/strong_parameters
     

     View Slide

128. class Api::DocumentsController <
     Api::BaseController
     def update
     @document.update_attributes!
     document_params
     render :show
     end
     def document_params
     params.required(:document).
     permit(:title, :content)
     end
     end
     

     View Slide

129. class Api::DocumentsController <
     Api::BaseController
     def update
     @document.update_attributes!
     document_params
     render :show
     end
     def document_params
     params.required(:document).
     permit(:title, :content)
     end
     end
     

     View Slide

130. class Api::DocumentsController <
     Api::BaseController
     def update
     @document.update_attributes!
     document_params
     render :show
     end
     def document_params
     params.required(:document).
     permit(:title, :content)
     end
     end
     

     View Slide

131. class Api::DocumentsController <
     Api::BaseController
     def update
     @document.update_attributes!
     document_params
     render :show
     end
     def document_params
     params.required(:document).
     permit(:title, :content)
     end
     end
     

     View Slide

132. why not attr_accessible?
     controller, not a model problem
     can still share via concerns
     easier to return error, status codes
     

     View Slide

133. HTTP 204 No Content
     http://httpstatus.es/204
     Use on create, destroy actions!
     

     View Slide

134. can’t read
     headers from
     HTTP 204
     http://bugs.jquery.com/ticket/1450
     

     View Slide

135. View Slide

136. View Slide

137. logging
     is worth it
     

     View Slide

138. View Slide

139. log EVERYTHING
     unique ID per request
     account ID for log splitting
     controller, actions on each SQL query
     

     View Slide

140. ActiveSupport::TaggedLogger
     new in Rails 3.2!
     

     View Slide

141. YourApp::Application.configure do
     config.log_tags = [:uuid]
     end
     [71ba53fd717c67a6677a058f4a5acdf4]
     Processing by ProjectsController#index
     as HTML
     

     View Slide

142. BCX::Application.configure do
     config.log_tags = [
     -> request {
     request.env['bcx.account.queenbee_id']
     }
     ]
     end
     

     View Slide

143. That just happened.
     <3 Ruby 1.9 lambdas
     Extremely configurable logging
     Use Rack’s environment for sharing data
     

     View Slide

144. # github.com/rails/rails
     # 3-2-stable
     # railties/lib/rails/application.rb
     middleware.use ::Rails::Rack::Logger,
     config.log_tags
     

     View Slide

145. module Rails::Rack::Logger
     # if tags are defined...
     Rails.logger.tagged(compute_tags(env)) {
     call_app(env)
     }
     # else
     call_app(env)
     

     View Slide

146. def compute_tags(env)
     request = ActionDispatch::Request.new(env)
     @tags.collect do |tag|
     case tag
     when Proc
     tag.call(request)
     when Symbol
     request.send(tag)
     else
     tag
     end
     end
     end
     

     View Slide

147. def compute_tags(env)
     request = ActionDispatch::Request.new(env)
     @tags.collect do |tag|
     case tag
     when Proc
     tag.call(request)
     when Symbol
     request.send(tag)
     else
     tag
     end
     end
     end
     

     View Slide

148. def compute_tags(env)
     request = ActionDispatch::Request.new(env)
     @tags.collect do |tag|
     case tag
     when Proc
     tag.call(request)
     when Symbol
     request.send(tag)
     else
     tag
     end
     end
     end
     

     View Slide

149. def compute_tags(env)
     request = ActionDispatch::Request.new(env)
     @tags.collect do |tag|
     case tag
     when Proc
     tag.call(request)
     when Symbol
     request.send(tag)
     else
     tag
     end
     end
     end
     

     View Slide

150. def compute_tags(env)
     request = ActionDispatch::Request.new(env)
     @tags.collect do |tag|
     case tag
     when Proc
     tag.call(request)
     when Symbol
     request.send(tag)
     else
     tag
     end
     end
     end
     

     View Slide

151. View Slide

152. marginalia
     http://github.com/37signals/marginalia
     

     View Slide

153. Account Load (0.3ms) SELECT `accounts`.*
     FROM `accounts`
     WHERE `accounts`.`queenbee_id` = 1234567890
     LIMIT 1
     /*application:BCX,
     controller:project_imports,action:show*/
     

     View Slide

154. # in your Gemfile
     gem 'query_comments'
     # in an initializer
     Marginalia.application_name = "BCX"
     

     View Slide

155. MORE logging?
     Know where slow queries came from
     Immediately identify web or job query
     MySQL, sqlite, maybe Postgres now?
     

     View Slide

156. stats
     count everything
     

     View Slide

157. statsd
     http://github.com/etsy/statsd
     http://github.com/jeremy/statsd-ruby
     

     View Slide

158. tracks:
     usage, performance
     successful, failed logins
     processed emails
     people invited
     

     View Slide

159. ActiveSupport::Notifications
     http://api.rubyonrails.org/classes/
     ActiveSupport/Notifications.html
     

     View Slide

160. ActiveSupport::Notifications.instrument
     :usage, measurement: "logins.succeeded"
     

     View Slide

161. ActiveSupport::Notifications.instrument
     :usage, measurement: "logins.succeeded"
     event name
     

     View Slide

162. ActiveSupport::Notifications.instrument
     :usage, measurement: "logins.succeeded"
     hash payload
     

     View Slide

163. ActiveSupport::Notifications.subscribe(/usage/) do
     |name, start, finish, id, payload|
     Statsd.increment(payload[:measurement])
     end
     

     View Slide

164. resque-statsd
     https://github.com/jamster/resque-statsd
     

     View Slide

165. View Slide

166. View Slide

167. tracks:
     enqueue count
     completion count
     failure count
     time to process
     

     View Slide

168. we’re still learning
     don’t stop evolving
     

     View Slide

169. thanks
     for listening!
     @qrush
     

     View Slide