$30 off During Our Annual Pro Sale. View Details »

Basecamp Next: Code Spelunking

Basecamp Next: Code Spelunking

Heard about the big Basecamp launch this March? Wondering what's new, how it's shaping Rails, and the tech behind it? We're going to go over some the practices and patterns in the new Basecamp's code base and you can learn how to improve your app with them.

Nick Quaranto

April 25, 2012
Tweet

More Decks by Nick Quaranto

Other Decks in Programming

Transcript

  1. code
    spelunking
    basecamp next
    @qrush

    View Slide

  2. i’m @qrush

    View Slide

  3. @
    i live here

    View Slide

  4. i work at 37signals

    View Slide

  5. old & new patterns
    neato gems
    how stuff works™

    View Slide

  6. 50 days
    months
    commits
    code-test
    12
    14k
    1:1

    View Slide

  7. architecture
    overview

    View Slide

  8. rails
    services

    View Slide

  9. rails
    mysql

    View Slide

  10. rails
    mysql memcached

    View Slide

  11. rails
    mysql
    redis memcached

    View Slide

  12. rails
    mysql
    redis memcached
    elastic
    search

    View Slide

  13. rails
    mysql
    redis memcached
    depot
    elastic
    search

    View Slide

  14. rails
    mysql
    redis memcached
    portfolio
    depot
    elastic
    search

    View Slide

  15. rails
    launchpad
    mysql
    redis memcached
    portfolio
    depot
    elastic
    search

    View Slide

  16. rails
    launchpad
    mysql
    redis memcached
    queenbee portfolio
    depot
    elastic
    search

    View Slide

  17. View Slide

  18. View Slide

  19. View Slide

  20. rails
    frontends

    View Slide

  21. poller rails

    View Slide

  22. poller rails
    rainbows

    View Slide

  23. poller rails
    rainbows
    nginx

    View Slide

  24. poller rails
    unicorn
    rainbows
    nginx

    View Slide

  25. poller rails
    unicorn
    nginx
    rainbows
    nginx

    View Slide

  26. poller rails
    unicorn unicorn
    nginx nginx
    rainbows ...
    ...
    nginx
    rails

    View Slide

  27. rails
    local

    View Slide

  28. rails
    mysql

    View Slide

  29. pow
    http://pow.cx

    View Slide

  30. rails
    mysql
    pow

    View Slide

  31. rails
    mysql
    pow
    launchpad

    View Slide

  32. rails
    mysql
    pow
    launchpad
    pow

    View Slide

  33. rails
    mysql
    pow
    launchpad portfolio
    pow

    View Slide

  34. rails
    mysql
    pow
    launchpad portfolio
    pow pow

    View Slide

  35. bcx.dev
    launchpad.dev
    37img.dev

    View Slide

  36. testing

    View Slide

  37. View Slide

  38. Test::Unit
    mocha
    capybara
    m http://github.com/qrush/m
    testing

    View Slide

  39. functional
    pow
    unit
    integration

    View Slide

  40. functional
    pow
    unit
    integration
    api

    View Slide

  41. functional
    pow
    unit
    integration
    api
    poller

    View Slide

  42. functional
    pow
    unit
    integration
    api
    poller
    37id

    View Slide

  43. functional
    pow
    unit
    integration
    api
    poller
    37id
    acceptance
    client

    View Slide

  44. chain rake tasks

    View Slide

  45. require 'rails/test_unit/sub_test_task'
    Rails::SubTestTask.new api: 'test:prepare' do |t|
    t.libs << 'test'
    t.pattern = 'test/api/**/*_test.rb'
    end

    View Slide

  46. Rake::Task['test:run'].enhance do
    Rake::Task['test:api'].invoke
    end

    View Slide

  47. development
    setup

    View Slide

  48. View Slide

  49. the reset button

    View Slide

  50. one bash script

    View Slide

  51. ./script/setup

    View Slide

  52. ./script/setup;
    rake

    View Slide

  53. 1. setup app
    2. reset database
    3. restart server
    4. custom wrangling

    View Slide

  54. keep it quiet

    View Slide

  55. echo "Installing libraries..."
    { (gem list -i bundler ||
    gem install bundler) &&
    rbenv rehash;
    bundle install && rbenv rehash
    } >&3 2>&1

    View Slide

  56. use flags to speed up

    View Slide

  57. if [ -z "$KEEPDB" ]; then
    echo "Reloading the database"
    {
    # lots o’ rake tasks
    } >&3 2>&1
    fi

    View Slide

  58. concerns
    everywhere

    View Slide

  59. sharing code is hard

    View Slide

  60. use a concern!

    View Slide

  61. module YourConcern
    extend ActiveSupport::Concern
    included do
    # class methods here
    end
    # instance methods here
    end

    View Slide

  62. module Ajax
    extend ActiveSupport::Concern
    included do
    before_filter :set_ajax_cookies
    end
    private
    def set_ajax_cookies

    View Slide

  63. models
    trashing
    searching
    buckets
    permissions

    View Slide

  64. controllers
    rendering assets
    authorization
    authentication
    current person, account

    View Slide

  65. group alike methods

    View Slide

  66. class ApplicationController
    include CurrentAccount
    include CurrentPerson
    include ExceptionNotification
    include Ajax
    include Mobile
    include EnsureCompatibleBrowser
    include WrongContentTypeFallback

    View Slide

  67. View Slide

  68. modules != classes

    View Slide

  69. still a ball of mud

    View Slide

  70. assets
    lots of them

    View Slide

  71. ~9700 ruby
    tests
    coffeescript
    ~10800
    ~9600
    lines of:

    View Slide

  72. lots of SCSS too!

    View Slide

  73. View Slide

  74. vendor/assets
    jquery
    backbone
    underscore
    node’s EventEmitter
    wysihtml5
    eco

    View Slide

  75. eco
    http://github.com/sstephenson/eco

    View Slide

  76. wysihtml5
    http://github.com/xing/wysihtml5

    View Slide

  77. jquery plugins
    scrollTo
    easing
    embedly
    cookies
    md5
    outsideEvents

    View Slide

  78. javascript
    patterns

    View Slide

  79. extend jQuery

    View Slide

  80. $.fn.highlight = ->
    $(this).
    effect("highlight", {}, 1500)

    View Slide

  81. $.fn.resetForm = ->
    @removeClass('submitting').
    get(0).
    reset()
    this

    View Slide

  82. window.bcx

    View Slide

  83. // application.js.coffee.erb
    window.bcx ?= {}
    _.extend bcx,
    models: {}
    collections: {}
    views: {}
    env: <%= Rails.env.to_json %>

    View Slide

  84. data-behaviors

    View Slide

  85. access-toggle
    account_name_cancel
    account_name_form
    account_name_header
    account_name_header_name
    account_name_link
    activate
    add_calendar
    add_to_all_calendars
    add_to_all_projects
    admin_permission
    alt_date_field
    assigned_to
    assignee_name
    assignee_options
    attachments_required
    autoresize
    autosave
    backbone_collection
    backbone_model
    bounce
    bounce_nav
    bucket_selector
    caching_classic_projects
    calendar_accesses
    calendar_alert
    calendar_event_drag_area
    calendar_invitees
    calendar_todo
    can_create_projects_permissi
    on
    collapse_on_click
    complete
    confirm
    confirm_action
    create_project
    date_entry
    date_picker
    delete
    dirty_tracking
    document_version_link
    due_date
    edit
    edit_bucket
    edit_calendar_event_form
    edit_calendar_todo
    edit_identity
    edit_project_header
    editable_field_prompt
    email_preferences
    enlargeable
    expand_exclusively
    expand_on_click
    expandable
    expandable
    expand_exclusively
    file_drop_target
    filter_and
    filter_due
    format_timeline
    grant-all
    has_hover_content
    input_change_emitter
    invite
    invite_notice
    invitees
    jump_to_month
    jump_to_new_calendar_event
    lazy_invitees
    lazy_load_subscribers
    link_container
    load_assignee_options
    load_completed_todos
    member
    members
    migrate_account
    move_action
    move_operation_form
    navigate
    new
    new_calendar_event
    new_group
    new_message
    new_project
    new_project_dialog
    new_subgroup
    new_todolist
    new_upload
    no_due_date
    no_reset
    notification_email
    nubbin
    pending_attachments
    project_palette
    project_star
    read_only
    refresh_timeline
    remove
    remove_duplicates
    remove_group
    remove_invitee
    remove_member
    rename_group
    resend_invitation
    resubscribe
    reveal_event_subsc
    revoke
    revoke-all
    save_document
    save_group
    scroll_content
    scroll_forward
    scroll_reverse
    scroll_view
    select_on_focus
    select_suggestion
    set_start_of_week
    show_all_todos
    show_link_if_acces
    slide
    slider
    sortable
    sortable_container
    sortable_handle

    View Slide

  86. access-toggle
    account_name_cancel
    account_name_form
    account_name_header
    account_name_header_name
    account_name_link
    activate
    add_calendar
    add_to_all_calendars
    add_to_all_projects
    admin_permission
    alt_date_field
    assigned_to
    assignee_name
    assignee_options
    attachments_required
    autoresize
    autosave
    backbone_collection
    backbone_model
    bounce
    bounce_nav
    bucket_selector
    caching_classic_projects
    calendar_accesses
    calendar_alert
    calendar_event_drag_area
    calendar_invitees
    calendar_todo
    can_create_projects_permissi
    on
    collapse_on_click
    complete
    confirm
    confirm_action
    create_project
    date_entry
    date_picker
    delete
    dirty_tracking
    document_version_link
    due_date
    edit
    edit_bucket
    edit_calendar_event_form
    edit_calendar_todo
    edit_identity
    edit_project_header
    editable_field_prompt
    email_preferences
    enlargeable
    expand_exclusively
    expand_on_click
    expandable
    expandable
    expand_exclusively
    file_drop_target
    filter_and
    filter_due
    format_timeline
    grant-all
    has_hover_content
    input_change_emitter
    invite
    invite_notice
    invitees
    jump_to_month
    jump_to_new_calendar_event
    lazy_invitees
    lazy_load_subscribers
    link_container
    load_assignee_options
    load_completed_todos
    member
    members
    migrate_account
    move_action
    move_operation_form
    navigate
    new
    new_calendar_event
    new_group
    new_message
    new_project
    new_project_dialog
    new_subgroup
    new_todolist
    new_upload
    no_due_date
    no_reset
    notification_email
    nubbin
    pending_attachments
    project_palette
    project_star
    read_only
    refresh_timeline
    remove
    remove_duplicates
    remove_group
    remove_invitee
    remove_member
    rename_group
    resend_invitation
    resubscribe
    reveal_event_subsc
    revoke
    revoke-all
    save_document
    save_group
    scroll_content
    scroll_forward
    scroll_reverse
    scroll_view
    select_on_focus
    select_suggestion
    set_start_of_week
    show_all_todos
    show_link_if_acces
    slide
    slider
    sortable
    sortable_container
    sortable_handle

    View Slide

  87. out the wazoo!
    keeps javascript logic out of views
    stops tying behavior to css classes
    still easy to query for

    View Slide

  88. $('[data-behavior~=date_picker]').
    live 'focus', ->
    $(this).datepicker()

    View Slide

  89. rails-behaviors
    http://josh.github.com/rails-behaviors/
    * not using this yet!

    View Slide

  90. page events

    View Slide

  91. use custom events!
    _.bind() is your new best friend
    trigger custom events for what you need
    keep namespacing consistent

    View Slide

  92. View Slide

  93. Stacker basics
    intercepts most clicks on
    requests the page via $.ajax
    renders the response as a new “page”
    smart about errors too!

    View Slide

  94. history.js
    https://github.com/balupton/History.js/

    View Slide

  95. bcx.on 'page:beforechange',
    'set bcx.currentBucket', ->
    bcx.on 'page:change',
    'lazily load invitees', ->
    bcx.on 'page:update',
    'install invitee field lists', ->

    View Slide

  96. page:beforechange
    setting the current project/calendar
    apply classes to
    start performance timing

    View Slide

  97. page:change
    fired when a new “page” is loaded
    install views
    hide/show elements based on roles
    lazily load data

    View Slide

  98. page:update
    fired after change, on every ajaxSuccess
    install views, behavior
    moved loaded data into place
    stop “busy” animations

    View Slide

  99. View Slide

  100. View Slide

  101. console
    mastery

    View Slide

  102. console.count()
    +1

    View Slide

  103. console.count("rendered a todo")
    1
    console.count("rendered a todo")
    2

    View Slide

  104. console.count("rendered a todo")
    1
    console.count("rendered a todo")
    2

    View Slide

  105. console.warn()
    console.error()
    make messages stand out

    View Slide

  106. console.group()
    console.groupEnd()
    clump together logs

    View Slide

  107. View Slide

  108. View Slide

  109. View Slide

  110. console.profile()
    console.profileEnd()
    see what’s slow

    View Slide

  111. View Slide

  112. View Slide

  113. View Slide

  114. console.profile("something slow")
    console.profileEnd()

    View Slide

  115. anywhere in your code:
    debugger
    when stopped:
    console.trace()

    View Slide

  116. chrome.csi()

    View Slide

  117. console.csi

    View Slide

  118. chrome.csi()
    see how long it’s been since page load

    View Slide

  119. View Slide

  120. View Slide

  121. api building
    with love

    View Slide

  122. Repo for API docs
    http://github.com/37signals/bcx-api

    View Slide

  123. jbuilder
    http://github.com/rails/jbuilder

    View Slide

  124. json.(@document, :id, :title, :content,
    :created_at, :updated_at)
    json.last_updater
    @document.last_updater, :id, :name
    json.partial! "api/comments/comments",
    comments: @document.comments

    View Slide

  125. {
    "id": 963979453,
    "title": "By Jove!",
    "content": "I’ve figured it out!",
    "created_at": "2012-03-27T13:19:29-05:0
    "updated_at": "2012-03-27T13:53:24-05:0
    "last_updater": {
    "id": 149087659,
    "name": "Sherlock Holmes"
    },
    "comments": [
    ]
    }

    View Slide

  126. why not to_json?
    keep view data in the views
    use partials!
    public vs private JSON

    View Slide

  127. strong_parameters
    http://github.com/rails/strong_parameters

    View Slide

  128. class Api::DocumentsController <
    Api::BaseController
    def update
    @document.update_attributes!
    document_params
    render :show
    end
    def document_params
    params.required(:document).
    permit(:title, :content)
    end
    end

    View Slide

  129. class Api::DocumentsController <
    Api::BaseController
    def update
    @document.update_attributes!
    document_params
    render :show
    end
    def document_params
    params.required(:document).
    permit(:title, :content)
    end
    end

    View Slide

  130. class Api::DocumentsController <
    Api::BaseController
    def update
    @document.update_attributes!
    document_params
    render :show
    end
    def document_params
    params.required(:document).
    permit(:title, :content)
    end
    end

    View Slide

  131. class Api::DocumentsController <
    Api::BaseController
    def update
    @document.update_attributes!
    document_params
    render :show
    end
    def document_params
    params.required(:document).
    permit(:title, :content)
    end
    end

    View Slide

  132. why not attr_accessible?
    controller, not a model problem
    can still share via concerns
    easier to return error, status codes

    View Slide

  133. HTTP 204 No Content
    http://httpstatus.es/204
    Use on create, destroy actions!

    View Slide

  134. can’t read
    headers from
    HTTP 204
    http://bugs.jquery.com/ticket/1450

    View Slide

  135. View Slide

  136. View Slide

  137. logging
    is worth it

    View Slide

  138. View Slide

  139. log EVERYTHING
    unique ID per request
    account ID for log splitting
    controller, actions on each SQL query

    View Slide

  140. ActiveSupport::TaggedLogger
    new in Rails 3.2!

    View Slide

  141. YourApp::Application.configure do
    config.log_tags = [:uuid]
    end
    [71ba53fd717c67a6677a058f4a5acdf4]
    Processing by ProjectsController#index
    as HTML

    View Slide

  142. BCX::Application.configure do
    config.log_tags = [
    -> request {
    request.env['bcx.account.queenbee_id']
    }
    ]
    end

    View Slide

  143. That just happened.
    <3 Ruby 1.9 lambdas
    Extremely configurable logging
    Use Rack’s environment for sharing data

    View Slide

  144. # github.com/rails/rails
    # 3-2-stable
    # railties/lib/rails/application.rb
    middleware.use ::Rails::Rack::Logger,
    config.log_tags

    View Slide

  145. module Rails::Rack::Logger
    # if tags are defined...
    Rails.logger.tagged(compute_tags(env)) {
    call_app(env)
    }
    # else
    call_app(env)

    View Slide

  146. def compute_tags(env)
    request = ActionDispatch::Request.new(env)
    @tags.collect do |tag|
    case tag
    when Proc
    tag.call(request)
    when Symbol
    request.send(tag)
    else
    tag
    end
    end
    end

    View Slide

  147. def compute_tags(env)
    request = ActionDispatch::Request.new(env)
    @tags.collect do |tag|
    case tag
    when Proc
    tag.call(request)
    when Symbol
    request.send(tag)
    else
    tag
    end
    end
    end

    View Slide

  148. def compute_tags(env)
    request = ActionDispatch::Request.new(env)
    @tags.collect do |tag|
    case tag
    when Proc
    tag.call(request)
    when Symbol
    request.send(tag)
    else
    tag
    end
    end
    end

    View Slide

  149. def compute_tags(env)
    request = ActionDispatch::Request.new(env)
    @tags.collect do |tag|
    case tag
    when Proc
    tag.call(request)
    when Symbol
    request.send(tag)
    else
    tag
    end
    end
    end

    View Slide

  150. def compute_tags(env)
    request = ActionDispatch::Request.new(env)
    @tags.collect do |tag|
    case tag
    when Proc
    tag.call(request)
    when Symbol
    request.send(tag)
    else
    tag
    end
    end
    end

    View Slide

  151. View Slide

  152. marginalia
    http://github.com/37signals/marginalia

    View Slide

  153. Account Load (0.3ms) SELECT `accounts`.*
    FROM `accounts`
    WHERE `accounts`.`queenbee_id` = 1234567890
    LIMIT 1
    /*application:BCX,
    controller:project_imports,action:show*/

    View Slide

  154. # in your Gemfile
    gem 'query_comments'
    # in an initializer
    Marginalia.application_name = "BCX"

    View Slide

  155. MORE logging?
    Know where slow queries came from
    Immediately identify web or job query
    MySQL, sqlite, maybe Postgres now?

    View Slide

  156. stats
    count everything

    View Slide

  157. statsd
    http://github.com/etsy/statsd
    http://github.com/jeremy/statsd-ruby

    View Slide

  158. tracks:
    usage, performance
    successful, failed logins
    processed emails
    people invited

    View Slide

  159. ActiveSupport::Notifications
    http://api.rubyonrails.org/classes/
    ActiveSupport/Notifications.html

    View Slide

  160. ActiveSupport::Notifications.instrument
    :usage, measurement: "logins.succeeded"

    View Slide

  161. ActiveSupport::Notifications.instrument
    :usage, measurement: "logins.succeeded"
    event name

    View Slide

  162. ActiveSupport::Notifications.instrument
    :usage, measurement: "logins.succeeded"
    hash payload

    View Slide

  163. ActiveSupport::Notifications.subscribe(/usage/) do
    |name, start, finish, id, payload|
    Statsd.increment(payload[:measurement])
    end

    View Slide

  164. resque-statsd
    https://github.com/jamster/resque-statsd

    View Slide

  165. View Slide

  166. View Slide

  167. tracks:
    enqueue count
    completion count
    failure count
    time to process

    View Slide

  168. we’re still learning
    don’t stop evolving

    View Slide

  169. thanks
    for listening!
    @qrush

    View Slide