Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolution of Rails application architecture: 14 years in production

Evolution of Rails application architecture: 14 years in production

The story about the architecture of famous and biggest Rails projects such as Shopify, Basecamp and Cookpad.

Nikolay Sverchkov

October 07, 2018
Tweet

More Decks by Nikolay Sverchkov

Other Decks in Programming

Transcript

  1. View Slide

  2. View Slide

  3. View Slide

  4. evilmartians.com

    View Slide

  5. WHAT
    AWAITS US
    NOW?

    View Slide

  6. CLOSING
    KEYNOTE

    View Slide

  7. WE LOVE CATS!

    View Slide

  8. TATARSTAN

    View Slide

  9. View Slide

  10. THE 2017/2018
    CHAMPIONS

    View Slide

  11. 15 DAKAR RALLY WINS

    View Slide

  12. 34TH BY POPULATION
    CITY IN RUSSIA

    View Slide

  13. 34TH BY POPULATION
    CITY IN RUSSIA
    THE THIRD AMONG
    THE CITIES THAT ARE
    NOT THE CENTRES OF
    THE SUBJECTS OF
    FEDERATIONS

    View Slide

  14. View Slide

  15. View Slide

  16. View Slide

  17. View Slide

  18. https://ru.wikipedia.org/wiki/Эволюция_(значения)

    View Slide

  19. https://ru.wikipedia.org/wiki/Эволюция_(значения)

    View Slide

  20. EVOLUTION OF RAILS
    APPLICATION ARCHITECTURE:
    14 YEARS IN PRODUCTION

    View Slide

  21. THE PAST

    View Slide

  22. Ruby on Rails to Basecamp
    21 Mar 2004 by David Heinemeier Hansson

    View Slide

  23. Ruby on Rails to Basecamp
    21 Mar 2004 by David Heinemeier Hansson

    View Slide

  24. Ruby on Rails to Basecamp
    21 Mar 2004 by David Heinemeier Hansson

    View Slide

  25. Ruby on Rails to Basecamp
    21 Mar 2004 by David Heinemeier Hansson

    View Slide

  26. 2008
    2009
    2010
    MAX ACCELERATION

    View Slide

  27. POPULAR GEMS
    Devise СanCan
    Will
    paginate
    Paperclip Capistrano Friendly Id SimleForm
    2 Jan
    2010
    14 Dec
    2009
    2 Mar
    2008
    19 Mar
    2008
    24 Dec
    2006
    (2009)
    26 Aug
    2008
    6 Feb
    2010

    View Slide

  28. View Slide

  29. 2008. PORTLAND
    https://conferences.oreilly.com/rails2008/public/schedule/full

    View Slide

  30. 2009. LAS VEGAS
    https://conferences.oreilly.com/rails2009/public/schedule/full

    View Slide

  31. https://www.youtube.com/watch?v=YX3iRjKj7C0

    View Slide

  32. 2013
    2014
    MAX SPEED

    View Slide

  33. 2013. PORTLAND

    View Slide

  34. 2014. CHICAGO
    Ultra Light and
    Maintainable
    Rails Wizards
    REFLECTION ON CODE

    View Slide

  35. 2014. CHICAGO REFLECTION ON CODE
    Domain Driven
    Design and
    Hexagonal
    Architecture
    with Rails

    View Slide

  36. 2014. CHICAGO
    Where did the
    OO go?
    Views should be
    objects too!
    REFLECTION ON CODE

    View Slide

  37. 2014. CHICAGO
    Make an Event
    of It
    REFLECTION ON CODE

    View Slide

  38. 2014. CHICAGO
    •Ultra Light and Maintainable Rails Wiz…
    •Domain Driven Design and Hexagonal …
    •Where did the OO go? Views should …
    •Modeling on the Right Side of the Brain
    •Make an Event of It
    •Service Oriented Authenication
    •Concerns, Decorators, Presenters…
    •Reading Code Good
    •Rails as an SOA Client
    •Refactoring Towards Component…
    •Authorization in a Service-Oriented …
    •What is REST? Why is it Part of the …
    •Service Extraction at Groupon Scale
    REFLECTION ON CODE

    View Slide

  39. 2014. CHICAGO PERFORMANCE
    ?

    View Slide

  40. 2014. CHICAGO PERFORMANCE
    Improve
    Performance
    Quick and Cheap:
    Optimize Memory
    and Upgrade
    to Ruby 2.1

    View Slide

  41. PERFORMANCE
    REFLECTION ON
    CODE
    15 1
    :

    View Slide

  42. BACK TO THE FUTURE
    2018

    And this is our future?

    View Slide

  43. 2018. PITTSBURGH

    View Slide

  44. 2018. PITTSBURGH

    View Slide

  45. TODAY

    View Slide

  46. View Slide

  47. View Slide

  48. View Slide

  49. View Slide

  50. Kir Shatrov
    Rails, Ruby and Kubernetes. Site
    Reliability Engineer at @Shopify.
    @evilmartians alumni.

    View Slide

  51. View Slide

  52. --------------------------------------------------------------------------------
    Language files blank comment code
    --------------------------------------------------------------------------------
    Ruby 13995 271122 19124 1290867
    YAML 1791 8962 1166 559395
    JSON 332 77 0 303475
    Sass 484 39191 4968 185032
    ERB 2584 7208 83 111490
    JavaScript 1065 22635 5956 109023
    CoffeeScript 667 12130 1003 45734
    SQL 4 14 0 28300
    TypeScript 455 3794 190 23939
    liquid 64 4391 28 21809
    ……
    ……
    --------------------------------------------------------------------------------
    SUM: 21742 376631 38507 2705978
    --------------------------------------------------------------------------------
    Shopify 30/05/2018
    306k commits
    SUM: 2.7M LOC

    View Slide

  53. WHY U STILL
    MONOLITHIC?

    View Slide

  54. MONOLITHIC







    MICROSERVICES
    >

    View Slide

  55. А ЧЕГО МЫ ДОСТИГНЕМ
    ЕСЛИ СЕЙЧАС
    ОСТАНОВИТЬСЯ И
    НАЧАТЬ РАСПИЛИВАТЬ
    МОНОЛИТ?

    View Slide

  56. КОМПАНИИ
    ОТКАЗЫВАЮТСЯ ОТ
    МОНОЛИТА НА РЕЛЬСАХ
    ПОТОМУ ЧТО РЕЛЬСЫ
    НЕ ПРЕДЛАГАЮТ ХОРОШИХ
    СПОСОБОВ ОРГАНИЗАЦИИ
    КОДА НА БОЛЬШОЙ
    КОДОВОЙ БАЗЕ

    View Slide

  57. У НАС НАКОПИЛСЯ ОЧЕНЬ
    КРУТОЙ ТУЛИНГ ДЛЯ
    МОНОЛИТА. К ПРИМЕРУ,
    ТЕСТЫ В 300 ПОТОКОВ ЗА
    10 МИНУТ

    View Slide

  58. View Slide

  59. View Slide

  60. > 400

    View Slide

  61. https://stackshare.io/shopify/e-commerce-at-scale-inside-shopifys-tech-stack

    View Slide

  62. SHOW ME YOUR
    CODE!
    ROBERT MARTIN

    View Slide

  63. COMPONENT BASED
    ARCHITECTURE
    ├── Gemfile
    ├── Gemfile.lock
    ├── README.md
    ├── components
    │ ├── component_a
    │ │ ├──
    │ │ ├──
    │ │ ├──
    │ │ ├──
    │ │ └──
    │ ├── component_b
    │ │ ├──
    │ │ ├──
    │ │ ├──
    │ │ ├──
    │ │ └──
    │ └── component_c
    │ │ ├──
    │ │ ├──
    │ │ ├──
    │ │ ├──
    │ │ └──
    ├── lib
    ├── test
    └── etc...

    View Slide

  64. COMPONENT BASED
    ARCHITECTURE
    ├── Gemfile
    ├── Gemfile.lock
    ├── README.md
    ├── components
    │ ├── component_a
    │ │ ├── README.md
    │ │ ├── app
    │ │ ├── component.yml
    │ │ ├── lib
    │ │ └── test
    │ ├── component_b
    │ │ ├── README.md
    │ │ ├── app
    │ │ ├── component.yml
    │ │ ├── lib
    │ │ └── test
    │ └── component_c
    │ │ ├── README.md
    │ │ ├── app
    │ │ ├── component.yml
    │ │ ├── lib
    │ │ └── test
    ├── lib
    ├── test
    └── etc...

    View Slide

  65. ENTRY POINTS
    LAYER

    View Slide

  66. НАДО ПОНИМАТЬ, ЧТО
    ЛЕГАСИ МНОГО И ЕЩЕ НЕ
    ВСЕ ПЕРЕЕХАЛИ НА
    ENTRY POINTS.
    МЫ СЧИТАЕМ НЕКИЙ SCORE
    КАЖДОГО КОМПОНЕНТА И
    ОЦЕНИВАЕМ КАКИЕ ИЗ НИХ
    БОЛЕЕ ЛЕГАСИ ЧЕМ
    ОСТАЛЬНЫЕ

    View Slide

  67. ЕСТЬ NATIVE GEM
    КОТОРЫЙ ИСПОЛЬЗУЕТ
    TRACEPOINTS ЧТОБЫ
    КОНТРОЛИРОВАТЬ
    ВЫЗОВЫ ВНЕ
    КОМПОНЕНТОВ

    View Slide

  68. ПРОВОДЯТСЯ
    ВНУТРЕННИЕ МИТАПЫ
    ДЛЯ ОЦЕНКИ КОНЦЕПЦИИ,
    СБОРА FEEDBACK’A И
    ПРЕЗЕНТАЦИИ
    СЛЕДУЮЩИХ ШАГОВ

    View Slide

  69. View Slide

  70. Tom Ward
    Developer @basecamp,
    substandard twitter
    biography writer

    View Slide

  71. THE FIRST BASECAMP WAS
    STARTED BACK IN 2004,
    BASECAMP 2 WAS STARTED
    IN 2011, AND BC3 IN 2014

    View Slide

  72. 3 VERSIONS OF BASECAMP

    View Slide

  73. 3 VERSIONS OF BASECAMP

    View Slide

  74. 3 VERSIONS OF BASECAMP

    View Slide

  75. STARTING A WHOLE NEW
    CODEBASE ALLOWS TO USE
    NEW, BETTER WAYS TO
    STRUCTURE OUR CODE,
    BASED ON THINGS WE’VE
    LEARNT BUILDING
    PREVIOUS ITERATIONS

    View Slide

  76. WE ALSO CREATE AN
    ENTIRELY NEW
    ARCHITECTURE, DATABASE
    MODEL, AND MORE WITH
    EACH ITERATION, AGAIN
    BASED ON WHAT WE’VE
    LEARNT

    View Slide

  77. THE PROS FOR OUR
    CUSTOMERS ARE THAT THEY
    CAN CONTINUE TO USE APPS
    THEY RELY ON UNTIL ‘THE END
    OF THE INTERNET’; THEY
    NEVER HAVE TO UPGRADE TO
    A NEW VERSION IF THE
    EXISTING VERSION WORKS
    FOR THEM

    View Slide

  78. [R]EVOLUTION OF RAILS
    APPLICATION ARCHITECTURE:
    14 YEARS IN PRODUCTION

    View Slide

  79. THE MAJESTIC MONOLITH
    https://signalvnoise.com/the-majestic-monolith-29166d022228

    View Slide

  80. WE DO HAVE AN
    INDEPENDENT SECURITY/
    AUTHORISATION APP
    SHARED ACROSS ALL OUR
    PRODUCTS, AS WELL AS A
    BILLING AND ACCOUNT
    MANAGEMENT APP

    View Slide

  81. WHEN WILL THE
    TIME COME FOR
    THE 4 VERSION?

    View Slide

  82. SOME REASONS MIGHT
    BE IF THE CODEBASE FEELS
    IT’S NO LONGER FUN TO
    WORK ON, IF THERE ARE
    NEW TECHNOLOGIES WE
    WANT TO COMPLETELY
    BASE A NEW VERSION
    AROUND

    View Slide

  83. DO YOU REALLY USE ONLY
    MVC AND PATTERNS BASED
    ON RAILS (E.G. SERIALIZERS)?
    WITHOUT ANY OTHER DESIGN
    PATTERNS LIKE SERVICE
    OBJECT, PRESENTER, FORM
    OBJECT AND ETC.

    View Slide

  84. MARTIN FOWLER
    DO YOU REALLY USE ONLY
    MVC AND PATTERNS BASED
    ON RAILS (E.G. SERIALIZERS)?
    WITHOUT ANY OTHER DESIGN
    PATTERNS LIKE SERVICE
    OBJECT, PRESENTER, FORM
    OBJECT AND ETC.

    View Slide

  85. WE TRY TO USE
    LIGHTWEIGHT
    CONTROLLERS, SIMPLE
    MODELS, AND BASIC
    PATTERNS. BUT IT’S NOT
    POSSIBLE TO IMPLEMENT
    EVERYTHING WE’D LIKE TO
    IN THIS WAY

    View Slide

  86. WE ALSO HAVE
    PRESENTERS, SOME CO-
    ORDINATING MODELS, AND
    EVEN A FEW TRANSACTION-
    LIKE OBJECTS FOR
    COMPLEX THINGS LIKE
    MOVES AND COPIES

    View Slide

  87. BC3

    View Slide

  88. BC3

    View Slide

  89. View Slide

  90. View Slide

  91. View Slide

  92. HELLO DARKNESS, MY OLD FRIEND
    NO ONE ANSWERED ME

    View Slide

  93. View Slide

  94. http://tenderlovemaking.com/railsconf2010.pdf
    Tasty Burgers
    Aaron Patterson
    [RELAX MODE ON/]

    View Slide

  95. http://tenderlovemaking.com/railsconf2010.pdf
    Aaron Patterson
    Tasty Burgers
    [RELAX MODE ON/]

    View Slide

  96. View Slide

  97. View Slide

  98. View Slide

  99. View Slide

  100. View Slide

  101. View Slide

  102. View Slide

  103. View Slide

  104. View Slide

  105. View Slide

  106. View Slide

  107. View Slide

  108. View Slide

  109. View Slide

  110. View Slide

  111. View Slide

  112. View Slide

  113. View Slide

  114. View Slide

  115. View Slide

  116. View Slide

  117. View Slide

  118. View Slide

  119. View Slide

  120. View Slide

  121. [/RELAX MODE OFF]

    View Slide

  122. View Slide

  123. Matz (the creator of Ruby), Koichi and Mame
    (full-time Ruby committers at Cookpad)
    Cookpad
    Ruby Hack
    Challenge
    2018

    View Slide

  124. AT COOKPAD WE HAVE TWO
    INDEPENDENT
    ENGINEERING TEAMS:
    COOKPAD JAPAN AND
    COOKPAD GLOBAL
    (WHERE I WORK)

    View Slide

  125. Sebastian Sogamoso
    Software engineer @Cookpad_Dev.
    Organizer @RubyConfCo

    View Slide

  126. IN COOKPAD GLOBAL WE
    HAVE A MONOLITH AND
    SEVERAL OTHER
    APPLICATIONS THAT
    COMMUNICATE BETWEEN,
    BUT I WOULDN’T CALL THEM
    MICRO SERVICES

    View Slide

  127. WE HAVE CREATED A FEW
    APPS AS SERVICES THAT WE
    HAVE MERGE INTO OUR
    MONOLITH AFTER WE’VE
    SEEN THAT IT INTRODUCED
    MORE PROBLEMS THAN THE
    ONES IT FIXED

    View Slide

  128. WE’RE STARTING TO BUILD
    A BUNCH OF DISTRIBUTED
    SERVICES RELYING ON
    EVENT SOURCING
    BECAUSE IT MAKES OUR
    LIVES EASIER

    View Slide

  129. WE’RE STARTING TO BUILD
    A BUNCH OF DISTRIBUTED
    SERVICES RELYING ON
    EVENT SOURCING
    BECAUSE IT MAKES OUR
    LIVES EASIER
    Events? Great!

    View Slide

  130. WE DO CARE A LOT ABOUT
    OUR ARCHITECTURE, WE
    HAVE BEEN WORKING ON
    IMPROVING IT

    View Slide

  131. View Slide

  132. Takamichi Yoshikawa
    Cat allergy
    [Ex] Software engineer COOKPAD Inc.
    Sunao Hogelog
    Software engineer di クックパッド

    View Slide

  133. https://speakerdeck.com/adorechic/how-cookpad-shifts-to-microservices

    View Slide

  134. 2011-2012
    WE HAVE
    SOME-INCREDIBLE-HUGE-
    MONOLITHIC-APPS THAT
    HAVE SHARED CODES AND
    DBS.

    View Slide

  135. WE USED SYMLINKS AT
    FIRST. WE EXTRACTED
    "SHARED" CODES TO A
    DIRECTORY, AND ALL
    RAILS APPLICATIONS
    REFERS IT VIA SYMLINK

    View Slide

  136. https://speakerdeck.com/adorechic/how-cookpad-shifts-to-microservices

    View Slide

  137. View Slide

  138. GOT A PROBLEM WITH
    SHARED CODE - DIFFICULT TO
    DO CHANGES, DIFFICULT TO
    COORDINATE COMMANDS,
    DIFFICULT TO TRACK
    DEPENDENCIES AND
    APPS HAD MONKEY PATCHING
    OF SHARED LOGIC

    View Slide

  139. PANTRY

    View Slide

  140. MICROSERVICES API
    UNCONTROLLED AND NOT
    UNIFIED. SOLUTION: OWN
    EXTENSION FOR RAILS APP
    FOR MAKING API - GARAGE!

    View Slide

  141. MICROSERVICES API
    UNCONTROLLED AND NOT
    UNIFIED. SOLUTION: OWN
    EXTENSION FOR RAILS APP
    FOR MAKING API - GARAGE!

    View Slide

  142. https://github.com/cookpad/garage

    View Slide

  143. MANY DIFFERENT
    MICROSERVICES ARE
    PRONE TO NETWORK
    PROBLEMS.
    SOLUTION: OWN GEM
    EXPEDITOR!

    View Slide

  144. https://github.com/cookpad/expeditor

    View Slide

  145. --------------------------------------------------------------------------------
    Language files blank comment code
    --------------------------------------------------------------------------------
    Ruby 1778 34618 4380 171259
    Sass 817 17876 449 110313
    Haml 3273 10047 649 83502
    YAML 37 450 313 33705
    CoffeeScript 296 2246 329 11993
    JavaScript 95 832 409 5560
    ERB 225 814 51 5293
    C 1 86 10 635
    Bourne Shell 6 78 77 289
    Bourne Again Shell 10 34 0 157
    builder 2 4 0 86
    Markdown 3 25 0 46
    Perl 2 12 0 38
    Ruby HTML 1 0 0 12
    CSS 1 0 0 1
    --------------------------------------------------------------------------------
    SUM: 6547 67122 6667 422889
    --------------------------------------------------------------------------------
    COOKPAD 27/09/2018
    THE MOST HUGE APP
    SUM: 422K LOC

    View Slide

  146. https://blog.envoyproxy.io/service-mesh-and-cookpad-ba4d5d915dbd

    View Slide

  147. https://blog.envoyproxy.io/service-mesh-and-cookpad-ba4d5d915dbd
    +
    go-control-plane.

    View Slide

  148. SUMM4RY

    View Slide

  149. RAILS IS AN YOUNG ADULT

    View Slide

  150. RAILS
    IS
    LEGO

    View Slide

  151. THE BEST
    OSS IS
    BORN FROM
    PRODUCTION

    View Slide

  152. TAKE CARE
    OF
    ARCHITECTURE

    View Slide

  153. BC3
    COMPETITION
    13

    View Slide

  154. I'M PRETTY SURE
    IT'S A CAT

    View Slide

  155. cultofmartians.com

    View Slide

  156. @ssnickolay
    Nikolay Sverchkov
    @ssnickolay
    THANK YOU!
    @evilmartians
    evilmartians.com/blog

    View Slide