Evolution of Rails application architecture: 14 years in production

The story about the architecture of famous and biggest Rails projects such as Shopify, Basecamp and Cookpad.

Transcript

4. evilmartians.com
5. WHAT AWAITS US NOW?
6. CLOSING KEYNOTE
7. WE LOVE CATS!
8. TATARSTAN
10. THE 2017/2018 CHAMPIONS
11. 15 DAKAR RALLY WINS
12. 34TH BY POPULATION CITY IN RUSSIA
13. 34TH BY POPULATION CITY IN RUSSIA THE THIRD AMONG THE CITIES THAT ARE NOT THE CENTRES OF THE SUBJECTS OF FEDERATIONS
18. https://ru.wikipedia.org/wiki/Эволюция_(значения)
19. https://ru.wikipedia.org/wiki/Эволюция_(значения)
20. EVOLUTION OF RAILS APPLICATION ARCHITECTURE: 14 YEARS IN PRODUCTION
21. THE PAST
22. Ruby on Rails to Basecamp 21 Mar 2004 by David Heinemeier Hansson
23. Ruby on Rails to Basecamp 21 Mar 2004 by David Heinemeier Hansson
24. Ruby on Rails to Basecamp 21 Mar 2004 by David Heinemeier Hansson
25. Ruby on Rails to Basecamp 21 Mar 2004 by David Heinemeier Hansson
26. 2008 2009 2010 MAX ACCELERATION
27. POPULAR GEMS Devise СanCan Will paginate Paperclip Capistrano Friendly Id SimleForm 2 Jan 2010 14 Dec 2009 2 Mar 2008 19 Mar 2008 24 Dec 2006 (2009) 26 Aug 2008 6 Feb 2010
29. 2008. PORTLAND https://conferences.oreilly.com/rails2008/public/schedule/full
30. 2009. LAS VEGAS https://conferences.oreilly.com/rails2009/public/schedule/full
31. https://www.youtube.com/watch?v=YX3iRjKj7C0
32. 2013 2014 MAX SPEED
33. 2013. PORTLAND
34. 2014. CHICAGO Ultra Light and Maintainable Rails Wizards REFLECTION ON CODE
35. 2014. CHICAGO REFLECTION ON CODE Domain Driven Design and Hexagonal Architecture with Rails
36. 2014. CHICAGO Where did the OO go? Views should be objects too! REFLECTION ON CODE
37. 2014. CHICAGO Make an Event of It REFLECTION ON CODE
38. 2014. CHICAGO •Ultra Light and Maintainable Rails Wiz… •Domain Driven Design and Hexagonal … •Where did the OO go? Views should … •Modeling on the Right Side of the Brain •Make an Event of It •Service Oriented Authenication •Concerns, Decorators, Presenters… •Reading Code Good •Rails as an SOA Client •Refactoring Towards Component… •Authorization in a Service-Oriented … •What is REST? Why is it Part of the … •Service Extraction at Groupon Scale REFLECTION ON CODE
39. 2014. CHICAGO PERFORMANCE ?
40. 2014. CHICAGO PERFORMANCE Improve Performance Quick and Cheap: Optimize Memory and Upgrade to Ruby 2.1
41. PERFORMANCE REFLECTION ON CODE 15 1 :
42. BACK TO THE FUTURE 2018 … And this is our future?
43. 2018. PITTSBURGH
44. 2018. PITTSBURGH
45. TODAY
50. Kir Shatrov Rails, Ruby and Kubernetes. Site Reliability Engineer at @Shopify. @evilmartians alumni.
52. -------------------------------------------------------------------------------- Language files blank comment code -------------------------------------------------------------------------------- Ruby 13995 271122 19124 1290867 YAML 1791 8962 1166 559395 JSON 332 77 0 303475 Sass 484 39191 4968 185032 ERB 2584 7208 83 111490 JavaScript 1065 22635 5956 109023 CoffeeScript 667 12130 1003 45734 SQL 4 14 0 28300 TypeScript 455 3794 190 23939 liquid 64 4391 28 21809 …… …… -------------------------------------------------------------------------------- SUM: 21742 376631 38507 2705978 -------------------------------------------------------------------------------- Shopify 30/05/2018 306k commits SUM: 2.7M LOC
53. WHY U STILL MONOLITHIC?
54. MONOLITHIC MICROSERVICES >
55. А ЧЕГО МЫ ДОСТИГНЕМ ЕСЛИ СЕЙЧАС ОСТАНОВИТЬСЯ И НАЧАТЬ РАСПИЛИВАТЬ МОНОЛИТ?
56. КОМПАНИИ ОТКАЗЫВАЮТСЯ ОТ МОНОЛИТА НА РЕЛЬСАХ ПОТОМУ ЧТО РЕЛЬСЫ НЕ ПРЕДЛАГАЮТ ХОРОШИХ СПОСОБОВ ОРГАНИЗАЦИИ КОДА НА БОЛЬШОЙ КОДОВОЙ БАЗЕ
57. У НАС НАКОПИЛСЯ ОЧЕНЬ КРУТОЙ ТУЛИНГ ДЛЯ МОНОЛИТА. К ПРИМЕРУ, ТЕСТЫ В 300 ПОТОКОВ ЗА 10 МИНУТ
60. > 400
61. https://stackshare.io/shopify/e-commerce-at-scale-inside-shopifys-tech-stack
62. SHOW ME YOUR CODE! ROBERT MARTIN
63. COMPONENT BASED ARCHITECTURE ├── Gemfile ├── Gemfile.lock ├── README.md ├── components │ ├── component_a │ │ ├── │ │ ├── │ │ ├── │ │ ├── │ │ └── │ ├── component_b │ │ ├── │ │ ├── │ │ ├── │ │ ├── │ │ └── │ └── component_c │ │ ├── │ │ ├── │ │ ├── │ │ ├── │ │ └── ├── lib ├── test └── etc...
64. COMPONENT BASED ARCHITECTURE ├── Gemfile ├── Gemfile.lock ├── README.md ├── components │ ├── component_a │ │ ├── README.md │ │ ├── app │ │ ├── component.yml │ │ ├── lib │ │ └── test │ ├── component_b │ │ ├── README.md │ │ ├── app │ │ ├── component.yml │ │ ├── lib │ │ └── test │ └── component_c │ │ ├── README.md │ │ ├── app │ │ ├── component.yml │ │ ├── lib │ │ └── test ├── lib ├── test └── etc...
65. ENTRY POINTS LAYER
66. НАДО ПОНИМАТЬ, ЧТО ЛЕГАСИ МНОГО И ЕЩЕ НЕ ВСЕ ПЕРЕЕХАЛИ НА ENTRY POINTS. МЫ СЧИТАЕМ НЕКИЙ SCORE КАЖДОГО КОМПОНЕНТА И ОЦЕНИВАЕМ КАКИЕ ИЗ НИХ БОЛЕЕ ЛЕГАСИ ЧЕМ ОСТАЛЬНЫЕ
67. ЕСТЬ NATIVE GEM КОТОРЫЙ ИСПОЛЬЗУЕТ TRACEPOINTS ЧТОБЫ КОНТРОЛИРОВАТЬ ВЫЗОВЫ ВНЕ КОМПОНЕНТОВ
68. ПРОВОДЯТСЯ ВНУТРЕННИЕ МИТАПЫ ДЛЯ ОЦЕНКИ КОНЦЕПЦИИ, СБОРА FEEDBACK’A И ПРЕЗЕНТАЦИИ СЛЕДУЮЩИХ ШАГОВ
70. Tom Ward Developer @basecamp, substandard twitter biography writer
71. THE FIRST BASECAMP WAS STARTED BACK IN 2004, BASECAMP 2 WAS STARTED IN 2011, AND BC3 IN 2014
72. 3 VERSIONS OF BASECAMP
73. 3 VERSIONS OF BASECAMP
74. 3 VERSIONS OF BASECAMP
75. STARTING A WHOLE NEW CODEBASE ALLOWS TO USE NEW, BETTER WAYS TO STRUCTURE OUR CODE, BASED ON THINGS WE’VE LEARNT BUILDING PREVIOUS ITERATIONS
76. WE ALSO CREATE AN ENTIRELY NEW ARCHITECTURE, DATABASE MODEL, AND MORE WITH EACH ITERATION, AGAIN BASED ON WHAT WE’VE LEARNT
77. THE PROS FOR OUR CUSTOMERS ARE THAT THEY CAN CONTINUE TO USE APPS THEY RELY ON UNTIL ‘THE END OF THE INTERNET’; THEY NEVER HAVE TO UPGRADE TO A NEW VERSION IF THE EXISTING VERSION WORKS FOR THEM
78. [R]EVOLUTION OF RAILS APPLICATION ARCHITECTURE: 14 YEARS IN PRODUCTION
79. THE MAJESTIC MONOLITH https://signalvnoise.com/the-majestic-monolith-29166d022228
80. WE DO HAVE AN INDEPENDENT SECURITY/ AUTHORISATION APP SHARED ACROSS ALL OUR PRODUCTS, AS WELL AS A BILLING AND ACCOUNT MANAGEMENT APP
81. WHEN WILL THE TIME COME FOR THE 4 VERSION?
82. SOME REASONS MIGHT BE IF THE CODEBASE FEELS IT’S NO LONGER FUN TO WORK ON, IF THERE ARE NEW TECHNOLOGIES WE WANT TO COMPLETELY BASE A NEW VERSION AROUND
83. DO YOU REALLY USE ONLY MVC AND PATTERNS BASED ON RAILS (E.G. SERIALIZERS)? WITHOUT ANY OTHER DESIGN PATTERNS LIKE SERVICE OBJECT, PRESENTER, FORM OBJECT AND ETC.
84. MARTIN FOWLER DO YOU REALLY USE ONLY MVC AND PATTERNS BASED ON RAILS (E.G. SERIALIZERS)? WITHOUT ANY OTHER DESIGN PATTERNS LIKE SERVICE OBJECT, PRESENTER, FORM OBJECT AND ETC.
85. WE TRY TO USE LIGHTWEIGHT CONTROLLERS, SIMPLE MODELS, AND BASIC PATTERNS. BUT IT’S NOT POSSIBLE TO IMPLEMENT EVERYTHING WE’D LIKE TO IN THIS WAY
86. WE ALSO HAVE PRESENTERS, SOME CO- ORDINATING MODELS, AND EVEN A FEW TRANSACTION- LIKE OBJECTS FOR COMPLEX THINGS LIKE MOVES AND COPIES
87. BC3
88. BC3
92. HELLO DARKNESS, MY OLD FRIEND NO ONE ANSWERED ME
94. http://tenderlovemaking.com/railsconf2010.pdf Tasty Burgers Aaron Patterson [RELAX MODE ON/]
95. http://tenderlovemaking.com/railsconf2010.pdf Aaron Patterson Tasty Burgers [RELAX MODE ON/]
121. [/RELAX MODE OFF]
123. Matz (the creator of Ruby), Koichi and Mame (full-time Ruby committers at Cookpad) Cookpad Ruby Hack Challenge 2018
124. AT COOKPAD WE HAVE TWO INDEPENDENT ENGINEERING TEAMS: COOKPAD JAPAN AND COOKPAD GLOBAL (WHERE I WORK)
125. Sebastian Sogamoso Software engineer @Cookpad_Dev. Organizer @RubyConfCo
126. IN COOKPAD GLOBAL WE HAVE A MONOLITH AND SEVERAL OTHER APPLICATIONS THAT COMMUNICATE BETWEEN, BUT I WOULDN’T CALL THEM MICRO SERVICES
127. WE HAVE CREATED A FEW APPS AS SERVICES THAT WE HAVE MERGE INTO OUR MONOLITH AFTER WE’VE SEEN THAT IT INTRODUCED MORE PROBLEMS THAN THE ONES IT FIXED
128. WE’RE STARTING TO BUILD A BUNCH OF DISTRIBUTED SERVICES RELYING ON EVENT SOURCING BECAUSE IT MAKES OUR LIVES EASIER
129. WE’RE STARTING TO BUILD A BUNCH OF DISTRIBUTED SERVICES RELYING ON EVENT SOURCING BECAUSE IT MAKES OUR LIVES EASIER Events? Great!
130. WE DO CARE A LOT ABOUT OUR ARCHITECTURE, WE HAVE BEEN WORKING ON IMPROVING IT
132. Takamichi Yoshikawa Cat allergy [Ex] Software engineer COOKPAD Inc. Sunao Hogelog Software engineer di クックパッド
133. https://speakerdeck.com/adorechic/how-cookpad-shifts-to-microservices
134. 2011-2012 WE HAVE SOME-INCREDIBLE-HUGE- MONOLITHIC-APPS THAT HAVE SHARED CODES AND DBS.
135. WE USED SYMLINKS AT FIRST. WE EXTRACTED "SHARED" CODES TO A DIRECTORY, AND ALL RAILS APPLICATIONS REFERS IT VIA SYMLINK
136. https://speakerdeck.com/adorechic/how-cookpad-shifts-to-microservices
138. GOT A PROBLEM WITH SHARED CODE - DIFFICULT TO DO CHANGES, DIFFICULT TO COORDINATE COMMANDS, DIFFICULT TO TRACK DEPENDENCIES AND APPS HAD MONKEY PATCHING OF SHARED LOGIC
139. PANTRY
140. MICROSERVICES API UNCONTROLLED AND NOT UNIFIED. SOLUTION: OWN EXTENSION FOR RAILS APP FOR MAKING API - GARAGE!
141. MICROSERVICES API UNCONTROLLED AND NOT UNIFIED. SOLUTION: OWN EXTENSION FOR RAILS APP FOR MAKING API - GARAGE!
142. https://github.com/cookpad/garage
143. MANY DIFFERENT MICROSERVICES ARE PRONE TO NETWORK PROBLEMS. SOLUTION: OWN GEM EXPEDITOR!
144. https://github.com/cookpad/expeditor
145. -------------------------------------------------------------------------------- Language files blank comment code -------------------------------------------------------------------------------- Ruby 1778 34618 4380 171259 Sass 817 17876 449 110313 Haml 3273 10047 649 83502 YAML 37 450 313 33705 CoffeeScript 296 2246 329 11993 JavaScript 95 832 409 5560 ERB 225 814 51 5293 C 1 86 10 635 Bourne Shell 6 78 77 289 Bourne Again Shell 10 34 0 157 builder 2 4 0 86 Markdown 3 25 0 46 Perl 2 12 0 38 Ruby HTML 1 0 0 12 CSS 1 0 0 1 -------------------------------------------------------------------------------- SUM: 6547 67122 6667 422889 -------------------------------------------------------------------------------- COOKPAD 27/09/2018 THE MOST HUGE APP SUM: 422K LOC
146. https://blog.envoyproxy.io/service-mesh-and-cookpad-ba4d5d915dbd
147. https://blog.envoyproxy.io/service-mesh-and-cookpad-ba4d5d915dbd + go-control-plane.
148. SUMM4RY
149. RAILS IS AN YOUNG ADULT
150. RAILS IS LEGO
151. THE BEST OSS IS BORN FROM PRODUCTION
152. TAKE CARE OF ARCHITECTURE
153. BC3 COMPETITION 13
154. I'M PRETTY SURE IT'S A CAT
155. cultofmartians.com
156. @ssnickolay Nikolay Sverchkov @ssnickolay THANK YOU! @evilmartians evilmartians.com/blog