How system for Harley-Davidson 115th anniversary with 110 000 attendees has been built

Tomáš will tell you a story how a small (10 people) company from Prague called Juicymo successfully built an information system and three mobile apps for a huge event with 110 000 attendees. It was an epic journey which started 2 years ago. It's full of funny moments, challenges and lessons learned. I will share it with you. DevOps included!

The talk was presented at the WebExpo 2018 conference in Prague. The video is available here: https://slideslive.com/38910598/how-system-for-harleydavidson-115th-anniversary-with-110-000-attendees-has-been-built

Transcript

1. How the system for 
 Harley-Davidson 
 115th Anniversary with 110 000 attendees has been built Tomáš Jukin Co-Founder at Juicymo @Inza
2. What is a key skill for creating system for 100 000+ users?
3. Paranoia
4. @Inza Co-Founder at Juicymo Tomáš Jukin
5. @Inza Co-Founder at Juicymo Tomáš Jukin
6. @Inza Co-Founder at Juicymo Tomáš Jukin
7. @Inza Co-Founder at Juicymo Tomáš Jukin
8. @Inza Co-Founder at Juicymo Tomáš Jukin
9. The Meeting @Inza @JuicymoEN
10. The Meeting @Inza @JuicymoEN How it all started
11. Monday 5. 9. 2016 @Inza @JuicymoEN
17. @Inza @JuicymoEN
18. The IT Guy @Inza @JuicymoEN
19. The IT Guy @Inza @JuicymoEN The
 Event Manager
20. The IT Guy @Inza @JuicymoEN The
 Event Manager The
 Event Coordinator
21. The IT Guy @Inza @JuicymoEN The
 Event Manager The
 Event Coordinator The
 CEO
22. The IT Guy @Inza @JuicymoEN The
 Event Manager The
 Event Coordinator The
 CEO
23. The IT Guy @Inza @JuicymoEN The
 Event Manager The
 Event Coordinator The
 Project Manager The
 CEO
24. The Event @Inza @JuicymoEN
25. 5 - 8 July 2018 In Prague @Inza @JuicymoEN
26. 115th Anniversary 
 of Harley-Davidson @Inza @JuicymoEN
27. 110 000 attendees @Inza @JuicymoEN
28. 110 000 attendees @Inza @JuicymoEN within 4 days
29. 110 000 attendees @Inza @JuicymoEN within 4 days 40 000 at once (!)
30. 110 000 attendees @Inza @JuicymoEN within 4 days 40 000 at once (!) 30 000 on bike (!!!)
32. I have one concern only - how we will process 40 000 bikers if they arrive in one day… @Inza @JuicymoEN — the event manager
 February 2018
34. The System @Inza @JuicymoEN
35. E-Shop: 30 / 5 / 2017 - 8 / 7 / 2018
 System: 1 / 7 / 2018 - 8 / 7 / 2018 @Inza @JuicymoEN
36. We need just a simple system. It will be an e-shop where tickets can be bought and mobile app to process them at the entrance. @Inza @JuicymoEN
37. and few “details” … @Inza @JuicymoEN
38. @Inza @JuicymoEN
39. 1. More than tickets in the e-shop @Inza @JuicymoEN
40. 1. More than tickets in the e-shop 2. Only ticket owners can buy them @Inza @JuicymoEN
41. 1. More than tickets in the e-shop 2. Only ticket owners can buy them 3. 115 additional items @Inza @JuicymoEN
42. 1. More than tickets in the e-shop 2. Only ticket owners can buy them 3. 115 additional items 4. Around 100 places to redeem them @Inza @JuicymoEN
43. 1. More than tickets in the e-shop 2. Only ticket owners can buy them 3. 115 additional items 4. Around 100 places to redeem them 5. 300 000 users @Inza @JuicymoEN
44. 1. More than tickets in the e-shop 2. Only ticket owners can buy them 3. 115 additional items 4. Around 100 places to redeem them 5. 300 000 users 6. Up to 80 000 at once @Inza @JuicymoEN
45. @Inza @JuicymoEN
46. 7. Buy for friends @Inza @JuicymoEN
47. 7. Buy for friends 8. 6 languages @Inza @JuicymoEN
48. 7. Buy for friends 8. 6 languages 9. 115 additional items @Inza @JuicymoEN
49. 7. Buy for friends 8. 6 languages 9. 115 additional items 10. Payment gateway @Inza @JuicymoEN
50. 7. Buy for friends 8. 6 languages 9. 115 additional items 10. Payment gateway 11. EET @Inza @JuicymoEN
51. 7. Buy for friends 8. 6 languages 9. 115 additional items 10. Payment gateway 11. EET 12. SLA Request time <1s at peak end user @Inza @JuicymoEN
52. The Users @Inza @JuicymoEN
53. The Biker
54. The Organiser
55. The Supervisor The picture is illustrative
56. The Accounting
 Lady The picture is illustrative
57. The DevOps Guys
58. The Wild
 Hostess
59. The Frontend @Inza @JuicymoEN
60. Responsive E-Shop
61. Biker Mobile App
62. Scanner Mobile App
63. The Backend @Inza @JuicymoEN
64. @Inza @JuicymoEN Backend
65. @Inza @JuicymoEN Supervizor Dashboard Backend Admin Wild Hostess Admin Finder Dashboard Backend
66. @Inza @JuicymoEN Supervizor Dashboard Monitoring Big Data Onsite Support Backend Admin Wild Hostess Admin Finder Dashboard SLA
 Disaster
 Recovery Backend
67. @Inza @JuicymoEN Supervizor Dashboard Monitoring Big Data Onsite Support Backend Admin Wild Hostess Admin Finder Dashboard SLA
 Disaster
 Recovery Backend
69. The DevOps @Inza @JuicymoEN
70. The DevOps @Inza @JuicymoEN For the monstrosity ;)
71. @Inza @JuicymoEN
72. @Inza @JuicymoEN Riding
 Biker
73. @Inza @JuicymoEN Walking
 Biker Riding
 Biker
74. @Inza @JuicymoEN Walking
 Biker Riding
 Biker Organiser
75. @Inza @JuicymoEN Walking
 Biker Riding
 Biker Organiser Wild Hostess
76. @Inza @JuicymoEN Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor
77. @Inza @JuicymoEN Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor Accounting Lady
78. @Inza @JuicymoEN Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Accounting Lady
79. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor E-Shop
 Backend HTTPs Accounting Lady
80. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor E-Shop
 Backend HTTPs Accounting Lady Admin
81. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor E-Shop
 Backend HTTPs Accounting Lady Admin Payment
 Gateway HTTPs
82. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor E-Shop
 Backend HTTPs Accounting Lady Admin Payment
 Gateway HTTPs EET
83. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor E-Shop
 Backend HTTPs Accounting Lady Admin Payment
 Gateway HTTPs EET XML Fujky
84. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor E-Shop
 Backend HTTPs Accounting Lady Admin Payment
 Gateway HTTPs EET XML Fujky Biker
 App Scanner
 App
85. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor HTTPs Accounting Lady Admin Payment
 Gateway HTTPs EET XML Fujky Biker
 App Scanner
 App API HTTPs
 REST E-Shop
 Backend
86. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor HTTPs Accounting Lady Admin Payment
 Gateway HTTPs EET XML Fujky Biker
 App Scanner
 App API HTTPs
 REST E-Shop
 Backend CRM ERP Mail Server
87. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor HTTPs Accounting Lady Admin Payment
 Gateway HTTPs EET XML Fujky Biker
 App Scanner
 App API HTTPs
 REST E-Shop
 Backend CRM ERP Mail Server Financial 
 .xls export
88. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor HTTPs Accounting Lady Admin Payment
 Gateway HTTPs EET XML Fujky Biker
 App Scanner
 App API HTTPs
 REST E-Shop
 Backend CRM ERP Mail Server Financial 
 .xls export Wild Hostess 
 Dashboard
89. @Inza @JuicymoEN E-Shop
 Frontend Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor HTTPs Accounting Lady Admin Payment
 Gateway HTTPs EET XML Fujky Biker
 App Scanner
 App API HTTPs
 REST E-Shop
 Backend CRM ERP Mail Server Financial 
 .xls export Wild Hostess 
 Dashboard Supervisor
 Dashboard
90. @Inza @JuicymoEN Admin API Payment
 Gateway CRM EET ERP Mail Server Wild Hostess 
 Dashboard Supervisor
 Dashboard E-Shop
 Frontend Biker
 App Scanner
 App Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor Financial 
 .xls export Kibana Elastic
 Search E-Shop
 Backend XML Fujky HTTPs HTTPs HTTPs
 REST HTTPs Accounting Lady
91. @Inza @JuicymoEN Admin API Payment
 Gateway CRM EET ERP Mail Server Wild Hostess 
 Dashboard Supervisor
 Dashboard E-Shop
 Frontend Biker
 App Scanner
 App Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor Financial 
 .xls export Kibana Elastic
 Search E-Shop
 Backend XML Fujky HTTPs HTTPs HTTPs
 REST HTTPs Accounting Lady Paranoia !!!
92. @Inza @JuicymoEN Admin API Payment
 Gateway CRM EET ERP Mail Server Wild Hostess 
 Dashboard Supervisor
 Dashboard E-Shop
 Frontend Biker
 App Scanner
 App Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor Financial 
 .xls export Grafana Influx Kibana Elastic
 Search E-Shop
 Backend XML Fujky HTTPs HTTPs HTTPs
 REST HTTPs Accounting Lady
93. @Inza @JuicymoEN Admin API Payment
 Gateway CRM EET ERP Mail Server Wild Hostess 
 Dashboard Supervisor
 Dashboard E-Shop
 Frontend Biker
 App Scanner
 App Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor Financial 
 .xls export Grafana Influx Kibana Elastic
 Search E-Shop
 Backend XML Fujky HTTPs HTTPs HTTPs
 REST HTTPs Accounting Lady
94. @Inza @JuicymoEN Admin API Payment
 Gateway CRM EET ERP Mail Server Wild Hostess 
 Dashboard Supervisor
 Dashboard E-Shop
 Frontend Biker
 App Scanner
 App Walking
 Biker Riding
 Biker Organiser Wild Hostess Supervisor Happy 
 Accounting Lady Financial 
 .xls export Grafana Influx Kibana Elastic
 Search E-Shop
 Backend XML Fujky HTTPs HTTPs HTTPs
 REST HTTPs
95. @JuicymoEN Admin API Wild Hostess 
 Dashboard Supervisor
 Dashboard E-Shop
 Backend @Inza Walking Biker Riding Biker Organiser Wild Hostess Supervisor
96. @JuicymoEN Admin API Wild Hostess 
 Dashboard Supervisor
 Dashboard E-Shop
 Backend Ruby on Rails App @Inza Walking Biker Riding Biker Organiser Wild Hostess Supervisor
97. @JuicymoEN Admin API Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) @Inza
98. @JuicymoEN Admin API Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) @Inza
99. @JuicymoEN Admin API Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) HTTP 
 Reverse Proxy 
 Cache
 (scalable) @Inza
100. @JuicymoEN Admin API Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) HTTP 
 Reverse Proxy 
 Cache
 (scalable) @Inza
101. @JuicymoEN Admin API Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) HTTP 
 Reverse Proxy 
 Cache
 (scalable) Postgres @Inza
102. @JuicymoEN Admin API Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) HTTP 
 Reverse Proxy 
 Cache
 (scalable) HA 
 Postgres 
 16GB RAM HA 
 Postgres
 Replica 
 16GB RAM
 Another DC @Inza
103. @JuicymoEN Admin API Mail 
 Server Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) HTTP 
 Reverse Proxy 
 Cache
 (scalable) HA 
 Postgres 
 16GB RAM HA 
 Postgres
 Replica 
 16GB RAM
 Another DC Ticket PDF
 Generation S3 @Inza
104. @JuicymoEN Admin API Mail 
 Server Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) HTTP 
 Reverse Proxy 
 Cache
 (scalable) HA 
 Postgres 
 16GB RAM HA 
 Postgres
 Replica 
 16GB RAM
 Another DC Ticket PDF
 Generation S3 @Inza
105. @JuicymoEN Admin API Mail 
 Server Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) HTTP 
 Reverse Proxy 
 Cache
 (scalable) HA 
 Postgres 
 16GB RAM HA 
 Postgres
 Replica 
 16GB RAM
 Another DC Ticket PDF
 Generation S3 @Inza Background Jobs
 Processor
 Sidekiq
106. @JuicymoEN Admin API Mail 
 Server Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) HTTP 
 Reverse Proxy 
 Cache
 (scalable) HA 
 Postgres 
 16GB RAM HA 
 Postgres
 Replica 
 16GB RAM
 Another DC Redis Ticket PDF
 Generation S3 @Inza Background Jobs
 Processor
 Sidekiq
107. @JuicymoEN Admin API Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) HTTP 
 Reverse Proxy 
 Cache
 (scalable) HA 
 Postgres 
 16GB RAM HA 
 Postgres
 Replica 
 16GB RAM
 Another DC Background Jobs
 Processor
 Sidekiq Redis @Inza
108. @JuicymoEN Admin API Wild Hostess 
 Dashboard Supervisor
 Dashboard Walking Biker Riding Biker Organiser Wild Hostess Supervisor E-Shop
 Backend Ruby on Rails App HTTP 
 Reverse Proxy 
 Load Balancer
 (scalable) HTTP 
 Reverse Proxy 
 Cache
 (scalable) HA 
 Postgres 
 16GB RAM HA 
 Postgres
 Replica 
 16GB RAM
 Another DC Background Jobs
 Processor
 Sidekiq Redis @Inza 6 x 2,5 GB RAM 10 x 
 1 GB RAM 2 x 
 16 GB RAM
109. Any Questions? @Inza @JuicymoEN
110. • https://www.flickr.com/photos/59592200@N05/7221589246/ • https://www.flickr.com/photos/60243770@N00/3123775954/ CC Photo Credits