A Path to Services

Old applications can easily grow out of control. Over the last few years we’ve heard about techniques to organize, improve and simplify our Monolithic Rails applications. Microservices have gotten a lot of attention latest. They’re not a cure-all. However, if implemented properly, microservices can ensure boundaries within an application are respected.

Most microservice implementations involve the addition of a messaging queue like RabbitMQ. But do they have to? What if we could leverage Sidekiq to handle real-time service communication?

This talk will explore how PipelineDeals moved from an 8 year-old monolithic Rails application to a microservice-based architecture with over the past 2 years. It will touch on inter-service communication, building for resiliency, leveraging tools we already know and love, and deciding which parts of the application are worthy of separation.

Transcript

1. A Path to Services Brandon  Hilkert   @brandonhilkert   h1p:/ /brandonhilkert.com
2. Topics When  to  consider  services? How  to  start? Modes  of  communica6on Services  at  PipelineDeals 1 2 3 4 Mantle  gem 5
3. When to consider services? 1
5. Strong  module   boundaries Distribu=on Independent   deployment Eventual   consistency Technology   diversity Opera=onal   complexity
6. …there’s  no  reason  why  a  monolithic   system  shouldn't  have  a  good  modular   structure.  But  many  people  have   observed  that  it  seems  rare,  hence  the   Big  Ball  of  Mud  is  most  common   architectural  pa>ern.  Indeed  this   frustra@on  with  the  common  fate  of   monoliths  is  what's  driven  several  teams   to  microservices.
7. The  trouble  is  that,  with  a  monolithic   system,  it's  usually  pre>y  easy  to  sneak   around  the  barrier.  Doing  this  can  be  a   useful  tac@cal  shortcut  to  geDng   features  built  quickly,  but  done  widely   they  undermine  the  modular  structure   and  trash  the  team's  produc@vity.  PuDng   the  modules  into  separate  services  makes   the  boundaries  firmer,  making  it  much   harder  to  find  these  cancerous   workarounds.
8. It's  important  to  stress  that  it's  perfectly   possible  to  have  firm  module  boundaries   with  a  monolith,  but  it  requires  discipline.   Similarly  you  can  get  a  Big  Ball  of   Microservice  Mud,  but  it  requires  more   effort  to  do  the  wrong  thing.  The  way  I   look  at,  using  microservices  increases  the   probability  that  you'll  get  be>er   modularity.
9. 8 years old Monolith Rails 2 2 hour test runs
11. Let’s do this
12. Modes of communication 2
13. Synchronous VS. ASynchronous
14. Synchronous
15. Synchronous
17. Synchronous
18. ASynchronous
20. How to start? 3
21. How to start? New  Features Refactoring
22. Start small
23. Services at PipelineDeals 4
24. Why email?
25. Asynchronous
26. Email Service
27. Sending email UserMailer.welcome(current_user).deliver Email.to current_user, :user_welcome
28. Email Transport
31. Email background Worker
32. async Email
33. Email service
34. synchronous
36. Synchronous
37. Billing Service
38. Features request
39. Billing service
41. What if 2 services need to listen to an event?
42. Analytics service
48. publisher
49. consumer
52. Requirements Publisher 1 Consumer 2 Pub/Sub 3 Redundancy 4 Resiliency 5
54. sidekiq Publisher 1 Consumer 2 Pub/Sub 3 Redundancy 4 Resiliency 5
55. Mantle gem 5
57. Mantle Message Bus Redis Sidekiq Redis Email Service Web Application
58. publisher initializer
59. publish Messages
60. Consumer initializer
61. listener bin/mantle
62. worker bin/sidekiq -q mantle -q default
63. Message Handler
64. testing
65. listeners
66. Requirements Publisher 1 Consumer 2 Pub/Sub 3 Redundancy 4 Resiliency 5
67. Mantle Message Bus Redis Sidekiq Redis Email Service Web Application X
68. RabbitMQ X X +++
69. publisher
70. Publisher
71. Redis sorted sets Sets w/ score
72. Add to catch-up
73. Last processed Time
74. listener bin/mantle
75. Service Procfile
76. listener I, [2015-09-03T14:44:38.701172 #32204] INFO -- : Configuring Mantle to listen on Redis namespace: sync 2015-09-03T18:44:38.701Z 32204 TID-521kc INFO: Booting Sidekiq 3.4.1 with redis options {:url=>"redis://webservices.jfxjg8.ng. 0001.use1.cache.amazonaws.com", :namespace=>"sync"} I, [2015-09-03T14:44:38.701540 #32204] INFO -- : Connecting to message bus redis: #<Redis client v3.2.1 for redis://pld.jfxjg8.ng.0001.use1.cache.amazonaws.com: 6379/0> I, [2015-09-03T14:44:38.737894 #32204] INFO -- : Catching up from time: 1441305877.258581 I, [2015-09-03T14:44:38.826141 #32204] INFO -- : Subscribing to message bus for ["sync:start", "message:read", "message:archive", "message:unread", "message:send", "message:ignore", "person:create", "lead:create", "contact:create"]
77. Catch-up
79. take aways No  new  infrastructure 1 Common  paIerns  for  push/pop 2 400  lines  of  code 3 BeIer  boundaries 4 Rela6vely  low  complexity 5
80. Questions? Brandon  Hilkert   @brandonhilkert   h1p:/ /brandonhilkert.com h1p:/ /mar=nfowler.com/ar=cles/microservice-­‐trade-­‐offs.html h1ps:/ /github.com/PipelineDeals/mantle