Distributed microservices in the real world by Imraan Parker

This talk is intended for anyone interested in deploying microservices in their environment. It examines what is involved in developing, deploying and maintaining a distributed microservices architecture. Moving from a monolithic architecture to a services oriented architecture has many benefits and tradeoffs that need to be addressed.

One has to consider whether to go with an off the shelf solution or build your own. When does it make sense to do the latter? Tasks like monitoring and debugging is more difficult with the added complexity that is inherent with a distributed system.

This first part of the talk reviews what needs to be considered when deciding to build applications on a distributed microservices architecture. Topics that will be covered include:

Why choose a microservices architecture, and when not to?
Objectives of a distributed services architecture

Service presence
Heartbeating
Logging
Security
Reliability>
Performance and Scalability
Monitoring
Troubleshooting
Message contracts

The second part of the talk delves into the micorservices framework built and used by CareerJunction. It is an in house framework written in Python 3 (3.5+) which uses asyncio for nonblocking I/O and ZeroMQ as a concurrency networking library.

In simple terms, the framework implements a reliable service-oriented request-reply dialog between a set of client applications, a set of brokers and a set of worker applications.

The features of the framework include those discussed in the first part of the talk and will be showcased by coding and deploying a service. Above and beyond that, the following topics will be covered:

Conceptual architecture
Architecture goals
Coding a service

Creating and running a service
Message contract parameters
Exposing services via HTTP
Scheduled Jobs

The last part of the talk examines the lessons learnt over the past few years, what to avoid and the benefits it brought not only to the IT team, but to the business as a whole.
Topics that will be covered include:

IT benefits
Business benefits
What to avoid?

Transcript

1. Distributed microservices in the real world By Imraan Parker
2. Python 3.6.6 [GCC 7.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> print(objective) print(about) >>> The purpose of this presentation is to pass on our knowledge regarding out implementation of microservices to empower you to make an informed decision based on your own situation. Hi PyConZA. My role is Head of Product. I code to make a difference.
3. @imraanparker Background  CareerJunction (https://www.careerjunction.co.za) is 21 this year  We build products to bring together those looking for jobs and those supplying them  We work primarily in Python to  Perform data capture, mining and analysis  Provide powerful tools for recruitment  Our technology stack is diverse
4. @imraanparker Agenda  Our journey from monolith to microservices  Modern software development principles  Why choose a microservices architecture?  Explore the CareerJunction microservices framework  Requirements  Conceptual architecture  Framework components  Features by coding and running a service  The benefits and considerations  IT and business benefits  What to consider and avoid?
5. @imraanparker Modern Software Development  Resources  How teams are organised  Reporting structure  Processes  How to respond to requirements  How to react to changes  Architecture  How to build software  Standards & guidelines
6. Modern Software Development Architecture Processes Resources Deliver often React faster to change Small autonomous teams
7. Architecture?
8. @imraanparker Application Monolithic Architecture  Easy to  Develop  Test  Deploy  Scale  Over time  Codebase grew  We saw diminishing returns Web UI Web UI Search Module Search Module Job Module Job Module User Module User Module DB HTML HTTP API
9. Monolithic Architecture Monolithic Architecture Processes Resources Deliver often React faster to change Small autonomous teams
10. @imraanparker Monolithic Architecture  What to do?  Break up the monolithic application, and  Implement microservices  Unfortunately we have to keep the planes in the air  Feature development trumps architecture development  We investigated different options, but ultimately came up short  We tried an approach that ultimately failed to live up to expectations  Broke up the monolithic application, into  Smaller applications
11. @imraanparker Web UI Web UI DB HTML HTTP API Search Search Job Job User User HTTP API No control over application data Extra overhead
12. @imraanparker Application Layers User Pylons WSGI Apache
13. @imraanparker Application Layers  Other Issues  Setup and configuration  Request monitoring  Troubleshooting User Pylons WSGI Apache
14. “ ” Insanity… Doing the same thing over and over again and expecting different results. It took us two years to finally bite the bullet and take the time to do it properly
15. @imraanparker Microservices Architecture  We dedicated resources to the project  Investigated available options in the market  Score each based on our requirements  Pickings were slim  A few were feature rich, but expensive and/or verbose  Others had some of the features, meaning we needed two or more systems combined
16. Build it and they will come
17. @imraanparker Microservices Architecture Web UI Web UI DB HTML RPC DB DB Search Service Search Service Job Service Job Service User Service User Service Gateway Gateway HTTP API RPC RPC
18. Microservices Architecture Microservices Architecture Processes Resources Deliver often React faster to change Small autonomous teams  
19. @imraanparker Summary – Our Journey  The monolith is not a bad idea  In the scope of a large scale application, microservices is better  If you are not prepared to do it right the first time, be prepared to do it twice.
20. CareerJunction Framework
21. @imraanparker Requirements  Developer adoption  Increase productivity  Language agnostic  Scalable  Highly available  Discoverable services
22. @imraanparker Requirements  Message contracts  Automated documentation  Task scheduling  Monitoring  Debugging
23. @imraanparker Conceptual Architecture  Technologies  Python 3 (3.4+)  ZeroMQ (http://zeromq.org/)  Messaging  Reliable request-reply dialog  Based on the Majordomo pattern (https://rfc.zeromq.org/spec:7/MDP/)  JSON for serialisation
24. @imraanparker Conceptual Architecture Client Client Client Broker Broker Broker Broker Worker Worker tea Worker Worker coffee Worker Worker water
25. @imraanparker Conceptual Architecture Client Client Client Broker Broker Broker Broker Worker Worker tea Worker Worker coffee Worker Worker water I want tea
26. @imraanparker Conceptual Architecture Client Client Client Broker Broker Broker Broker Worker Worker tea Worker Worker coffee Worker Worker water I want tea
27. @imraanparker Conceptual Architecture Client Client Client Broker Broker Broker Broker Worker Worker tea Worker Worker coffee Worker Worker water I want tea
28. @imraanparker Conceptual Architecture Client Client Client Broker Broker Broker Broker Worker Worker tea Worker Worker coffee Worker Worker water I want tea
29. @imraanparker Conceptual Architecture Client Client Client Broker Broker Broker Broker Worker Worker tea Worker Worker coffee Worker Worker water I want tea
30. @imraanparker Components - Broker  Responsible for  Communication between clients and workers  Exposing services via HTTP  Distribution of requests to workers  Logging of requests
31. @imraanparker Components - Broker  To accomplish this the broker consists of four sub systems  Transport  Queue  Logger  Heartbeat Client Worker Worker tea Broker Transport Queue Logger Heartbeat
32. @imraanparker Components - Broker  Managing connections  Handling data transfer  Broadcasting broker’s presence Client Worker Worker tea Broker Transport Queue Logger Heartbeat
33. @imraanparker Components - Broker  Request distribution  Round-robin fashion  Handles queues for services Client Worker Worker tea Broker Transport Queue Logger Heartbeat
34. @imraanparker Components - Broker  Records all requests  Received -> processing -> complete  Exposes a dashboard for monitoring and debugging Client Worker Worker tea Broker Transport Queue Logger Heartbeat
35. @imraanparker Components - Broker  Monitors connections between clients and workers  Disconnects peers on heartbeat failure Client Worker Worker tea Broker Transport Queue Logger Heartbeat
36. @imraanparker Components - Worker  Responsible for  Connecting to brokers by listening for multicast messages  Registering services (including documentation) with brokers  Support scaling using threads  Processing requests
37. @imraanparker Components - Client  Responsible for  Exposing a RPC type syntax for calling services  Encode and decoding language specific data types  Sending requests to the broker and handling replies  Raising exceptions in a language specific manner on failure
38. Coding a Service
39. @imraanparker Definitions  A worker is an instance of a service  A service is a collection of microservices  A microservice is a method
40. Service Definition
41. Service Definition
42. Calling the Service
43. Service Scaling
44. Asynchronous
45. Minimum Timeout
46. Exposing via HTTP
47. Scheduled Jobs
48. @imraanparker Dashboard  Overall health of the system  Monitor and debug requests  Requests  Summary  Failed  In Progress  Successful
49. @imraanparker Summary
50. @imraanparker Failed Requests
51. Request Information
52. Request Information
53. Request Information
54. @imraanparker In Progress
55. @imraanparker Successful
56. @imraanparker Service Documentation
57. @imraanparker Scheduled Jobs
58. @imraanparker HTTP API
59. Demo
60. @imraanparker Wrap-up  Benefits  Four fold increase in productivity  Lower IT cost  Infrastructure  Maintenance  Human Resources  Reflections  Microservices is best suited for large scale applications  Do it right the first time  Learn from your mistakes, but importantly from ours
61. @imraanparker Getting Involved https://github.com/ careerjunction
62. @imraanparker Final Thoughts  How to deploy services?  How to deal with distributed data?  How to manage feature development that extends over multiple teams?
63. Thank you
64. This will all end in tears. I just know it.