Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Rocky Road from Monolithic to Microservice...
Search
cnu
November 25, 2017
Programming
0
1k
The Rocky Road from Monolithic to Microservices Architecture
The explanation of our Microservices architecture and the lessons we learnt from it.
cnu
November 25, 2017
Tweet
Share
More Decks by cnu
See All by cnu
Redisconf 2018: Probabilistic Data Structures
cnu
1
970
Probabilistic Data Structures
cnu
0
630
AWS Lambda - Pycon India 2016
cnu
0
500
ZeroMQ - PyCon India 2013
cnu
2
1.5k
Other Decks in Programming
See All in Programming
A2A プロトコルを試してみる
azukiazusa1
2
1.4k
ニーリーにおけるプロダクトエンジニア
nealle
0
840
猫と暮らす Google Nest Cam生活🐈 / WebRTC with Google Nest Cam
yutailang0119
0
120
PicoRuby on Rails
makicamel
2
130
Azure AI Foundryではじめてのマルチエージェントワークフロー
seosoft
0
170
第9回 情シス転職ミートアップ 株式会社IVRy(アイブリー)の紹介
ivry_presentationmaterials
1
320
すべてのコンテキストを、 ユーザー価値に変える
applism118
3
1.3k
システム成長を止めない!本番無停止テーブル移行の全貌
sakawe_ee
1
200
Result型で“失敗”を型にするPHPコードの書き方
kajitack
5
660
#kanrk08 / 公開版 PicoRubyとマイコンでの自作トレーニング計測装置を用いたワークアウトの理想と現実
bash0c7
1
770
PHP 8.4の新機能「プロパティフック」から学ぶオブジェクト指向設計とリスコフの置換原則
kentaroutakeda
2
900
スタートアップの急成長を支えるプラットフォームエンジニアリングと組織戦略
sutochin26
1
5.8k
Featured
See All Featured
Optimizing for Happiness
mojombo
379
70k
The Straight Up "How To Draw Better" Workshop
denniskardys
234
140k
A Modern Web Designer's Workflow
chriscoyier
695
190k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
6
300
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Fireside Chat
paigeccino
37
3.5k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
The World Runs on Bad Software
bkeepers
PRO
69
11k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
20
1.3k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Transcript
THE ROCKY ROAD FROM MONOLITHIC TO MICROSERVICES ARCHITECTURE
THE ROCKY ROAD FROM MONOLITHIC TO MICROSERVICES ARCHITECTURE
SRINIVASAN RANGARAJAN Head of Product Engineering
SRINIVASAN RANGARAJAN https://cnu.name Twitter: @cnu Github: @cnu
RETAIL AUTOMATION PRODUCT
Catalog & User Events Processing Recommendation
MONOLITHIC ARCHITECTURE
MONOLITHIC ARCHITECTURE Image Processing API Image Searcher File Storage
MINIMUM TWO SERVERS BEHIND ELB Load Balancer
CLOUD VS BARE METAL
EXPENSIVE
NOT REALTIME
NOT PERSONALIZABLE
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
GOTHAM
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
JOKER • Convert client’s catalog into one common MAD Format
• Normalization of fields and metadata • Can process batch and streaming data • Major cause of chaos in the system
GORDON • Routes the product metadata to the right micro
services • Is it a new product? or update to an existing product? • Streaming data from AWS SQS
WONDER WOMAN • Not a microservice, But a tool used
to generate rules for the catalog • Rules are send to the Image Processing microservice • Works on Samples of data and not entire dataset
WATCHTOWER • Central Source of Truth for all metadata •
Backed by an RDBMS Database (Postgresql) • Input via SQS and REST API • Output via REST API
INGESTION Gordon Joker Joker Joker Wonder Woman Watchtower Next Stage
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
NIGHTWING • Computer Vision and Deep Learning Models • Convert
Image to high dimensional vectors • Tag image with visual attributes • Computer Intensive
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
BATMAN • Custom very fast Vector Indexer and Search Engine
• Stores everything in memory • Two sub-parts: Indexer and Searcher • Store binary information about image in DynamoDB
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
SUPERMAN • User behaviour based recommendation • Multiple products like
Collaborative filtering, Cross Product recommendation • Records every user event data and stores in a data warehouse
TWO FACE • Individual User level Personalization • Shows a
different “face” to each user • Dynamic and realtime
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
FLASH • Very fast data structure storage - redis instance
• User session level history, Product Availability, etc. • Fast access, but non- expirable
GCPD • “Global Cache for Products Digested” • Rough first
level of cache for the results
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
ROBIN • API Gateway for all our products • Combines
data from other micro services like Batman, Two Face, Watchtower, Superman, etc and returns JSON Response
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine Joker, Gordon, Wonder Woman Nightwing Batman Robin Watchtower, GCPD, Flash Superman, Two Face
LESSONS WE LEARNT
START WITH A MONOLITH. CHIP OFF PIECES AND BUILD THE
MICROSERVICES. Lesson 0
–Melvin Conway “… organizations which design systems ... are constrained
to produce designs which are copies of the communication structures of these organizations."
DEPLOY HETEROGENOUS MICROSERVICES IN A SINGLE SERVER Lesson 1
Compute Optimized Server Memory Optimized Server Nightwing Batman Robin Robin
Robin Joker Joker Joker Gordon Watch tower Watch tower Joker
IMMUTABLE MICROSERVICES Lesson 2
Constable Inspector Assistant Commissioner Commissioner
ASYNCHRONOUS IS BETTER THAN SYNCHRONOUS Lesson 3
None
NOT ALL MICROSERVICES NEED TO BE SERVERS Lesson 4
ADD REQUEST ID OR TRANSACTION ID TO DEBUG EASILY Lesson
5
GIVE A CHARACTER TO YOUR MICROSERVICES Lesson 6
None
THANK YOU
http://cnu.name/talks/ @cnu