Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Rocky Road from Monolithic to Microservice...
Search
cnu
November 25, 2017
Programming
0
1.1k
The Rocky Road from Monolithic to Microservices Architecture
The explanation of our Microservices architecture and the lessons we learnt from it.
cnu
November 25, 2017
Tweet
Share
More Decks by cnu
See All by cnu
Redisconf 2018: Probabilistic Data Structures
cnu
1
990
Probabilistic Data Structures
cnu
0
650
AWS Lambda - Pycon India 2016
cnu
0
520
ZeroMQ - PyCon India 2013
cnu
2
1.5k
Other Decks in Programming
See All in Programming
コードとあなたと私の距離 / The Distance Between Code, You, and I
hiro_y
0
190
理論と実務のギャップを超える
eycjur
0
170
「ちょっと古いから」って避けてた技術書、今だからこそ読もう
mottyzzz
12
7.1k
TransformerからMCPまで(現代AIを理解するための羅針盤)
mickey_kubo
6
3.1k
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
570
AI 駆動開発におけるコミュニティと AWS CDK の価値
konokenj
4
140
AkarengaLT vol.38
hashimoto_kei
1
110
Android16 Migration Stories ~Building a Pattern for Android OS upgrades~
reoandroider
0
130
CSC305 Lecture 10
javiergs
PRO
0
220
3年ぶりにコードを書いた元CTOが Claude Codeと30分でMVPを作った話
maikokojima
0
620
登壇は dynamic! な営みである / speech is dynamic
da1chi
0
350
kiroとCodexで最高のSpec駆動開発を!!数時間で web3ネイティブなミニゲームを作ってみたよ!
mashharuki
0
860
Featured
See All Featured
Large-scale JavaScript Application Architecture
addyosmani
514
110k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
285
14k
Gamification - CAS2011
davidbonilla
81
5.5k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
RailsConf 2023
tenderlove
30
1.3k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
34
2.3k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.2k
The Cost Of JavaScript in 2023
addyosmani
55
9.1k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Scaling GitHub
holman
463
140k
Designing for humans not robots
tammielis
254
26k
Learning to Love Humans: Emotional Interface Design
aarron
274
41k
Transcript
THE ROCKY ROAD FROM MONOLITHIC TO MICROSERVICES ARCHITECTURE
THE ROCKY ROAD FROM MONOLITHIC TO MICROSERVICES ARCHITECTURE
SRINIVASAN RANGARAJAN Head of Product Engineering
SRINIVASAN RANGARAJAN https://cnu.name Twitter: @cnu Github: @cnu
RETAIL AUTOMATION PRODUCT
Catalog & User Events Processing Recommendation
MONOLITHIC ARCHITECTURE
MONOLITHIC ARCHITECTURE Image Processing API Image Searcher File Storage
MINIMUM TWO SERVERS BEHIND ELB Load Balancer
CLOUD VS BARE METAL
EXPENSIVE
NOT REALTIME
NOT PERSONALIZABLE
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
GOTHAM
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
JOKER • Convert client’s catalog into one common MAD Format
• Normalization of fields and metadata • Can process batch and streaming data • Major cause of chaos in the system
GORDON • Routes the product metadata to the right micro
services • Is it a new product? or update to an existing product? • Streaming data from AWS SQS
WONDER WOMAN • Not a microservice, But a tool used
to generate rules for the catalog • Rules are send to the Image Processing microservice • Works on Samples of data and not entire dataset
WATCHTOWER • Central Source of Truth for all metadata •
Backed by an RDBMS Database (Postgresql) • Input via SQS and REST API • Output via REST API
INGESTION Gordon Joker Joker Joker Wonder Woman Watchtower Next Stage
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
NIGHTWING • Computer Vision and Deep Learning Models • Convert
Image to high dimensional vectors • Tag image with visual attributes • Computer Intensive
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
BATMAN • Custom very fast Vector Indexer and Search Engine
• Stores everything in memory • Two sub-parts: Indexer and Searcher • Store binary information about image in DynamoDB
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
SUPERMAN • User behaviour based recommendation • Multiple products like
Collaborative filtering, Cross Product recommendation • Records every user event data and stores in a data warehouse
TWO FACE • Individual User level Personalization • Shows a
different “face” to each user • Dynamic and realtime
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
FLASH • Very fast data structure storage - redis instance
• User session level history, Product Availability, etc. • Fast access, but non- expirable
GCPD • “Global Cache for Products Digested” • Rough first
level of cache for the results
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine
ROBIN • API Gateway for all our products • Combines
data from other micro services like Batman, Two Face, Watchtower, Superman, etc and returns JSON Response
API Data Store Ingestion Image Processing Image Searcher User Event
Personalization Engine Joker, Gordon, Wonder Woman Nightwing Batman Robin Watchtower, GCPD, Flash Superman, Two Face
LESSONS WE LEARNT
START WITH A MONOLITH. CHIP OFF PIECES AND BUILD THE
MICROSERVICES. Lesson 0
–Melvin Conway “… organizations which design systems ... are constrained
to produce designs which are copies of the communication structures of these organizations."
DEPLOY HETEROGENOUS MICROSERVICES IN A SINGLE SERVER Lesson 1
Compute Optimized Server Memory Optimized Server Nightwing Batman Robin Robin
Robin Joker Joker Joker Gordon Watch tower Watch tower Joker
IMMUTABLE MICROSERVICES Lesson 2
Constable Inspector Assistant Commissioner Commissioner
ASYNCHRONOUS IS BETTER THAN SYNCHRONOUS Lesson 3
None
NOT ALL MICROSERVICES NEED TO BE SERVERS Lesson 4
ADD REQUEST ID OR TRANSACTION ID TO DEBUG EASILY Lesson
5
GIVE A CHARACTER TO YOUR MICROSERVICES Lesson 6
None
THANK YOU
http://cnu.name/talks/ @cnu