DR. MICROSERVICES
Or How I Learned to Stop Worrying and Love the API
PyCon 2017
Ryan Anguiano
Slide 2
Slide 2 text
WHERE TO START
• Don't start a new project with microservices
• Hard to pivot from business standpoint
• Core of project should be rigidly defined
• Switch to microservices because you need to,
not because you want to
Slide 3
Slide 3 text
THE MONOLITH
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
THE MONOLITH
• Very large Django app
• Old Python
• Dependency Hell (80+ lines requirements.txt)
• Deployment interrupts entire app
Slide 7
Slide 7 text
BREAKING UP THE
MONOLITH
• Analyze your data flow
• Divide application into logical services
• Leave complicated business logic intact
• Prioritize obvious services (accounts, location, etc)
Slide 8
Slide 8 text
CORE LOGIC A
CORE LOGIC B
CORE LOGIC C
BILLING
ACCOUNTS
LOCATION
EMAIL
CORE LOGIC D
MESSAGING
Slide 9
Slide 9 text
CORE LOGIC A
CORE LOGIC B
CORE LOGIC C
BILLING
ACCOUNTS
LOCATION
EMAIL
CORE LOGIC D
MESSAGING
Slide 10
Slide 10 text
BILLING
ACCOUNTS
LOCATION
EMAIL
MESSAGING
CORE LOGIC A
CORE LOGIC B
CORE LOGIC C
CORE LOGIC D
Slide 11
Slide 11 text
CORE LOGIC A
CORE LOGIC B
CORE LOGIC C
CORE LOGIC D
ACCOUNTS
BILLING
LOCATION
EMAIL
MESSAGING
Slide 12
Slide 12 text
CORE LOGIC A
CORE LOGIC B
CORE LOGIC C
CORE LOGIC D
ACCOUNTS
BILLING
LOCATION
EMAIL
MESSAGING
Slide 13
Slide 13 text
BUILDING A MIGRATION
ROADMAP
• Evaluate different tools
• Use solutions that best meet your needs
• You can keep your existing project running along
side new services
Slide 14
Slide 14 text
BUILDING A MIGRATION
ROADMAP
Slide 15
Slide 15 text
SERVICES AND API DESIGN
• Standardize two methods of communication
• Synchronous
• Asynchronous
Slide 16
Slide 16 text
SERVICES AND API DESIGN
• Standardize two methods of communication
• Synchronous
• Asynchronous
- HTTP REST
- Kafka Messages
Slide 17
Slide 17 text
SERVICES AND API DESIGN
• The Twelve-Factor App (https://12factor.net)
• Declarative configuration
• Environment portability
• Use as guide, not dogma
Slide 18
Slide 18 text
SERVICES AND API DESIGN
• Do not make breaking changes to API endpoints
• Increment endpoint version or add new endpoint
• Always test every older endpoint version
Slide 19
Slide 19 text
SEPARATING DATA STORES
• Every database should only be accessed by a single
service
• If a database needs to be shared, wrap it in a REST
endpoint
Slide 20
Slide 20 text
SEPARATING DATA STORES
DB
ACCOUNTS
DB
BILLING REPORTING
Slide 21
Slide 21 text
SEPARATING DATA STORES
DB
ACCOUNTS
DB
BILLING REPORTING
TRANSACTIONS
MICROSERVICES TOOLSET
CREATION
• On Django, add a backend_api to request objects
• def my_view(request):
user = request.backend_api.get(
('accounts', 'user')
)
return {'user_id': user['id']}
Slide 24
Slide 24 text
DEVOPS AND
INFRASTRUCTURE DESIGN
• Use Docker to develop in the same environment
as production
• Use Terraform and Ansible for configuration-based
infrastructure
• Use Continuous Integration to automate
deployments
Slide 25
Slide 25 text
CENTRALIZED DATA PIPELINE
• Apache Kafka™
• Confluent Open Source Platform
• All data has a schema
• Producers, Consumers and Connectors
Slide 26
Slide 26 text
WEB APP
DB
Slide 27
Slide 27 text
WEB APP
DB
CACHE
WORKERS
REPORTING
SEARCH
Slide 28
Slide 28 text
WEB APP
DB
CACHE
REPORTING
SEARCH
WORKERS
Slide 29
Slide 29 text
DATA PIPELINE
WEB APP
DB
WORKERS
REPORTING
SEARCH ETL
CACHE
INDEX
Slide 30
Slide 30 text
DB
CACHE
DATA PIPELINE
WEB APP
REPORTING INDEX
WORKERS SEARCH ETL
Slide 31
Slide 31 text
LOGGING AND ANALYTICS
• Send all logs into data pipeline
• Send all Docker logs into Kafka using logspout
• Send all syslog into Kafka using Kafka Connect
Slide 32
Slide 32 text
LOGGING AND ANALYTICS
• Generate Correlation IDs (CIDs) on all initial
actions
• Use CIDs in logs and all communications between
services
• CIDs can be used to trace issues across entire
distributed system