Ryan Anguiano - Dr. Microservices, Or How I Learned to Stop Worrying and Love the API

Assuming that you already know how to build a monolithic app, you must be wondering how you can use all this "microservice" stuff that you keep hearing about. Well, a good word of advice is that you probably don't need it. If designed properly, a monolithic app should be able to scale and fit the needs of most businesses. Even so, you should keep your development as simple as possible until you have proven and solidified your business concepts. But if you do need to grow to Internet scale, then you have a long road ahead of you.

Moving from a monolithic application to microservices is a natural evolution that is often of necessity. There are several competing schools of thought that are still being battle-tested in these early days of microservice architecture. Among all the competing paradigms, most of the requirements can be agreed upon, but are mostly differentiated by the tools used to fulfill the requirements.

This talk will cover setting up the required infrastructure, and demonstrate how to migrate a sample monolithic Django application into a microservices platform.

The demo application will use the following technologies: Django, Flask, Fabric, Terraform, Ansible, CentOS, Docker, Mesos, Consul, Nginx, Pgbouncer, Kafka

https://us.pycon.org/2017/schedule/presentation/356/

Transcript

1. DR. MICROSERVICES Or How I Learned to Stop Worrying and Love the API PyCon 2017 Ryan Anguiano
2. WHERE TO START • Don't start a new project with microservices • Hard to pivot from business standpoint • Core of project should be rigidly defined • Switch to microservices because you need to, 
 not because you want to
3. THE MONOLITH
6. THE MONOLITH • Very large Django app • Old Python • Dependency Hell (80+ lines requirements.txt) • Deployment interrupts entire app
7. BREAKING UP THE MONOLITH • Analyze your data flow • Divide application into logical services • Leave complicated business logic intact • Prioritize obvious services (accounts, location, etc)
8. CORE LOGIC A CORE LOGIC B CORE LOGIC C BILLING ACCOUNTS LOCATION EMAIL CORE LOGIC D MESSAGING
9. CORE LOGIC A CORE LOGIC B CORE LOGIC C BILLING ACCOUNTS LOCATION EMAIL CORE LOGIC D MESSAGING
10. BILLING ACCOUNTS LOCATION EMAIL MESSAGING CORE LOGIC A CORE LOGIC B CORE LOGIC C CORE LOGIC D
11. CORE LOGIC A CORE LOGIC B CORE LOGIC C CORE LOGIC D ACCOUNTS BILLING LOCATION EMAIL MESSAGING
12. CORE LOGIC A CORE LOGIC B CORE LOGIC C CORE LOGIC D ACCOUNTS BILLING LOCATION EMAIL MESSAGING
13. BUILDING A MIGRATION ROADMAP • Evaluate different tools • Use solutions that best meet your needs • You can keep your existing project running along side new services
14. BUILDING A MIGRATION ROADMAP
15. SERVICES AND API DESIGN • Standardize two methods of communication • Synchronous • Asynchronous
16. SERVICES AND API DESIGN • Standardize two methods of communication • Synchronous • Asynchronous - HTTP REST - Kafka Messages
17. SERVICES AND API DESIGN • The Twelve-Factor App (https://12factor.net) • Declarative configuration • Environment portability • Use as guide, not dogma
18. SERVICES AND API DESIGN • Do not make breaking changes to API endpoints • Increment endpoint version or add new endpoint • Always test every older endpoint version
19. SEPARATING DATA STORES • Every database should only be accessed by a single service • If a database needs to be shared, wrap it in a REST endpoint
20. SEPARATING DATA STORES DB ACCOUNTS DB BILLING REPORTING
21. SEPARATING DATA STORES DB ACCOUNTS DB BILLING REPORTING TRANSACTIONS
22. MICROSERVICES TOOLSET CREATION • Auth • Messaging • Log Settings • Correlation IDs • Middleware
23. MICROSERVICES TOOLSET CREATION • On Django, add a backend_api to request objects • def my_view(request):
 user = request.backend_api.get(
 ('accounts', 'user')
 )
 return {'user_id': user['id']}
24. DEVOPS AND INFRASTRUCTURE DESIGN • Use Docker to develop in the same environment as production • Use Terraform and Ansible for configuration-based infrastructure • Use Continuous Integration to automate deployments
25. CENTRALIZED DATA PIPELINE • Apache Kafka™ • Confluent Open Source Platform • All data has a schema • Producers, Consumers and Connectors
26. WEB APP DB
27. WEB APP DB CACHE WORKERS REPORTING SEARCH
28. WEB APP DB CACHE REPORTING SEARCH WORKERS
29. DATA PIPELINE WEB APP DB WORKERS REPORTING SEARCH ETL CACHE INDEX
30. DB CACHE DATA PIPELINE WEB APP REPORTING INDEX WORKERS SEARCH ETL
31. LOGGING AND ANALYTICS • Send all logs into data pipeline • Send all Docker logs into Kafka using logspout • Send all syslog into Kafka using Kafka Connect
32. LOGGING AND ANALYTICS • Generate Correlation IDs (CIDs) on all initial actions • Use CIDs in logs and all communications between services • CIDs can be used to trace issues across entire distributed system
33. THANKS FOR LISTENING Questions?