Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Microservices with Python and Flask

Microservices with Python and Flask

This presentation was given at EuroPython on July 11, 2017.

Miguel Grinberg

July 11, 2017
Tweet

More Decks by Miguel Grinberg

Other Decks in Programming

Transcript

  1. Agenda • First hour ◦ Introduction & demo ◦ Microservices

    concepts ◦ Class project design discussion • Remaining time ◦ Step-by-step development of the class project ◦ (Focus is on techniques more than on a particular implementation)
  2. About the MicroFlack Project • MicroFlack is a microservices version

    of Flack ◦ Flack is a chat server app I used in the “Flask at Scale” class I gave at PyCon 2016 • The application lives on 7 (yes, seven!) GitHub repositories • Runs on Python 3.4+ and Docker • Not tied to any specific cloud or container orchestration technology
  3. Deploying MicroFlack to your Laptop • Requirements ◦ 4GB RAM

    (8GB recommended) ◦ Vagrant ◦ VirtualBox ◦ Everything is installed in an Ubuntu 16.04 VM (Windows, Mac, Linux laptops are all OK!) • Deployment commands: git clone https://github.com/miguelgrinberg/microflack_admin cd microflack_admin vagrant up # to create the VM or restart it after shutdown vagrant ssh # to open a shell session on the VM vagrant halt # to shutdown the VM (without destroying it) vagrant snapshot save clean # to save a snapshot with name “clean” vagrant snapshot restore clean --no-provision # to restore the snapshot vagrant destroy # to delete the VM
  4. MicroFlack Features • Five microservices, four HTTP/REST, one WebSocket •

    Each service is a standalone Flask app • Two of the services use MySQL databases • Services run in Docker containers • Services communicate over HTTP, message queue or service registry • All services are load balanced • Services scale independently of each other • Upgrades can be done without downtime
  5. Microservice Benefits • Less complexity (maybe) ◦ Awesome for teams

    with varying degrees of experience • Scaling flexibility • More reliability • Less coupling • More choice • Deploy/upgrade while running So where is performance in this list?
  6. Breaking up a Monolith • Going from monolith to microservices

    is very hard • Several strategies ◦ Microservices only going forward ◦ Break pieces of functionality into microservices over time ◦ Refactor the entire monolith into microservices • In all cases, a base platform needs to be put in place before refactoring work begins • Good automated tests are crucial in avoiding bugs when refactoring functionality into a microservice
  7. Load Balancer • All services are load balanced • You

    think you don’t need load balancing? ◦ Upgrades without downtime require a load balancer for a seamless transition ◦ Autoscaling, A/B testing, green/blue deployments, etc. become possible • Many options ◦ Open source: Nginx, HAProxy, Træfik ◦ As a service: AWS, OpenStack, Azure, Google Cloud, etc. ◦ Serverless: Load balancing and scaling are implicitly done by the cloud operator ▪ The Lambda and API Gateway services on AWS are by far the best in this category ◦ Hardware: F5
  8. Service Registry • A distributed data store that keeps track

    of running services • All running services maintain an entry in the service registry • The load balancer’s configuration is generated and updated from the registry contents • Many great open source projects to choose from: ◦ Etcd (CoreOS) ◦ Consul (HashiCorp) ◦ Zookeeper (Apache) ◦ Eureka (Netflix) ◦ SmartStack (Airbnb) ◦ and more!
  9. Logging • Logs from all the services are typically consolidated

    to a single stream to avoid the complexity of maintaining lots of individual log files • Open source: ◦ ELK stack (elasticsearch + logstash + kibana) ◦ Logspout • Several commercial options: ◦ Papertrail ◦ Splunk ◦ Loggly ◦ and more!
  10. Containers • Provide an additional layer of isolation over processes

    • Each container runs a virtualized host ◦ You can have containers using different Linux distros on the same host ◦ Dependencies that would conflict if installed on the same host can be installed on containers ◦ Virtualized network ports • A container maps to one main process, but there can be additional tasks • Not a required component of the stack, but very convenient
  11. Application State • Stateful services are hard to manage ◦

    No way to avoid them for service registry, databases, message queues, etc. • Stateless services are easily scalable, replaceable and disposable • Application-level services should ideally be stateless, and should use stateful services for storage • The state associated with a service should be private ◦ Each service must use its own database ◦ This prevents coupled services that are difficult to upgrade independently of each other ◦ Database joins across services must be done in the application
  12. Retries • Distributed systems have a lot of moving parts

    • It is always a good idea to implement retries for all network calls • The “exponential backoff” strategy works nicely in this environment
  13. The “Backwards-Compatible” Rule • Changes to services must be backwards

    compatible ◦ Why? Because a distributed system cannot be updated atomically without downtime • Database migrations must not break any code that may still be deployed ◦ Phased micro-upgrades can help with column renames or deletes, constraints, etc. • API changes must not break any code that may still be deployed ◦ Why? Need a way to upgrade the API consumer and producer independently • Complex changes that span several services must be “orchestrated” so that they can be applied as micro-deployments without breaking the system
  14. Lifecycle of a Microservice • On startup, the microservice registers

    with the service registry, or is “discovered” by it • The load balancer watches the registry and updates itself to include the new microservice • The new service starts receiving traffic from the load balancer • If more than one instance of the service exist, the traffic is split among them • The service sends “keep-alive” signals, or responds to periodic health checks • When the service is stopped, or stops sending keep-alives, or fails a health check, it is removed from the registry, and in turn from the load balancer
  15. Recommended reading: The Twelve-Factor App https://12factor.net • Codebase • Dependencies

    • Config • Backing services • Build, release, run • Processes • Port binding • Concurrency • Disposability • Dev/prod parity • Logs • Admin Processes
  16. Choosing a Stack • For the services: ◦ Flask and

    friends • For the platform: ◦ Load balancer: HAProxy ◦ Service registry: Etcd ◦ Confd (http://www.confd.io/) configures the load balancer dynamically ◦ Docker containers ◦ Logspout log consolidation ◦ MySQL databases ◦ Redis message queue
  17. Leveraging Public Container Images • miguelgrinberg/easy-etcd ◦ Container image that

    deploys etcd clusters • miguelgrinberg/easy-lb-haproxy ◦ Load balancer + confd preconfigured • gliderlabs/logspout ◦ Consolidated log output • mysql:5.7 ◦ Official Docker image for MySQL 5.7 • redis:3.2-alpine ◦ Official Docker image for Redis 3.2
  18. Administration Scripts: microflack_admin • Vagrantfile: deploy to a single-node Vagrant

    VM • setup-host.sh, setup-all-in-one.sh, make-db-passwords.sh: deploy scripts • mfvars: common environment variables • mfclone: clone the repositories for all the services • mfbuild: build Docker images for services • mfrun: start services • mfkill: stop services • mflogs: consolidated log stream of all services • mfupgrade: upgrade services • mfenv: generate a .env file with environment needed for development • mfdev: attach a locally running service to a deployed system for debugging • etcd-dump: dump the contents of the service registry to the console
  19. Common Code: microflack_common • There is some functionality that all

    microservices need ◦ Service registration ◦ Unit testing helpers ◦ Inter-service communication ◦ Authentication handlers • We’ll use a Python package that services can install with pip • Easy option: install from pypi (if you don’t mind making it public) • Less easy option: private package installed from a local file ◦ We’ll use the Python wheel format for this (pip install wheel) ◦ The --find-links option in pip can install packages from a local file system directory ◦ The mkwheel script builds the wheel packages
  20. Authentication: Let’s switch to JWTs • Tokens stored in a

    database are inconvenient ◦ Services would need to send a request to the tokens service for verification • JSON Web Tokens (JWTs) can be verified just with cryptography ◦ A JWT token stores data inside it, such as a username or id ◦ When the token is generated, a cryptographic signature is added to it ◦ Signature can only be generated or verified if you have a secret key ◦ The data in a token can be trusted only if the token has a valid signature ◦ Not everything is great with JWTs: token revocations become harder • Since tokens are opaque, switching to JWT is not a breaking change • Beware of JWT exploits: always set and check signing algorithm
  21. MicroFlack Service Boundaries Microservice URL(s) User interface / /static/... Users

    /api/users /api/users/:id Tokens /api/tokens Messages /api/messages /api/messages/:id Socket.IO /socket.io
  22. UI Service Summary • Endpoints Method Endpoint Authentication Description GET

    / None Client HTML page GET /static/app.js None Main client application code GET /static/*.js None Client application code GET /static/*.css None Client application stylesheets
  23. UI Service: microflack_ui • app.py, config.py, templates/, static/, requirements.txt ◦

    Ordinary Flask app that serves the index HTML page plus all the JavaScript and CSS files that make up the client application ◦ To ease the transition, at this stage we’ll use an older version of the UI that does not use Socket.IO (we’ll add Socket.IO later) • .env ◦ Environment variables ◦ This file should not be added to source control, as it can contain secrets • tests.py, tox.ini ◦ Unit tests, code coverage and linting • Dockerfile, boot.sh, build.sh ◦ Docker support
  24. Running the UI service • vagrant ssh (connect to the

    VM) • mfkill all (reset your VM to an initial state without any services) ◦ Watch the load balancer at http://192.168.33.10/stats • cd ~/microflack_ui • git checkout 1 (get version 1 of the UI service) • ./build.sh (build the service) • mfrun ui (run the service) • Connect to the application at http://192.168.33.10 ◦ Browser errors are expected, as no other services are yet running
  25. Incorporating the Flack Monolith • The service registry has a

    simple tree structure (use etcd-dump to see it) • You can register the Flack monolith with the load balancer in the VM: ◦ curl -X PUT $ETCD/v2/keys/services/monolith/location -d value="/api" ◦ curl -X PUT $ETCD/v2/keys/services/monolith/upstream/server -d value="10.0.2.2:5000" ◦ Note: 10.0.2.2 is the IP address the host machine has inside a vagrant VM • Now the UI is served by the new microservice, while everything else comes from the old Flack+Celery application • To remove: ◦ curl -X DELETE $ETCD/v2/keys/services/monolith?recursive=true
  26. Users Service Summary • Endpoints (new endpoints in red) Method

    Endpoint Authentication Description POST /api/users None Register a new user GET /api/users Token Optional Get list of users GET /api/users/:id Token Optional Get user by id PUT /api/users/:id Token Modify user by id GET /api/users/me Basic Authenticate user
  27. Users service: microflack_users • Same basic structure as the ui

    microservice • Includes User model and all /api/users endpoints from original Flack • Token authentication imported from microflack_common • Ported existing unit tests and used them as a guide to fix everything up • Add database migration support (Flask-Migrate) ◦ Databases are created by mfrun if they don’t exist yet ◦ Migrations are executed in the container startup script • Add new /api/users/me endpoint to validate username and password and return user information
  28. Running the Users service • cd ~/microflack_users • git checkout

    1 (select version 1 of the service) • ./build.sh (build the service) • mfrun users (run the service) ◦ The /api/users family of endpoints should now be working!
  29. Tokens Service Summary • Endpoints (postponed endpoints grayed out) Method

    Endpoint Authentication Description POST /api/tokens Basic Request a token DELETE /api/tokens Token Revoke a token
  30. Token service: microflack_tokens • Just one endpoint copied from the

    monolithic app: /api/tokens • Authentication is relayed to the users service /me endpoint • Generated JWT token contains the numeric user id • No token revocations for now
  31. Running the Tokens service • cd ~/microflack_tokens • git checkout

    1 (select version 1 of the service) • ./build.sh (build the service) • mfrun tokens (run the service) ◦ The /api/tokens endpoint should now be working
  32. Messages Service Summary • Endpoints Method Endpoint Authentication Description POST

    /api/messages Token Post a new message GET /api/messages Token Optional Get list of messages GET /api/messages/:id Token Optional Get message by id PUT /api/messages/:id Token Modify message by id
  33. Messages Service: microflack_messages • Structure based on the user service

    • Models and endpoints copied from original Flack code • Removed all asynchronous functions for now ◦ We want a basic app up and running from which we can build on • Ported unit tests to verify the code works
  34. Running the Messages service • cd ~/microflack_messages • git checkout

    1 (select version 1 of the service) • ./build.sh (build the service) • mfrun messages (run the service) • The application should be fully functional (though not very performant yet)
  35. MicroFlack v0.5: Async message rendering • Original Flack used Celery

    for asynchronous message rendering ◦ Unfortunately, Celery workers are by design tightly coupled with the caller process • Instead of Celery, we will use background threads for rendering ◦ Our render task is not CPU intensive, so this works very well ◦ For CPU intensive tasks, the multiprocessing module can be used instead ◦ If a very high volume of tasks must be supported, an asynchronous server can be used
  36. Development Workflow: Common Package • Build the common packages locally:

    ◦ cd microflack_admin; source mfvars; cd .. ◦ git clone https://github.com/miguelgrinberg/microflack_common ◦ cd microflack_common ◦ ./mkwheel all ◦ cd ..
  37. Development Workflow: Running a Service • Set up the source

    code for the desired microservice: ◦ cd microflack_admin; source mfvars; cd .. ◦ git clone https://github.com/miguelgrinberg/microflack_messages ◦ cd microflack_messages ◦ python3 -m venv venv ◦ source venv/bin/activate ◦ pip install -r requirements.txt ◦ flask run • Test by sending requests with curl, httpie, postman, etc. • For integration testing with an actual system running in a VM: ◦ Create .env file (run mfenv inside VM to get the variables you need) ◦ mfdev start messages ◦ flask run
  38. Upgrading the Messages service • cd ~/microflack_messages • git checkout

    2 (select version 2 of the service) • ./build.sh (build the service) • mfupgrade try messages (start upgrade) ◦ After watching the log for a few seconds, hit Ctrl-C • mfupgrade roll (rolling upgrade)
  39. Tokens Service Summary • Endpoints (new endpoints in red) Method

    Endpoint Authentication Description POST /api/tokens Basic Request a token GET /api/tokens Token Check if a token is revoked DELETE /api/tokens Token Revoke a token
  40. Token revocation • We need to maintain a list of

    revoked tokens ◦ The best place to implement this is the tokens service ◦ Revoked tokens need to be kept in a list only until they expire ◦ We can keep the list in etcd, and write all entries with the appropriate expiration • Services need to check tokens against that list ◦ We can encapsulate this inside the verify_token function in microflack_common ◦ Option 1 (more correct): send a request to the tokens service to check revocation status ◦ Option 2 (more performant): check the list in etcd directly ◦ Improvement for both options: cache calls to verify_token
  41. Upgrading the Tokens service • cd ~/microflack_tokens • git checkout

    2 (select version 2 of the service) • ./build.sh (build the service) • mfupgrade roll tokens (rolling upgrade, skipping the “try” step) • Upgrade the services that work with tokens: ◦ users to version 2 ◦ messages to version 3
  42. Socket.IO Service Summary • Socket.IO client to server events •

    Socket.IO server to client events (new in red) Event Authentication Description ping_user Token Mark a user as online post_message Token Post a message disconnect Token (from session) Mark the user as offline Event Description updated_model Render updated user or message expired_token Ask user to log in again
  43. System changes for Socket.IO • Task list for the Socket.IO

    service ◦ Implement “ping_user”, “post_message” and “disconnect” events ◦ Push “expired_token” notifications to clients when appropriate • Task list for the common package ◦ Add support for setting sticky sessions in the load balancer • Task list for the UI service ◦ Add Socket.IO support ◦ Handle expired tokens in Socket.IO calls (bug in old version) • Task list for users service ◦ Add “ping” and “user offline” endpoints ◦ Push “updated_model” notifications to message queue • Task list for messages service ◦ Push “updated_model” notifications to message queue
  44. Socket.IO service: microflack_socketio • Same structure as a no-database service

    • No HTTP endpoints, only the three Socket.IO events • Needs to be an async service due to the long term WebSocket connections ◦ Don’ t understand why? Come to my talk “Asynchronous Python” on Sunday! ◦ We have several options ▪ python-socketio supports WSGI-compatible async frameworks (eventlet, gevent) and also asyncio ▪ Flask-SocketIO builds on python-socketio, but drops asyncio support ◦ While we don’t need Flask for this service, having access to Flask’s user session is handy ◦ We’ll go with Flask-SocketIO and eventlet for this service
  45. Users Service Summary • Endpoints (new endpoints in red) Method

    Endpoint Authentication Description POST /api/users None Register a new user GET /api/users Token Optional Get list of users GET /api/users/:id Token Optional Get user by id PUT /api/users/:id Token Modify user by id GET /api/users/me Basic Authenticate user PUT /api/users/me Token Set user online DELETE /api/users/me Token Set user offline
  46. Running the Socket.IO service • cd ~/microflack_socketio • git checkout

    1 (select version 1 of the service) • ./build.sh (build the service) • mfrun socketio (run the service) • Upgrade the services that work with Socket.IO (order is important!): ◦ users to version 3 ◦ messages to version 4 ◦ ui to version 2
  47. Kubernetes • Open-source container orchestration, from Google • Manages a

    cluster of nodes (container hosts) transparently • Works with Docker images • Has its own service registry and load balancer • Gives each service a DNS name (i.e. http://users connects to the Users service) • Stores secrets securely • Handles service replication, and does rolling upgrades • microflack_admin includes example Kubernetes deployment scripts in install/kubernetes
  48. Amazon ECS • A cluster of Docker hosts running on

    EC2 instances • Access to the AWS ecosystem • MicroFlack platform and application containers can run without change • Some effort required in configuring roles and security groups (as with everything done on AWS)
  49. AWS Serverless (Lambda & API Gateway) • Very different paradigm,

    only the application logic is uploaded to AWS ◦ No need for gunicorn, just the application code that handles the endpoints ◦ Tools like Zappa (or my own Slam) enable transparent support for WSGI apps • AWS Lambda provides automatic load balancing and auto-scaling • Access to the AWS ecosystem • Cons: ◦ No WebSocket support, since there is no server running all the time ◦ Response times are not great
  50. Good ol’ Processes • Deploying MicroFlack on a host without

    Docker is possible • All the core components of the platform can be installed without Docker: ◦ etcd, haproxy, confd, mysql, redis • The MicroFlack application services can run as regular gunicorn processes ◦ But a network port assignment strategy needs to be implemented
  51. Improvement Ideas • Revoke all tokens for a user, and

    revoke all tokens for everybody • Use multiple chat rooms instead of just one (this will require some client-side work as well) • Protect the /stats and /logs endpoints with authentication • Add SSL termination to (or in front of) the load balancer • Deploy the ELK stack and configure the logspout container to forward logs to it • Add a secrets store component (maybe HashiCorp’s Vault) • Create a multi-host deployment, possibly with redundant load balancers • Replace the single-node Redis deployment with a Redis or RabbitMQ cluster • Replace the single-node MySQL with a Galera cluster • Replace MySQL with NoSQL databases of your choice • With all services running 3+ instances, create a “chaos monkey” script • Implement a platform service that recycles application services that die or are unresponsive, and maybe even auto-scales them based on load