Dorian Pula - Pythons in A Container - Lessons Learned Dockerizing Python Micro-Services

Micro-services and Docker are all the rage for developing scalable systems. But what challenges will you face when developing and deploying Python apps using Docker to production? This talk goes into the real-life lessons learned from creating, deploying and scaling Dockerized Python applications.

https://us.pycon.org/2016/schedule/presentation/2096/

Transcript

1. PYTHONS IN A
   CONTAINER
   LESSONS LEARNED DOCKERIZING PYTHON MICROSERVICES...
   ...THE HARD WAY
    
   
   Presented by /
   Dorian Puła @dorianpula
   

   View Slide

2. INTRODUCTION
   
   

   View Slide

3. WHO AM I?
   So ware Development Engineer @
    Develop eCommerce platform for Loyalty Programs
   (Buy, Gi + Transfer points)
    Flask REST APIs + Apps
    Dockerized microservices
   Open Source
    - Yet Another CMS
    Contributed to Fabric, Ansible & core Python
    Ansible roles for NGINX, UWSGI, NodeJS and Supervisor
   Points
   Rookeries
   

   View Slide

4. WHAT IS THIS TALK ABOUT?
   Lessons learned using Docker for Flask REST API and apps.
   Incorporating various tools that Docker and docker-compose
   provide for better DevOps workflow.
   The usefulness of unlearning some accepted patterns in
   Python development, when working with Docker.
   

   View Slide

5. WHAT IS THIS TALK NOT ABOUT?
   An introduction to basic Docker or WSGI apps.
   Docker Machine (cool as it is).
   Advanced Docker wizardery. See Dockercon next week for that.
   An exposé on why you must or must not use Docker.
   

   View Slide

6. MICROSERVICES + DOCKER
   
   

   View Slide

7. EXAMPLE APP + API - POINTS FOR
   PYTHONISTAS
   Imagine having to build an app for a new hypothetical loyalty
   program for sprint contributers at PyCon.
   Earn points per commit or issue resolved. Redeem points for
   essential sprint goods. (e.g. coffee, poptarts or dogecoin.)
   Has the following components:
   REST API
   Frontend App
   Redemption of Points
   User + Project Registration/Linking
   Database
   

   View Slide

8. WHY A MICROSERVICES
   ARCHITECTURE?
   Imagine implementing said example using a microservices
   architecture, with multiple services built by multiple teams.
   Benefits:
    Smaller less complex codebases.
    Enable independence between codebases & teams.
    More flexible scaling schemes (tech & organizational).
   Drawbacks:
    Distributed codebases harder to infer, and may contain
   implicit inter-service dependencies.
    More complex orchestration, monitoring & provisioning.
   

   View Slide

9. EXAMPLE ARCHITECTURE
   Points
   App + API
   Redeem
   Service
   User + Project
   Registry Service
   Database
   Database
   Dogecoin
   Database
   Bitbucket
   

   View Slide

10. WHY USE DOCKER?
     Containers vs. Virtual machines
    Containers lighter in memory and processing than VMs.
    Isolated user-space instances vs. machine emulation.
    Docker uses cached/immutable layered file systems.
     Tooling for Managing Containers
    Quick spin up of container/environments.
    Easily create, share and publish images to registries.
    Unified workflow that replaces other tools:
    e.g. chroot jails, LXC, Vagrant, etc.
    

    View Slide

11. DEVELOPMENT AND TESTING
    
    

    View Slide

12. DOCKER COMPOSE
    Specify with docker-compose.yaml...
    p
    o
    i
    n
    t
    s
    _
    a
    p
    p
    :
    b
    u
    i
    l
    d
    : .
    p
    o
    r
    t
    s
    :
    ­ "
    5
    0
    0
    0
    :
    5
    0
    0
    0
    "
    e
    n
    v
    i
    r
    o
    n
    m
    e
    n
    t
    :
    ­ A
    P
    I
    _
    K
    E
    Y
    =
    M
    Y
    _
    S
    U
    P
    E
    R
    _
    S
    E
    C
    R
    E
    T
    _
    K
    E
    Y
    h
    o
    s
    t
    n
    a
    m
    e
    : a
    p
    p
    l
    i
    n
    k
    s
    :
    "
    c
    o
    u
    c
    h
    d
    b
    :
    c
    o
    u
    c
    h
    "
    c
    o
    u
    c
    h
    d
    b
    :
    i
    m
    a
    g
    e
    : c
    o
    u
    c
    h
    d
    b
    p
    o
    r
    t
    s
    :
    ­ "
    5
    9
    8
    4
    :
    5
    9
    8
    4
    "
    v
    o
    l
    u
    m
    e
    s
    :
    ­ d
    a
    t
    a
    :
    /
    u
    s
    r
    /
    l
    o
    c
    a
    l
    /
    v
    a
    r
    /
    l
    i
    b
    /
    c
    o
    u
    c
    h
    d
    b
    o
    t
    h
    e
    r
    _
    s
    e
    r
    v
    i
    c
    e
    s
    : .
    .
    .
    ...and start up with:
    d
    o
    c
    k
    e
    r
    ­
    c
    o
    m
    p
    o
    s
    e u
    p
    

    View Slide

13. DOCKER WORKFLOW
    Docker + Compose replaces a Vagrant + VM workflow
    vagrant up + vagrant ssh + run $app_command  docker run
    $app_command
    vagrant halt  docker stop
    vagrant status  docker ps
    vagrant provision  docker build
    vagrant destroy  docker stop + docker rm
    vagrant box list, remove  docker images, docker rmi
    

    View Slide

14. BUILDING GOOD DOCKER IMAGES
     Sample Dockerfile
    F
    R
    O
    M u
    b
    u
    n
    t
    u
    :
    1
    6
    .
    0
    4
    R
    U
    N a
    p
    t
    ­
    g
    e
    t u
    p
    d
    a
    t
    e &
    & a
    p
    t
    ­
    g
    e
    t i
    n
    s
    t
    a
    l
    l ­
    y p
    y
    t
    h
    o
    n p
    y
    t
    h
    o
    n
    ­
    d
    e
    v g
    c
    c \
    p
    y
    t
    h
    o
    n
    ­
    p
    i
    p p
    y
    t
    h
    o
    n
    ­
    s
    e
    t
    u
    p
    t
    o
    o
    l
    s
    A
    D
    D w
    s
    g
    i
    _
    a
    p
    p /
    a
    p
    p
    W
    O
    R
    K
    D
    I
    R /
    a
    p
    p
    R
    U
    N p
    i
    p i
    n
    s
    t
    a
    l
    l ­
    r r
    e
    q
    u
    i
    r
    e
    m
    e
    n
    t
    s
    .
    t
    x
    t &
    & p
    i
    p i
    n
    s
    t
    a
    l
    l u
    w
    s
    g
    i
    C
    M
    D u
    w
    s
    g
    i ­
    ­
    h
    t
    t
    p :
    5
    0
    0
    0 ­
    ­
    m
    a
    s
    t
    e
    r ­
    ­
    p
    r
    o
    c
    e
    s
    s
    e
    s 4 ­
    ­
    w
    s
    g
    i
    ­
    f
    i
    l
    e a
    p
    p
    _
    w
    s
    g
    i
    .
    p
    y
    # C
    M
    D p
    y
    t
    h
    o
    n a
    p
    p
    _
    w
    s
    g
    i
    .
    p
    y
    E
    X
    P
    O
    S
    E 5
    0
    0
    0
    Each step in a Dockerfile can create a new layer in filesystem.
    Minimize steps number of separate RUN steps.
    Try to make layers cacheable:
    Cached layer reused if no checksum change in source.
    Use base images for heavily repeated steps.
    See ONBUILD command for making dynamic base images.
    Expose ports and volumes to document image.
    

    View Slide

15. PYTHON AND WSGI APPS
     Web Servers
    Don't run a web server on your container. Use an external
    proxy or container instead.
    Just run WSGI apps using a WSGI app server:
    uWSGI
    Gunicorn
     Virtualenvs
    Don't use virtualenvs inside Docker containers!
    Install directly into the system Python site packages.
    

    View Slide

16. DEBUGGING CONTAINERS
    Want a minimal image, so no SSH daemon...
    ...so how do we debug a running container?
     Run Bash (or other command) on a Running Service
     Inspecting a Service's Logs (Standard Out & Error)
     Inspecting a Running Container's Setup
    d
    o
    c
    k
    e
    r
    ­
    c
    o
    m
    p
    o
    s
    e e
    x
    e
    c $
    S
    E
    R
    V
    I
    C
    E
    _
    N
    A
    M
    E /
    b
    i
    n
    /
    b
    a
    s
    h
    d
    o
    c
    k
    e
    r
    ­
    c
    o
    m
    p
    o
    s
    e l
    o
    g
    s $
    S
    E
    R
    V
    I
    C
    E
    _
    N
    A
    M
    E
    d
    o
    c
    k
    e
    r i
    n
    s
    p
    e
    c
    t $
    C
    O
    N
    T
    A
    I
    N
    E
    R
    _
    I
    D
    > .
    .
    .
    d
    o
    c
    k
    e
    r i
    n
    s
    p
    e
    c
    t ­
    ­
    f
    o
    r
    m
    a
    t '
    {
    {
    j
    s
    o
    n .
    C
    o
    n
    f
    i
    g
    .
    E
    x
    p
    o
    s
    e
    d
    P
    o
    r
    t
    s }
    }
    ' \
    $
    C
    O
    N
    T
    A
    I
    N
    E
    R
    _
    I
    D
    > {
    "
    5
    0
    0
    0
    /
    t
    c
    p
    "
    : {
    }
    }
    

    View Slide

17. PERSISTANCE, CONFIGS & PROCESSES
     Volume Maps
    Changes to container lost a er container destroyed.
    Volume maps to external host folder for persistence.
    Another pattern is using separate Docker data containers.
     Configuration
    Prefer using environment variables for configuration.
    Volume mapped configs maybe a warning sign of a overly
    complex setup or a config in need of refactoring.
     Managing Processes
    Use supervisord or runit to control multiple processes.
    Consider refactoring containers to not need that.
    

    View Slide

18. TESTING + TOOLING
     Testing
    Docker adds consistency in your CI environments!
    Simple setup for a Docker host.
    Control over what is in container = Repeatable workflow
    and simpler test environment.
    Cloud-based CI options with Docker support out there.
     Tooling
    Docker tool defaults, options, and internal API can radically
    from version to version.
    Don't build your own tooling! If you can avoid it...
    docker-py: a Python client library for working with Docker*
    

    View Slide

19. DEPLOYMENT AND SCALING
    
    

    View Slide

20. EXAMPLE ARCHITECTURE
    Points
    App + API
    Redeem
    Service
    User + Project
    Registry Service
    Database
    Database
    Dogecoin
    Database
    Bitbucket
    

    View Slide

21. EXAMPLE PROD ENVIRONMENT
    Datacenter 2
    Datacenter 2
    LoadBalancer
    User + Project
    Registry Service
    Redeem
    Service
    Points
    App + API
    Datacenter 1
    Datacenter 1
    Points
    App + API
    Points
    App + API
    Redeem
    Service
    User + Project
    Registry Service
    Database
    Database Database
    LoadBalancer LoadBalancer
    User + Project
    Registry Service
    Redeem
    Service
    Points
    App + API
    Points
    App + API
    Points
    App + API
    Redeem
    Service
    User + Project
    Registry Service
    Database
    Database Database
    LoadBalancer LoadBalancer
    

    View Slide

22. SETTING UP A CLOUD
    Looks like you're trying to build a cloud of microservices...
     Load Balancing + Network Topology:
    e.g. HAProxy & Nginx, etc.
     Provisioning:
    Automated, repeatable setup for non-Docker systems.
    e.g. Ansible, Puppet & Salt.
     Monitoring:
    Look at app health, app behaviour & system resources.
    e.g. Nagios, Pingdom & New Relic.
     Logging:
    Aggregate various logs and correlate events.
    e.g. Splunk.
    

    View Slide

23. CLOUD INFRASTRUCTURE
     Managing cloud infrastructure is hard!
     Need tooling and automation for all that stuff.
     Don't build your own tool unless you want to support it to
    end of time. (Unless you're a cloud tech vendor.)
     Consider using one of these instead:
    Docker
    Swarm
    Kubernetes OpenStack
    Magnum
    CoreOS
    Fleet
    

    View Slide

24. LESSONS LEARNED
    
    

    View Slide

25. LESSONS LEARNED
    Microservices and Docker can improve building and deploying
    complex systems. But neither is a cure-all.
    Good development & deployment processes matter. Docker
    has a decent workflow to help shape those processes.
    Expect lots of additional infrastructure around microservices.
    Avoid building your own tooling.
    Use Docker containers to do effective isolation.
    Good app design goes a long way.
    

    View Slide

26. RESOURCES
    Jared Kerim's Django Docker template:
    12 Factor apps:
    Rookeries - Dockerized Workflow Example:
    https://github.com/jaredkerim/django-docker-compose
    http://12factor.net/
    https://bitbucket.org/dorianpula/rookeries/
    (docker_compose_workflow branch)
    

    View Slide

27. THANK YOU!
     Twitter - @dorianpula
     WWW - http://dorianpula.ca/
    ANY QUESTIONS?
    GO FORTH AND BUILD AWESOME STUFF!!!
    

    View Slide