Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Fabric and Docker for deployment testing

Using Fabric and Docker for deployment testing

"Using Fabric and Docker for deployment testing". Slide of my presentation @ PyCon APAC 2014, Taiwan. [*]

[*] https://tw.pycon.org/2014apac


Cheng-Lung Sung

May 18, 2014


  1. Using Fabric and Docker for deployment testing Cheng-Lung Sung (clsung@)

  2. about.me/clsung

  3. Working at Studio Engineering, hTC

  4. Lots of Golang Few Python

  5. 因為缺⼈人,所以很忙 (這絕不是徵⼈人廣告)

  6. Love Testing

  7. Code with no test is like turning with no direction

  8. You don’t meet trouble Trouble meets you.

  9. • Motivation • Fabric Introduction • Introduction to Docker •

    Fabric + Docker for deployment testing Outline
  10. Motivation

  11. 有⼀一天我夢到我做了 ⼀一個夢

  12. (most) Developers use Mac (圖⽚片引⾃自 pycontw fb)

  13. • Developers use Mac • brew install ... • Development

    site is Ubuntu • apt-get install ...
  14. • Developer use Mac • Development site is Ubuntu •

    Production site is CentOS • yum install ...
  15. 如果這不是惡夢 什麼才是?

  16. • Developer use Mac • Development site is Ubuntu •

    Production site is CentOS • Demo site is Windows • Windows Update
  17. 夢到這我就醒了

  18. Fabric Introduction

  19. Fabric from README • Fabric is a Python (2.5-2.7) library

    and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
  20. --- a/pep-0373.txt +++ b/pep-0373.txt +Update +====== + +The End Of

    Life date (EOL, sunset date) for Python 2.7 has been moved +five years into the future, to 2020. This decision was made to +clarify the status of Python 2.7 and relieve worries for those users +who cannot yet migrate to Python 3. See also PEP 466. + +This declaration does not guarantee that bugfix releases will be made +on a regular basis, but it should enable volunteers who want to +contribute bugfixes for Python 2.7 and it should satisfy vendors who +still have to support Python 2 for years to come. + +There will be no Python 2.8. + Release Manager and Crew ======================== @@ -50,15 +65,16 @@ Maintenance releases
  21. Fabric is • A tool that lets you execute arbitrary

    Python functions via the command line; • A library of subroutines (built on top of a lower-level library) to make executing shell commands over SSH easy and Pythonic.
  22. Fabric Features • execute local() or run() remote shell commands

    (via ssh) • parallel execution (via multiprocessing) • prompt input and managed output • abort execution • define tasks
  23. from fabric.contrib.console import confirm def coverage(): local('coverage erase') local('coverage run

    ./manage.py test') local("coverage report --omit='*/site-packages/*'") local("coverage html --omit='*/site-packages/*'") def pre_commit(): create_folder() with settings(warn_only=True): result = local('./manage.py test', capture=True) if result.failed and \ not confirm("Tests failed. Continue anyway?"): abort("Aborting at user request.") coverage()
  24. Fabric Environment • Environment as configuration (dict, fabric.state.env) • env.host,

    env.password • env.hosts (list), env.passwords (dict) • env.port • env.local_user, env.user • env.roledefs (dict of name to host list mapping) • The settings context manager • with settings(warn_only=true):
  25. Fabric Execution model • Execution strategy • A list of

    tasks is created • For each task, a task-specific host list is generated from various sources • The task list is walked through in order, and each task if run once per host in its hosts list • Tasks with no hosts in their host list are considered local-only, and will alway run once and only onces
  26. from fabric.api import run, task, env def set_hosts(): env.hosts =

    ['dev.clsung.tw', 'linode.clsung.tw'] def task(): run('uname -s') $ fab task No hosts found. Please specify (single) host string for connection: $ fab set_hosts task [dev.clsung.tw] Executing task 'task' [dev.clsung.tw] run: uname -s [dev.clsung.tw] out: Linux [dev.clsung.tw] out: [linode.clsung.tw] Executing task 'task' [linode.clsung.tw] run: uname -s [linode.clsung.tw] out: Linux [linode.clsung.tw] out:
  27. Fabric @roles from fabric.api import task, env, run, roles env.roledefs

    = { 'db': ['linode.clsung.tw'], 'dev': ['dev.clsung.tw'], } @roles('db') def task(): run('uname -s')
  28. Fabric 總結

  29. Fabric 是⽤用 python 寫的

  30. 也是寫成 .py

  31. 但是你不⽤用很懂 python

  32. 但是你要很懂你要做什麼

  33. Fabric 是讓懶⼈人⽤用的

  34. 懶,也是進步的原動⼒力

  35. 因為我懶

  36. 所以我⽤用 Fabric

  37. None
  38. Developer 的夢靨

  39. "It worked yesterday." “昨天還是好的啊“ "You must have the wrong version."

    “你⼀一定是⽤用到舊版了“ "THIS can't be the source of THAT." “這個不可能是那個的原始碼” "It works, but it hasn't been tested." “這程式應該是會動的,只是我寫好後還沒做測試。” "It works on my machine.” “在我的機器明明就是好的”
  40. None
  41. Developer 的夢想

  42. Build once...run anywhere

  43. Devops 的夢靨

  44. None
  45. Devops 的夢想

  46. Configure once...run anything

  47. 地⽅方的 Devops 需要 Docker

  48. 為什麼不⽤用 VM 就好?

  49. None
  50. • Resource Allocation in VM setting • CPU, RAM •

    Preallocated disk, .. • The modification is recorded • Boot once, test twice, result changes • If always use fresh image • And always re-run the whole setup process
  51. LinuX Container(LXC) • Run a Linux system within another Linux

    system • Isolated environment for processes in the container • From inside, like a VM • From outside, like a normal process • “chroot on steroids”
  52. Docker ⽤用 golang 寫的 所以不要介紹太多

  53. 簡單地說,下載 docker image

  54. Just run it!

  55. # A basic apache server. To use either add or

    bind mount content under /var/www FROM ubuntu:12.04 MAINTAINER Kimbro Staken version: 0.1 RUN apt-get update && \ apt-get install -y apache2 && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* ENV APACHE_RUN_USER www-data ENV APACHE_RUN_GROUP www-data ENV APACHE_LOG_DIR /var/log/apache2 EXPOSE 80 CMD ["/usr/sbin/apache2", "-D", "FOREGROUND"]
  56. Dockerfile rules!

  57. ⾃自已的 image ⾃自已 build

  58. Docker caches

  59. $ fab build [localhost] local: docker build . Uploading context

    3.584 kB Uploading context Step 0 : FROM phusion/baseimage ---> 35c2d5a269ff Step 1 : ENV HOME /root ---> Using cache ---> 50937eb28ee8 .... Successfully built 532a5f458a34
  60. Docker refreshes

  61. # VERSION 0.0.1 FROM centos:latest MAINTAINER Cheng-Lung Sung <clsung@gmail.com> #

    Enviroment ENV HOME /root ENV GOPATH /root/go ENV GOROOT /root/hg/go ENV PATH /root/go/bin:/root/hg/go/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/ bin # for source code RUN mkdir -p /root/go/src/clsung.tw # Development tools RUN rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm RUN yum -y groupinstall "Development Tools" RUN yum -y python pip # build Go RUN cd /root/go/src/clsung.tw && git clone https://github.com/clsung/django_poster RUN cd /root/go/src/clsung.tw && pip -r requirement.txt # set working directory (entrypoint root) WORKDIR /root/go/src/clsung.tw/django_poster ENTRYPOINT ['./manage.py', 'runserver']
  62. Lesson learned: Put fixed RUN cmds earlier

  63. Docker repos

  64. index.docker.io

  65. Notes on MacOSX • Require boot2docker (linux) • VOLUME is

    useless on MacOSX • docker cp in running container • use vagrant
  66. Fabric + Docker for deployment testing

  67. • Replace virtualenv with docker in testing • Use docker

    to test backend (web and db) • After testing, convert Dockerfile to Salt File • Before docker 1.0 released Goal
  68. run docker command in fabric @task def docker(cmd): """ Run

    Docker cmd """ return run("docker %s" % cmd)
  69. clean container by name @task def clean_env(): with settings(warn_only=True): docker('kill

    simplehttpd') docker('rm simplehttpd') docker('rm client')
  70. build images @task def build(): docker_ids = {} for d

    in ['httpd', 'client']: with lcd(d): result =local('docker build .', capture=True) # handy hacks: get the image id docker_ids[d] = result.stdout.split()[-1] return docker_ids
  71. @task def pre_deploy(): clean_env() d_ids = build() docker('run --name simplehttpd

    -d {0}'.format( d_ids['httpd'])) with settings(warn_only=True): result = docker('run -i --rm --link simplehttpd:httpd ' '--name "client" {0}'.format( d_ids['client']), capture=True) if result.failed and not confirm("Tests failed. Continue anyway?"): abort("Aborting at user request.") logger.info(result.stdout) docker('kill simplehttpd')
  72. FROM phusion/baseimage ENV HOME /root RUN /etc/my_init.d/00_regen_ssh_host_keys.sh CMD ["/sbin/my_init"] RUN

    apt-get update RUN apt-get install -y python EXPOSE 8000 ENTRYPOINT ["python", "-m", "SimpleHTTPServer", "8000"] httpd/Dockerfile
  73. client/Dockerfile FROM phusion/baseimage ENV HOME /root RUN apt-get update RUN

    apt-get install -y python python-pip RUN pip install requests ADD ping.py /opt/ping.py CMD ["/usr/bin/python", "/opt/ping.py"]
  74. db/ping.py #!/usr/bin/env python import requests resp = requests.get('http://httpd:8001/') resp.raise_for_status() print

  75. $ fab pre_deploy [localhost] local: docker build . [localhost] local:

    docker run --name simplehttpd -d b6c6ab6afc4d 9c2515094c384099a4d867747e0a9de2f72b0244e86f0263b559e555 98dd4239 [localhost] local: docker run -i --rm --link simplehttpd:httpd --name "client" f546ae5530c5 INFO:fabfile:200 [localhost] local: docker kill simplehttpd simplehttpd Done.
  76. if any error..... [localhost] local: docker build . [localhost] local:

    docker run --name simplehttpd -d b6c6ab6afc4d 97da8851573380116f83b9f11f7a884591ebfffa53875ce91defd7810359ba7b [localhost] local: docker run -i --rm --link simplehttpd:httpd --name "client" cc5951661843 Warning: local() encountered an error (return code 1) while executing 'docker run -i --rm --link simplehttpd:httpd --name "client" cc5951661843' Tests failed. Continue anyway? [Y/n] n Fatal error: Aborting at user request. Aborting.
  77. None
  78. Other • Shipper - fabric for docker • https://github.com/mailgun/shipper •

    easy to move containers from a developer environment to a staging environment
  79. Thank you!