Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Docker使ってたらサーバがゴミ捨て場みたいになってた話

 Docker使ってたらサーバがゴミ捨て場みたいになってた話

Immutable Infrastructure Conference #1

Yuuki Tsubouchi (yuuk1)

March 25, 2014
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Technology

Transcript

  1. id:y_uuki / @y_uuk1

    View Slide

  2. View Slide

  3. %PDLFS࢖ͬͯͨΒαʔό͕
    ΰϛࣺͯ৔Έ͍ͨʹͳͬͯͨ࿩
    *NNVUBCMF*OGSBTUSVDUVSF$POGFSFODF

    View Slide

  4. View Slide


  5. View Slide

  6. MPBEBWH͕௒͑ͨ

    View Slide

  7. • curl http://:8000/job/project
    • timeout
    • ssh
    • timeout
    • Dom0 ͔Β ήετOSʹϩάΠϯͯ͠ top

    View Slide

  8. UPQVQEBZT VTFST MPBEBWFSBHF
    5BTLTUPUBM SVOOJOH TMFFQJOH TUPQQFE [PNCJF
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    $QVVT TZ OJ JE XB IJ TJ TU
    ,J#.FNUPUBM VTFE GSFF CV⒎FST
    ,J#4XBQUPUBM VTFE GSFF DBDIFE

    View Slide

  9. w QSPDFTTFT
    w -9$ͷϓϩηε͕͏Α͏Αͯ͠ɺϝϞϦ৯͍ͭ
    Ϳͯ͠TXBQ
    |-docker -d -p /var/run/docker.pid -H tcp://0.0.0.0:5555 -H unix:///var/run/docker.sock
    | |-lxc-start -n 718e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652abfs -f…
    | |-lxc-start -n 542e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a2b3 -f…
    | |-lxc-start -n 983e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652ad00 -f…
    | |-lxc-start -n 271e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a111 -f…
    | |-lxc-start -n 323e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a1f2 -f…
    | |-lxc-start -n 789e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a1f2 -f...
    w EPDLFSTUPQLJMMDPOUBJOFS@JE͚ͩͩͱϓϩ
    ηε͸ࢮΜͰ͘Εͳ͍

    View Slide

  10. 0 * * * * root docker rm $(docker ps -a -q)

    View Slide

  11. σΟεΫϦϓλ਺͕
    ௒͑ͨ

    View Slide

  12. 2014/02/13 18:18:46 Error: start: Cannot start container
    d42e47e9e6bce9770f1d6d940944fc9e3f9a50a00baceaf21bfa9b1bbfced6d6: pipe2:
    too many open files

    View Slide

  13. # lsof -p | wc -l
    1021
    !
    # cat /proc//limits | grep files
    Max open files 1024 4096 files

    View Slide

  14. root 31084 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit]
    root 31105 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush]
    root 31107 0.0 0.0 0 0 ? S< 2013 0:00 [bioset]
    root 31109 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1793-8]
    root 31110 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit]
    root 31126 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush]
    root 31128 0.0 0.0 0 0 ? S< 2013 0:00 [bioset]
    root 31131 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1794-8]
    root 31132 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit]
    root 31153 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush]
    root 31155 0.0 0.0 0 0 ? S< 2013 0:00 [bioset]
    root 31157 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1795-8]
    root 31159 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit]
    root 31176 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush]
    root 31178 0.0 0.0 0 0 ? S< 2013 0:00 [bioset]
    root 31180 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1796-8]
    root 31181 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit]
    root 31203 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush]
    root 31204 0.0 0.0 0 0 ? S< 2013 0:00 [bioset]
    root 31206 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1797-8]
    root 31207 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit]
    root 31229 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush]
    root 31231 0.0 0.0 0 0 ? S< 2013 0:00 [bioset]
    root 31234 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1798-8]
    root 31235 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit]
    root 31247 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush]
    root 31249 0.0 0.0 0 0 ? S< 2013 0:00 [bioset]
    root 31251 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1799-8]
    root 31252 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit]

    View Slide

  15. • lsof -p
    • /dev/mapper/docker-xxx … Έ͍ͨͳΛ΍ͨΒ
    ͱั·͑ͯΔ
    • mount
    /dev/mapper/docker-202:1-3019790- on /var/lib/docker/devicemapper/mnt/
    type ext4 (rw,relatime,discard,stripe=16,data=ordered)
    /dev/mapper/docker-202:1-3019790- on /var/lib/docker/devicemapper/mnt/
    type ext4 (rw,relatime,discard,stripe=16,data=ordered)
    /dev/mapper/docker-202:1-3019790- on /var/lib/docker/devicemapper/mnt/
    type ext4 (rw,relatime,discard,stripe=16,data=ordered)
    ……..

    View Slide

  16. docker
    lxc-start …
    lxc-start …
    lxc-start …
    lxc-start …
    /dev/mapper/docker-xxx:…
    ɾ
    ɾ
    ɾ
    /dev/mapper/docker-xxx:…
    /dev/mapper/docker-xxx:…
    /dev/mapper/docker-xxx:…
    /dev/mapper/docker-xxx:…
    /dev/mapper/docker-xxx:…
    /dev/mapper/docker-xxx:…
    /dev/mapper/docker-xxx:…
    /dev/mapper/docker-xxx:…
    /dev/mapper/docker-xxx:…
    /dev/mapper/docker-xxx:…
    /dev/mapper/docker-xxx:…
    fork
    open
    open
    open
    open

    View Slide

  17. Device mapper
    • Docker ͷΠϝʔδࠩ෼؅ཧΛࢧ͑Δٕज़ͷ1ͭ
    • aufs, btrfs ͳͲͰ୅ସՄೳ
    • Ծ૝తͳϒϩοΫσόΠεΛͭ͘ΕΔ
    • Ծ૝తͳϒϩοΫσόΠεΛελοΫͰ͖Δ
    • ࠩ෼؅ཧ
    • ࠩ෼͋ͨΓɺ1 mount -> 1 open ??

    View Slide

  18. 0 * * * * root docker rm -v $(docker ps -a -q)
    1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount
    /FX

    View Slide

  19. κϯϏϓϩηε਺͕
    ௒͑ͨ

    View Slide

  20. !
    Tasks: 3020 total, 5 running, 2700 sleeping, 0 stopped, 315 zombie

    View Slide

  21. • κϯϏʹͳͬͨLXCϓϩηε͕௫ΜͰΔ /dev/
    mapper-xxx ͕ Resource Busy ʹͳͬͯͯ
    umount Ͱ͖ͳ͍
    • ଟ෼ɺdocker rm ͰLXCϓϩηεΛ͏·͘ࡴͤͯ
    ͳͯ͘κϯϏʹͳͬͯΔ

    View Slide

  22. 0 * * * * root docker rm -v $(docker ps -a -q)
    1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount
    0 10 * * * root service docker restart
    /FX

    View Slide

  23. 0 * * * * root docker rm -v $(docker ps -a -q)
    1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount
    0 10 * * * root service docker restart
    59 * * * * root docker ps | grep Ghost | cut -d" " -f1 | xargs docker kill
    2 * * * * root docker rmi $(docker images | awk '/^/ { print $3 }')
    /FX
    /FX

    View Slide

  24. %JTQPTBCMF

    View Slide


  25. ͜͜·Ͱ

    View Slide


  26. https://github.com/dotcloud/docker/blob/master/CHANGELOG.md#080-2014-02-04
    https://github.com/dotcloud/docker/blob/master/CHANGELOG.md#081-2014-02-18

    View Slide

  27. • Refactor the devicemapper code to avoid many mount/unmount race
    conditions and failures!
    • Remove directory when removing devicemapper device. This cleans up
    leftover mount directories!
    • Avoid extra mount/unmount during container registration. This
    removes an unneeded mount/unmount operation which was
    causing problems with devicemapper!
    • Clean up archive closing. This fixes and improves archive handling!
    • Avoid temporarily unmounting the container when restarting it.
    This fixes a race for devicemapper during restart

    View Slide


  28. https://github.com/dotcloud/docker/blob/master/CHANGELOG.md#090-2014-03-10

    View Slide

  29. • Avoid extra mount/unmount during build. This fixes mount/unmount
    related errors during build.!
    • Make --rm the default for docker build.!
    • Devicemapper: cleanups and fix for unmount. Fixes two problems which
    were causing unmount to fail intermittently.!
    • Devicemapper: remove directory when removing device. Directories
    don't get left behind when removing the device.!
    • Add -f flag to docker rm to force removal of running containers.!
    • Kill ghost containers and restart all ghost containers when the docker
    daemon restarts.

    View Slide

  30. Clean Infrastructure

    View Slide