Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Docker使ってたらサーバがゴミ捨て場みたいになってた話

 Docker使ってたらサーバがゴミ捨て場みたいになってた話

Immutable Infrastructure Conference #1

Avatar for Yuuki Tsubouchi (yuuk1)

Yuuki Tsubouchi (yuuk1)

March 25, 2014
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Technology

Transcript

  1. UPQVQEBZT  VTFST MPBEBWFSBHF   5BTLTUPUBM SVOOJOH TMFFQJOH TUPQQFE

    [PNCJF $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU ,J#.FNUPUBM VTFE GSFF CV⒎FST ,J#4XBQUPUBM VTFE GSFF DBDIFE
  2. w QSPDFTTFT w -9$ͷϓϩηε͕͏Α͏Αͯ͠ɺϝϞϦ৯͍ͭ Ϳͯ͠TXBQ |-docker -d -p /var/run/docker.pid -H

    tcp://0.0.0.0:5555 -H unix:///var/run/docker.sock | |-lxc-start -n 718e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652abfs -f… | |-lxc-start -n 542e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a2b3 -f… | |-lxc-start -n 983e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652ad00 -f… | |-lxc-start -n 271e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a111 -f… | |-lxc-start -n 323e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a1f2 -f… | |-lxc-start -n 789e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a1f2 -f... w EPDLFSTUPQLJMMDPOUBJOFS@JE͚ͩͩͱϓϩ ηε͸ࢮΜͰ͘Εͳ͍
  3. # lsof -p <docker pid> | wc -l 1021 !

    # cat /proc/<docker pid>/limits | grep files Max open files 1024 4096 files
  4. root 31084 0.0 0.0 0 0 ? S< 2013 0:00

    [ext4-dio-unwrit] root 31105 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31107 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31109 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1793-8] root 31110 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31126 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31128 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31131 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1794-8] root 31132 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31153 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31155 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31157 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1795-8] root 31159 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31176 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31178 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31180 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1796-8] root 31181 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31203 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31204 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31206 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1797-8] root 31207 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31229 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31231 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31234 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1798-8] root 31235 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31247 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31249 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31251 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1799-8] root 31252 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit]
  5. • lsof -p <docker pid> • /dev/mapper/docker-xxx … Έ͍ͨͳΛ΍ͨΒ ͱั·͑ͯΔ

    • mount /dev/mapper/docker-202:1-3019790-<sha1> on /var/lib/docker/devicemapper/mnt/ <sha1> type ext4 (rw,relatime,discard,stripe=16,data=ordered) /dev/mapper/docker-202:1-3019790-<sha1> on /var/lib/docker/devicemapper/mnt/ <sha1> type ext4 (rw,relatime,discard,stripe=16,data=ordered) /dev/mapper/docker-202:1-3019790-<sha1> on /var/lib/docker/devicemapper/mnt/ <sha1> type ext4 (rw,relatime,discard,stripe=16,data=ordered) ……..
  6. docker lxc-start … lxc-start … lxc-start … lxc-start … /dev/mapper/docker-xxx:…

    ɾ ɾ ɾ /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… fork open open open open
  7. Device mapper • Docker ͷΠϝʔδࠩ෼؅ཧΛࢧ͑Δٕज़ͷ1ͭ • aufs, btrfs ͳͲͰ୅ସՄೳ •

    Ծ૝తͳϒϩοΫσόΠεΛͭ͘ΕΔ • Ծ૝తͳϒϩοΫσόΠεΛελοΫͰ͖Δ • ࠩ෼؅ཧ • ࠩ෼͋ͨΓɺ1 mount -> 1 open ??
  8. 0 * * * * root docker rm -v $(docker

    ps -a -q) 1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount /FX
  9. • κϯϏʹͳͬͨLXCϓϩηε͕௫ΜͰΔ /dev/ mapper-xxx ͕ Resource Busy ʹͳͬͯͯ umount Ͱ͖ͳ͍

    • ଟ෼ɺdocker rm ͰLXCϓϩηεΛ͏·͘ࡴͤͯ ͳͯ͘κϯϏʹͳͬͯΔ
  10. 0 * * * * root docker rm -v $(docker

    ps -a -q) 1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount 0 10 * * * root service docker restart /FX
  11. 0 * * * * root docker rm -v $(docker

    ps -a -q) 1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount 0 10 * * * root service docker restart 59 * * * * root docker ps | grep Ghost | cut -d" " -f1 | xargs docker kill 2 * * * * root docker rmi $(docker images | awk '/^<none>/ { print $3 }') /FX /FX
  12. • Refactor the devicemapper code to avoid many mount/unmount race

    conditions and failures! • Remove directory when removing devicemapper device. This cleans up leftover mount directories! • Avoid extra mount/unmount during container registration. This removes an unneeded mount/unmount operation which was causing problems with devicemapper! • Clean up archive closing. This fixes and improves archive handling! • Avoid temporarily unmounting the container when restarting it. This fixes a race for devicemapper during restart
  13. • Avoid extra mount/unmount during build. This fixes mount/unmount related

    errors during build.! • Make --rm the default for docker build.! • Devicemapper: cleanups and fix for unmount. Fixes two problems which were causing unmount to fail intermittently.! • Devicemapper: remove directory when removing device. Directories don't get left behind when removing the device.! • Add -f flag to docker rm to force removal of running containers.! • Kill ghost containers and restart all ghost containers when the docker daemon restarts.