Docker使ってたらサーバがゴミ捨て場みたいになってた話

A658ec7f1badf73819dfa501165016c1?s=47 Yuuki Tsubouchi (yuuk1)
March 25, 2014
Technology
5
23k

 Docker使ってたらサーバがゴミ捨て場みたいになってた話

Immutable Infrastructure Conference #1

A658ec7f1badf73819dfa501165016c1?s=128

Yuuki Tsubouchi (yuuk1)

March 25, 2014
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

See All by Yuuki Tsubouchi (yuuk1)
A658ec7f1badf73819dfa501165016c1?s=48 yuukit
0
490
A658ec7f1badf73819dfa501165016c1?s=48 yuukit
0
260
A658ec7f1badf73819dfa501165016c1?s=48 yuukit
5
760
A658ec7f1badf73819dfa501165016c1?s=48 yuukit
0
61
A658ec7f1badf73819dfa501165016c1?s=48 yuukit
0
40
A658ec7f1badf73819dfa501165016c1?s=48 yuukit
15
23k
A658ec7f1badf73819dfa501165016c1?s=48 yuukit
7
860
A658ec7f1badf73819dfa501165016c1?s=48 yuukit
9
5.4k
A658ec7f1badf73819dfa501165016c1?s=48 yuukit
2
16k

Other Decks in Technology

See All in Technology
380bcd2ab751f5838abd8219df53e5fe?s=48 tomoki10
0
1.1k
2ca9e6e68b43a796a8add2bcb9bbad2e?s=48 mizutani
0
260
9e7a653ed313b3da4b263a772bfc5026?s=48 clustervr
0
170
8aba6de431668fc8d09977c0e33c63c1?s=48 jaguar_imo
4
1.1k
Ecad9d801d79f6c6e5df93094690685e?s=48 sinsoku
6
2.2k
91e7aee9769836f448a9f1ce91255749?s=48 mathatelle
0
1.5k
1820130315ee3f2e970aeeef918b264a?s=48 yuzoshigematsu
2
1.3k
Cf18e42df2acc7a1fc1700bcd8e76487?s=48 syrusakbary
0
240
9e7a653ed313b3da4b263a772bfc5026?s=48 clustervr
0
170
93fc2d713bc4d0e9c594bf8c9e6bcc62?s=48 oztick139
0
130
1d8947ec45ce78c83251f4d076abe329?s=48 rpabank
0
150
0d29d155ce5c5a93ed46363626da3ea7?s=48 ocise
1
1.9k

Featured

See All Featured
D36d2c1821af9249b69ff7f5ed60529b?s=48 tammielis
236
23k
5ce91fa49a613cbc3e20d5f96856473f?s=48 edds
55
9.3k
651dcfe41723207fbb0aa044a4f7c07f?s=48 dotmariusz
94
5.1k
9fa239b3154a0169d99ab7bf1422dd12?s=48 marcelosomers
220
14k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
227
17k
39488f9d172ab92fd352f2cd7b73258d?s=48 mza
80
4k
A9179349dd2bdc67f377719f56d85656?s=48 garrettdimon
285
110k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
682
180k
48c098b6579220a67a6ba949be6e4b5a?s=48 revolveconf
197
9.5k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
1
360
D09fad3e36ae6841f54820be0e907f7a?s=48 rocio
155
11k
D47656e20ff5e42f125fc5ea0fd636c6?s=48 tmm1
61
7.5k

Transcript

  1. id:y_uuki / @y_uuk1

  2. None

  3. %PDLFS࢖ͬͯͨΒαʔό͕ ΰϛࣺͯ৔Έ͍ͨʹͳͬͯͨ࿩ *NNVUBCMF*OGSBTUSVDUVSF$POGFSFODF

  4. None

  5. ✘

  6. MPBEBWH͕௒͑ͨ

  7. • curl http://<jenkins>:8000/job/project • timeout • ssh <jenkins> • timeout

    • Dom0 ͔Β ήετOSʹϩάΠϯͯ͠ top

  8. UPQVQEBZT  VTFST MPBEBWFSBHF   5BTLTUPUBM SVOOJOH TMFFQJOH TUPQQFE

    [PNCJF $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU ,J#.FNUPUBM VTFE GSFF CV⒎FST ,J#4XBQUPUBM VTFE GSFF DBDIFE

  9. w QSPDFTTFT w -9$ͷϓϩηε͕͏Α͏Αͯ͠ɺϝϞϦ৯͍ͭ Ϳͯ͠TXBQ |-docker -d -p /var/run/docker.pid -H

    tcp://0.0.0.0:5555 -H unix:///var/run/docker.sock | |-lxc-start -n 718e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652abfs -f… | |-lxc-start -n 542e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a2b3 -f… | |-lxc-start -n 983e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652ad00 -f… | |-lxc-start -n 271e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a111 -f… | |-lxc-start -n 323e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a1f2 -f… | |-lxc-start -n 789e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a1f2 -f... w EPDLFSTUPQLJMMDPOUBJOFS@JE͚ͩͩͱϓϩ ηε͸ࢮΜͰ͘Εͳ͍

  10. 0 * * * * root docker rm $(docker ps

    -a -q)

  11. σΟεΫϦϓλ਺͕ ௒͑ͨ

  12. 2014/02/13 18:18:46 Error: start: Cannot start container d42e47e9e6bce9770f1d6d940944fc9e3f9a50a00baceaf21bfa9b1bbfced6d6: pipe2: too

    many open files

  13. # lsof -p <docker pid> | wc -l 1021 !

    # cat /proc/<docker pid>/limits | grep files Max open files 1024 4096 files

  14. root 31084 0.0 0.0 0 0 ? S< 2013 0:00

    [ext4-dio-unwrit] root 31105 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31107 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31109 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1793-8] root 31110 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31126 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31128 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31131 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1794-8] root 31132 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31153 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31155 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31157 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1795-8] root 31159 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31176 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31178 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31180 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1796-8] root 31181 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31203 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31204 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31206 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1797-8] root 31207 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31229 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31231 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31234 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1798-8] root 31235 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31247 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31249 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31251 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1799-8] root 31252 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit]

  15. • lsof -p <docker pid> • /dev/mapper/docker-xxx … Έ͍ͨͳΛ΍ͨΒ ͱั·͑ͯΔ

    • mount /dev/mapper/docker-202:1-3019790-<sha1> on /var/lib/docker/devicemapper/mnt/ <sha1> type ext4 (rw,relatime,discard,stripe=16,data=ordered) /dev/mapper/docker-202:1-3019790-<sha1> on /var/lib/docker/devicemapper/mnt/ <sha1> type ext4 (rw,relatime,discard,stripe=16,data=ordered) /dev/mapper/docker-202:1-3019790-<sha1> on /var/lib/docker/devicemapper/mnt/ <sha1> type ext4 (rw,relatime,discard,stripe=16,data=ordered) ……..

  16. docker lxc-start … lxc-start … lxc-start … lxc-start … /dev/mapper/docker-xxx:…

    ɾ ɾ ɾ /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… fork open open open open

  17. Device mapper • Docker ͷΠϝʔδࠩ෼؅ཧΛࢧ͑Δٕज़ͷ1ͭ • aufs, btrfs ͳͲͰ୅ସՄೳ •

    Ծ૝తͳϒϩοΫσόΠεΛͭ͘ΕΔ • Ծ૝తͳϒϩοΫσόΠεΛελοΫͰ͖Δ • ࠩ෼؅ཧ • ࠩ෼͋ͨΓɺ1 mount -> 1 open ??

  18. 0 * * * * root docker rm -v $(docker

    ps -a -q) 1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount /FX

  19. κϯϏϓϩηε਺͕ ௒͑ͨ

  20. ! Tasks: 3020 total, 5 running, 2700 sleeping, 0 stopped,

    315 zombie

  21. • κϯϏʹͳͬͨLXCϓϩηε͕௫ΜͰΔ /dev/ mapper-xxx ͕ Resource Busy ʹͳͬͯͯ umount Ͱ͖ͳ͍

    • ଟ෼ɺdocker rm ͰLXCϓϩηεΛ͏·͘ࡴͤͯ ͳͯ͘κϯϏʹͳͬͯΔ

  22. 0 * * * * root docker rm -v $(docker

    ps -a -q) 1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount 0 10 * * * root service docker restart /FX

  23. 0 * * * * root docker rm -v $(docker

    ps -a -q) 1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount 0 10 * * * root service docker restart 59 * * * * root docker ps | grep Ghost | cut -d" " -f1 | xargs docker kill 2 * * * * root docker rmi $(docker images | awk '/^<none>/ { print $3 }') /FX /FX

  24. %JTQPTBCMF

  25.  ͜͜·Ͱ

  26.  https://github.com/dotcloud/docker/blob/master/CHANGELOG.md#080-2014-02-04 https://github.com/dotcloud/docker/blob/master/CHANGELOG.md#081-2014-02-18

  27. • Refactor the devicemapper code to avoid many mount/unmount race

    conditions and failures! • Remove directory when removing devicemapper device. This cleans up leftover mount directories! • Avoid extra mount/unmount during container registration. This removes an unneeded mount/unmount operation which was causing problems with devicemapper! • Clean up archive closing. This fixes and improves archive handling! • Avoid temporarily unmounting the container when restarting it. This fixes a race for devicemapper during restart

  28.  https://github.com/dotcloud/docker/blob/master/CHANGELOG.md#090-2014-03-10

  29. • Avoid extra mount/unmount during build. This fixes mount/unmount related

    errors during build.! • Make --rm the default for docker build.! • Devicemapper: cleanups and fix for unmount. Fixes two problems which were causing unmount to fail intermittently.! • Devicemapper: remove directory when removing device. Directories don't get left behind when removing the device.! • Add -f flag to docker rm to force removal of running containers.! • Kill ghost containers and restart all ghost containers when the docker daemon restarts.

  30. Clean Infrastructure