Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Docker Performance on Web Application

Docker Performance on Web Application

Docker Meetup Tokyo #4

Yuuki Tsubouchi (yuuk1)

January 17, 2015
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Technology

Transcript

  1. Docker Performance
    on Web Application
    JEZ@VVLJ
    %PDLFS.FFUVQ5PLZP

    View Slide

  2. id:y_uuki
    ӡ༻
    @y_uuk1

    View Slide

  3. Docker Engine্Ͱɹ
    ΞϓϦέʔγϣϯΛɹ
    ಈ͔ͯ͠ੑೳྼԽ
    ͠ͳ͍ͷʁ

    View Slide

  4. View Slide

  5. 1
    DockerͷύϑΥʔϚϯεʹ
    ͓͍ͯॏཁͳ͜ͱ͸ͳʹ͔

    View Slide

  6. Linux Containers

    View Slide

  7. /FUXPSL
    /BNFTQBDF
    *1$
    /BNFTQBDF
    654
    /BNFTQBDF
    .PVOU
    /BNFTQBDF
    6TFS
    /BNFTQBDF
    1*%
    /BNFTQBDF

    View Slide

  8. /FUXPSL
    /BNFTQBDF
    *1$
    /BNFTQBDF
    654
    /BNFTQBDF
    .PVOU
    /BNFTQBDF
    6TFS
    /BNFTQBDF
    1*%
    /BNFTQBDF
    -JOVYΧʔωϧͷ໊લۭؒػೳͷ
    ू·Γ

    View Slide

  9. Linux Containers
    Χʔωϧͷ֤ϦιʔεΛ໊લۭؒͰִ཭͠
    ۭͨؒͰϓϩηεΛ࣮ߦ͢Δͱ͍͏Ϟσϧ
    ϑΝΠϧγεςϜɼϢʔβɼϓϩηεςʔ
    ϒϧɼωοτϫʔΫ… etc
    LXC ͸ Linux Containers ͷϑϩϯτΤϯυ

    View Slide

  10. Linux Containers Overhead
    ୯ମͷLinux ΧʔωϧͰ׬݁͢ΔͷͰ
    ΦϒδΣΫτͷڞ༗ޮ཰͕Α͍
    Hypervisor ͷΑ͏ʹ֤ϦιʔεΛɹɹ
    ೋॏॲཧ͠ͳͯ͘Α͍

    View Slide

  11. example/packet receive
    /*$ ,FSOFM 6TFSMBOE
    $PQZ $PQZ
    4JOHMF
    ,FSOFM

    View Slide

  12. example/packet receive
    /*$ ,FSOFM 6TFSMBOE
    $PQZ $PQZ
    /*$ ,FSOFM ,FSOFM
    $PQZ $PQZ
    4JOHMF
    ,FSOFM
    )ZQFSWJTPS
    9FO ,7.
    6TFSMBOE
    $PQZ
    ˞ 43*07ͳͲ)ZQFSWJTPSͰ΋ύέοτίϐʔΦʔόϔουΛগͳ͘
    ͢Δ࣮૷͸͋Δ

    View Slide

  13. Linux Containers ͩͱ
    Single Kernel Ͱ଎͍

    View Slide

  14. Docker Filesystem
    -JOVY$POUBJOFST

    View Slide

  15. UNION Filesystem
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN

    View Slide

  16. IUUQTEPDTEPDLFSDPNUFSNTJNBHFTEPDLFSpMFTZTUFNTNVMUJMBZFSQOH
    UNION Filesystem Image

    View Slide

  17. 8SJUF*0 3FBE*0 ADBUFUDEFCJBO@WFSTJPOA

    View Slide

  18. 8SJUF*0 3FBE*0
    8SJUF*0͸࠷্૚΁ॻ͖ࠐΈ
    3FBE*0͸֘౰ϑΝΠϧΛ
    ֨ೲ͢Δ૚Λ୳ࡧ

    View Slide

  19. UNION Filesystem
    • AUFS, Btrfs, OverlayFS … ͳͲΛ࢖࣮ͬͯݱ
    • Copy On Write: ݪຊΛෳ੡ͨ͠;ΓΛͯ͠
    ͓͍ͯ৽نॻ͖ࠐΈ͕͋ͬͨΒෳ੡Λ࡞Δ
    • LinuxΧʔωϧඪ४ͷ Device MapperΛ࢖ͬ
    ͯ΋࣮ݱͰ͖Δ

    View Slide

  20. Storage drivers
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN

    View Slide

  21. EFWJDFNBQQFS
    BVGT
    CUSGT
    PWFSMBZGT

    View Slide

  22. EFWJDFNBQQFS
    BVGT
    CUSGT
    PWFSMBZGT
    6OJPO'4ͷ࣮૷Λબ୒Ͱ͖Δ

    View Slide

  23. Storage drivers
    devicemapper, aufs, btrfs, overrayfs
    (vfs)
    Χʔωϧඪ४ͳͨΊRHELܥͰͷɹɹ
    σϑΥϧτ͸ devicemapper

    View Slide

  24. Device Mapper
    ϒϩοΫσόΠε΁ͷI/Oʹ༷ʑͳม׵ΛՃ͑Δ͜ͱ͕
    Ͱ͖Δ (҉߸ԽɺετϥΠϓɺϛϥʔ ͳͲ)
    ϒϩοΫσόΠευϥΠόϨϕϧͷ࣮૷ͳͷͰɹɹɹɹ
    ಛఆͷϑΝΠϧγεςϜʹґଘ͠ͳ͍
    LVMͳͲͰ࢖ΘΕ͍ͯΔ
    docker commit ͕૸ΔͱsnapshotσόΠεΛ࡞੒ͯ͠Π
    ϝʔδ૚ͱ͢Δ

    View Slide

  25. Volume
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN
    4UPSBHF%SJWFST

    View Slide

  26. Volume
    ίϯςφؒͰσΟϨΫτϦΛڞ༗͢ΔͨΊͷ΋ͷ
    ίϯςφ͝ͱͰ͸ͳ͘DockerάϩʔόϧͳྖҬʹ֨
    ೲ͞ΕΔ
    ࢦఆͨ͠σΟϨΫτϦҎԼͷΠϝʔδ૚ΛόΠύε͠
    ͯΞΫηε
    I/Oཁٻ͕Union FS෦෼Λ௨Βͳ͍ͷͰΦʔόϔου
    ͕গͳ͍

    View Slide

  27. howto/Volume
    EPDLFSSVOWWBSMJCNZTRMNZTRM
    70-6.&WBSMJCNZTRM
    EPDLFSSVONZTRM
    %PDLFSpMF
    4IFMM
    4IFMM
    PS

    View Slide

  28. Docker Network
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN
    4UPSBHF%SJWFST
    7PMVNF

    View Slide

  29. Portmapper
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN
    4UPSBHF%SJWFST
    7PMVNF
    %PDLFS/FUXPSL

    View Slide

  30. Portmapper
    ίϯςφؒ௨৴΍ίϯςφɾϗετؒ௨৴͸ϗετଆ
    ͷ iptables ͰNAT
    !
    iptables͕ͳ͍؀ڥͩͱಠࣗͷϢʔβϥϯυͷϓϩη
    ε(docker-proxy)Ͱίϯςφؒ௨৴
    "%0$,&3JEPDLFSQUDQNUDQŠEQPSUK%/"5
    ŠUPEFTUJOBUJPO
    EPDLFSQSPYZQSPUPUDQIPTUJQIPTUQPSU
    DPOUBJOFSJQDPOUBJOFSQPSU

    View Slide

  31. Host Networking
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN
    4UPSBHF%SJWFST
    7PMVNF
    %PDLFS/FUXPSL
    1PSUNBQQFS

    View Slide

  32. Host Networking
    ίϯςφ༻ͷNetwork NamespaceΛ࡞Β
    ͣʹϗετͷωοτϫʔΫΛͦͷ··࢖͏
    ίϯςφ͸ϗετଆͷportΛlisten͢Δ
    iptables΍docker-proxyΛܦ༝͠ͳͯ͘Α
    ͘ͳΓɺΦʔόϔου͕খ͘͞ͳΔ

    View Slide

  33. howto/Host Networking
    !
    ŠFYFDESJWFSOBUJWF
    PS
    -9$Ҏ্ͰŠFYFDESJWFSMYD
    ͕ඞཁ
    EPDLFSSVOŠOFUIPTUNZTRM

    View Slide

  34. -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN
    4UPSBHF%SJWFST
    7PMVNF
    %PDLFS/FUXPSL
    1PSUNBQQFS
    )PTU/FUXPSLJOH
    DockerͷύϑΥʔϚϯεʹ
    ͓͍ͯॏཁͳ͜ͱ͸ͳʹ͔

    View Slide

  35. 2
    DockerԽͨ͠
    ISUCONΞϓϦέʔγϣϯ
    ͷϕϯνϚʔΫ

    View Slide

  36. ISUCON
    *JLBOKJOJ4QFFE6Q$0/UFTU

    View Slide

  37. /HJOY
    CFODI
    NBSLFS
    .Z42-
    "QQ
    .FNDBDIFE
    ISUCON4 ༧બ

    View Slide

  38. Machine & Software Spec
    instance type: m3.xlarge
    CPU: Xeon E5-2670 v2 @ 2.50GHz 4 vCPU
    Memory: 16GB RAM
    Storage: Magnetic volume
    OS: Ubuntu 14.04 LTS Kernel 3.18.0
    Docker: 1.4.1 (latest)
    MySQL: 5.5.40, memcached: 1.4.14, Nginx:1.4.6

    View Slide

  39. νϡʔχϯά಺༰
    • είΞ 38446 (໿ 3000 req/s)
    • ࢼߦ͝ͱʹ +-1000 είΞఔ౓ͷޡࠩ
    • ༧બಥഁϨϕϧ
    • σʔλ͸શ෦ϝϞϦʹ৐Δ
    • ηογϣϯ৘ใͳͲ͸ memcached
    • Nginx Ͱ੩తϑΝΠϧΛฦ͢
    • ωοτϫʔΫελοΫɺNginx, MySQL ͸ී௨ͷ
    νϡʔχϯά

    View Slide

  40. Evaluation

    View Slide

  41. • Nginx ͱ MySQL ΛͦΕͧΕDockerԽ
    • ҎԼͷ֤ํࣜΛnative(default)ͱൺֱ
    • Nginx͚ͩDockerԽ
    • —net=host ͱ —net=bridge
    • MySQL͚ͩDockerԽ
    • storage-driver=devicemapper ͱ
    storage-driver=overlayfs
    • Volume ͷ ON/OFF

    View Slide

  42. Result

    View Slide

  43. EFGBVMU
    /HJOY
    OFUCSJEHF
    /HJOY
    OFUIPTU

    View Slide

  44. /HJOYʹύέοτ͕ू໿͢Δ
    /"15͢ΔΦʔόϔου͕ߴ͍

    View Slide

  45. EFGBVMU
    .Z42-
    EFWJDFNBQQFS
    WPMVNFP⒎
    .Z42-
    EFWJDFNBQQFS
    WPMVNFPO
    .Z42-
    PWFSMBZGT
    WPMVNFP⒎
    .Z42-
    PWFSMBZGT
    WPMVNFPO

    View Slide

  46. 7PMVNFͷ0/0''ͱ
    TUPSBHFESJWFSʹΑΔ
    ੑೳྼԽͳ͠

    View Slide

  47. 7PMVNF0''ͰมΘΒͳ͍
    3FBE*0͸ϝϞϦʹશ෦ͷΔ
    8SJUF*0͸࠷্૚͚ͩ

    View Slide

  48. NAPTͷߴ଎Խ

    View Slide

  49. -A OUTPUT ! -d 127.0.0.0/8
    -m addrtype --dst-type LOCAL
    -j DOCKER

    View Slide

  50. -A OUTPUT E
    -m addrtype --dst-type LOCAL
    -j DOCKER

    View Slide

  51. 127.0.0.1Ѽͩͱiptablesͷ
    DockerνΣʔϯʹೖΒͳ͍

    View Slide

  52. docker-proxy
    EPDLFSQSPYZQSPUPUDQIPTUJQ
    IPTUQPSU
    DPOUBJOFSJQDPOUBJOFSQPSUA

    View Slide

  53. docker-proxy
    EPDLFSQSPYZQSPUPUDQIPTUJQ
    IPTUQPSU
    DPOUBJOFSJQDPOUBJOFSQPSUA
    CPU 50%

    View Slide

  54. benchmarker bench
    --host 172.31.8.235
    --init ~/init.sh
    --workload 8

    View Slide

  55. 32172 37931

    View Slide

  56. ͳͥ userland ͷ
    proxy ͕͍Δͷ͔

    View Slide

  57. IUUQTHJUIVCDPNEPDLFSEPDLFSQVMM

    View Slide

  58. Hairpin NAT

    View Slide

  59. )PTU $POUBJOFS
    EPDLFS

    -*45&/
    FUI
    FUI
    MP
    FUI
    FUI

    View Slide

  60. ETU
    MP
    FUI
    FUI
    )PTU $POUBJOFS
    EPDLFS

    -*45&/
    FUI
    FUI

    View Slide

  61. ETU
    "%0$,&3JEPDLFSQUDQNUDQEQPSU
    K%/"5UPEFTUJOBUJPO
    )PTU $POUBJOFS
    MP
    FUI
    FUI
    EPDLFS

    -*45&/
    FUI
    FUI

    View Slide

  62. ETU
    "%0$,&3JEPDLFSQUDQNUDQEQPSU
    K%/"5UPEFTUJOBUJPO
    )PTU $POUBJOFS
    MP
    FUI
    FUI
    EPDLFS

    -*45&/
    -*45&/
    Ծ૝bridge(docker0)Λ
    Hairpin NAT modeʹ͠ͳ͍ͱ
    NAT͞Εͳ͍

    View Slide

  63. FDIPTZTDMBTTOFUCSCSJGFUIIBJSQJO@NPEF

    View Slide

  64. IUUQTHJUIVCDPNEPDLFSEPDLFSQVMM

    View Slide

  65. View Slide

  66. 3)&-BOE$FOU04
    SFBEPOMZTZT

    View Slide

  67. 1. DockerͷύϑΥʔϚϯεʹ͍ͭͯ
    ॏཁͳࣄ͸ͳʹ͔
    2. DockerԽͨ͠ISUCONΞϓϦέʔ
    γϣϯͷϕϯνϚʔΫ

    View Slide

  68. Linux Containers
    Docker Filesystem
    UNION Filesystem
    Storage drivers
    Volume
    Docker Network
    Portmapper
    Host Networking

    View Slide

  69. Linux Containers
    Docker Filesystem
    UNION Filesystem
    Storage drivers
    Volume
    Docker Network
    Portmapper (Performance issue)
    Host Networking

    View Slide

  70. View Slide

  71. IUUQIBUFOBDPSQKQSFDSVJUDBSFFSTBMFTFOHJOFFS
    ηʔϧεΤϯδχΞ৬

    View Slide

  72. !
    8FCΦϖϨʔγϣϯΤϯδχΞ
    Λืू͓ͯ͠Γ·͢

    View Slide