Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Docker Performance on Web Application

Docker Performance on Web Application

Docker Meetup Tokyo #4

Yuuki Tsubouchi (yuuk1)

January 17, 2015
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Technology

Transcript

  1. Docker Performance
    on Web Application
    JEZ@VVLJ
    %PDLFS.FFUVQ5PLZP

    View full-size slide

  2. id:y_uuki
    ӡ༻
    @y_uuk1

    View full-size slide

  3. Docker Engine্Ͱɹ
    ΞϓϦέʔγϣϯΛɹ
    ಈ͔ͯ͠ੑೳྼԽ
    ͠ͳ͍ͷʁ

    View full-size slide

  4. 1
    DockerͷύϑΥʔϚϯεʹ
    ͓͍ͯॏཁͳ͜ͱ͸ͳʹ͔

    View full-size slide

  5. Linux Containers

    View full-size slide

  6. /FUXPSL
    /BNFTQBDF
    *1$
    /BNFTQBDF
    654
    /BNFTQBDF
    .PVOU
    /BNFTQBDF
    6TFS
    /BNFTQBDF
    1*%
    /BNFTQBDF

    View full-size slide

  7. /FUXPSL
    /BNFTQBDF
    *1$
    /BNFTQBDF
    654
    /BNFTQBDF
    .PVOU
    /BNFTQBDF
    6TFS
    /BNFTQBDF
    1*%
    /BNFTQBDF
    -JOVYΧʔωϧͷ໊લۭؒػೳͷ
    ू·Γ

    View full-size slide

  8. Linux Containers
    Χʔωϧͷ֤ϦιʔεΛ໊લۭؒͰִ཭͠
    ۭͨؒͰϓϩηεΛ࣮ߦ͢Δͱ͍͏Ϟσϧ
    ϑΝΠϧγεςϜɼϢʔβɼϓϩηεςʔ
    ϒϧɼωοτϫʔΫ… etc
    LXC ͸ Linux Containers ͷϑϩϯτΤϯυ

    View full-size slide

  9. Linux Containers Overhead
    ୯ମͷLinux ΧʔωϧͰ׬݁͢ΔͷͰ
    ΦϒδΣΫτͷڞ༗ޮ཰͕Α͍
    Hypervisor ͷΑ͏ʹ֤ϦιʔεΛɹɹ
    ೋॏॲཧ͠ͳͯ͘Α͍

    View full-size slide

  10. example/packet receive
    /*$ ,FSOFM 6TFSMBOE
    $PQZ $PQZ
    4JOHMF
    ,FSOFM

    View full-size slide

  11. example/packet receive
    /*$ ,FSOFM 6TFSMBOE
    $PQZ $PQZ
    /*$ ,FSOFM ,FSOFM
    $PQZ $PQZ
    4JOHMF
    ,FSOFM
    )ZQFSWJTPS
    9FO ,7.
    6TFSMBOE
    $PQZ
    ˞ 43*07ͳͲ)ZQFSWJTPSͰ΋ύέοτίϐʔΦʔόϔουΛগͳ͘
    ͢Δ࣮૷͸͋Δ

    View full-size slide

  12. Linux Containers ͩͱ
    Single Kernel Ͱ଎͍

    View full-size slide

  13. Docker Filesystem
    -JOVY$POUBJOFST

    View full-size slide

  14. UNION Filesystem
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN

    View full-size slide

  15. IUUQTEPDTEPDLFSDPNUFSNTJNBHFTEPDLFSpMFTZTUFNTNVMUJMBZFSQOH
    UNION Filesystem Image

    View full-size slide

  16. 8SJUF*0 3FBE*0 ADBUFUDEFCJBO@WFSTJPOA

    View full-size slide

  17. 8SJUF*0 3FBE*0
    8SJUF*0͸࠷্૚΁ॻ͖ࠐΈ
    3FBE*0͸֘౰ϑΝΠϧΛ
    ֨ೲ͢Δ૚Λ୳ࡧ

    View full-size slide

  18. UNION Filesystem
    • AUFS, Btrfs, OverlayFS … ͳͲΛ࢖࣮ͬͯݱ
    • Copy On Write: ݪຊΛෳ੡ͨ͠;ΓΛͯ͠
    ͓͍ͯ৽نॻ͖ࠐΈ͕͋ͬͨΒෳ੡Λ࡞Δ
    • LinuxΧʔωϧඪ४ͷ Device MapperΛ࢖ͬ
    ͯ΋࣮ݱͰ͖Δ

    View full-size slide

  19. Storage drivers
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN

    View full-size slide

  20. EFWJDFNBQQFS
    BVGT
    CUSGT
    PWFSMBZGT

    View full-size slide

  21. EFWJDFNBQQFS
    BVGT
    CUSGT
    PWFSMBZGT
    6OJPO'4ͷ࣮૷Λબ୒Ͱ͖Δ

    View full-size slide

  22. Storage drivers
    devicemapper, aufs, btrfs, overrayfs
    (vfs)
    Χʔωϧඪ४ͳͨΊRHELܥͰͷɹɹ
    σϑΥϧτ͸ devicemapper

    View full-size slide

  23. Device Mapper
    ϒϩοΫσόΠε΁ͷI/Oʹ༷ʑͳม׵ΛՃ͑Δ͜ͱ͕
    Ͱ͖Δ (҉߸ԽɺετϥΠϓɺϛϥʔ ͳͲ)
    ϒϩοΫσόΠευϥΠόϨϕϧͷ࣮૷ͳͷͰɹɹɹɹ
    ಛఆͷϑΝΠϧγεςϜʹґଘ͠ͳ͍
    LVMͳͲͰ࢖ΘΕ͍ͯΔ
    docker commit ͕૸ΔͱsnapshotσόΠεΛ࡞੒ͯ͠Π
    ϝʔδ૚ͱ͢Δ

    View full-size slide

  24. Volume
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN
    4UPSBHF%SJWFST

    View full-size slide

  25. Volume
    ίϯςφؒͰσΟϨΫτϦΛڞ༗͢ΔͨΊͷ΋ͷ
    ίϯςφ͝ͱͰ͸ͳ͘DockerάϩʔόϧͳྖҬʹ֨
    ೲ͞ΕΔ
    ࢦఆͨ͠σΟϨΫτϦҎԼͷΠϝʔδ૚ΛόΠύε͠
    ͯΞΫηε
    I/Oཁٻ͕Union FS෦෼Λ௨Βͳ͍ͷͰΦʔόϔου
    ͕গͳ͍

    View full-size slide

  26. howto/Volume
    EPDLFSSVOWWBSMJCNZTRMNZTRM
    70-6.&WBSMJCNZTRM
    EPDLFSSVONZTRM
    %PDLFSpMF
    4IFMM
    4IFMM
    PS

    View full-size slide

  27. Docker Network
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN
    4UPSBHF%SJWFST
    7PMVNF

    View full-size slide

  28. Portmapper
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN
    4UPSBHF%SJWFST
    7PMVNF
    %PDLFS/FUXPSL

    View full-size slide

  29. Portmapper
    ίϯςφؒ௨৴΍ίϯςφɾϗετؒ௨৴͸ϗετଆ
    ͷ iptables ͰNAT
    !
    iptables͕ͳ͍؀ڥͩͱಠࣗͷϢʔβϥϯυͷϓϩη
    ε(docker-proxy)Ͱίϯςφؒ௨৴
    "%0$,&3JEPDLFSQUDQNUDQŠEQPSUK%/"5
    ŠUPEFTUJOBUJPO
    EPDLFSQSPYZQSPUPUDQIPTUJQIPTUQPSU
    DPOUBJOFSJQDPOUBJOFSQPSU

    View full-size slide

  30. Host Networking
    -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN
    4UPSBHF%SJWFST
    7PMVNF
    %PDLFS/FUXPSL
    1PSUNBQQFS

    View full-size slide

  31. Host Networking
    ίϯςφ༻ͷNetwork NamespaceΛ࡞Β
    ͣʹϗετͷωοτϫʔΫΛͦͷ··࢖͏
    ίϯςφ͸ϗετଆͷportΛlisten͢Δ
    iptables΍docker-proxyΛܦ༝͠ͳͯ͘Α
    ͘ͳΓɺΦʔόϔου͕খ͘͞ͳΔ

    View full-size slide

  32. howto/Host Networking
    !
    ŠFYFDESJWFSOBUJWF
    PS
    -9$Ҏ্ͰŠFYFDESJWFSMYD
    ͕ඞཁ
    EPDLFSSVOŠOFUIPTUNZTRM

    View full-size slide

  33. -JOVY$POUBJOFST
    %PDLFS'JMFTZTUFN
    6/*0/'JMFTZTUFN
    4UPSBHF%SJWFST
    7PMVNF
    %PDLFS/FUXPSL
    1PSUNBQQFS
    )PTU/FUXPSLJOH
    DockerͷύϑΥʔϚϯεʹ
    ͓͍ͯॏཁͳ͜ͱ͸ͳʹ͔

    View full-size slide

  34. 2
    DockerԽͨ͠
    ISUCONΞϓϦέʔγϣϯ
    ͷϕϯνϚʔΫ

    View full-size slide

  35. ISUCON
    *JLBOKJOJ4QFFE6Q$0/UFTU

    View full-size slide

  36. /HJOY
    CFODI
    NBSLFS
    .Z42-
    "QQ
    .FNDBDIFE
    ISUCON4 ༧બ

    View full-size slide

  37. Machine & Software Spec
    instance type: m3.xlarge
    CPU: Xeon E5-2670 v2 @ 2.50GHz 4 vCPU
    Memory: 16GB RAM
    Storage: Magnetic volume
    OS: Ubuntu 14.04 LTS Kernel 3.18.0
    Docker: 1.4.1 (latest)
    MySQL: 5.5.40, memcached: 1.4.14, Nginx:1.4.6

    View full-size slide

  38. νϡʔχϯά಺༰
    • είΞ 38446 (໿ 3000 req/s)
    • ࢼߦ͝ͱʹ +-1000 είΞఔ౓ͷޡࠩ
    • ༧બಥഁϨϕϧ
    • σʔλ͸શ෦ϝϞϦʹ৐Δ
    • ηογϣϯ৘ใͳͲ͸ memcached
    • Nginx Ͱ੩తϑΝΠϧΛฦ͢
    • ωοτϫʔΫελοΫɺNginx, MySQL ͸ී௨ͷ
    νϡʔχϯά

    View full-size slide

  39. • Nginx ͱ MySQL ΛͦΕͧΕDockerԽ
    • ҎԼͷ֤ํࣜΛnative(default)ͱൺֱ
    • Nginx͚ͩDockerԽ
    • —net=host ͱ —net=bridge
    • MySQL͚ͩDockerԽ
    • storage-driver=devicemapper ͱ
    storage-driver=overlayfs
    • Volume ͷ ON/OFF

    View full-size slide

  40. EFGBVMU
    /HJOY
    OFUCSJEHF
    /HJOY
    OFUIPTU

    View full-size slide

  41. /HJOYʹύέοτ͕ू໿͢Δ
    /"15͢ΔΦʔόϔου͕ߴ͍

    View full-size slide

  42. EFGBVMU
    .Z42-
    EFWJDFNBQQFS
    WPMVNFP⒎
    .Z42-
    EFWJDFNBQQFS
    WPMVNFPO
    .Z42-
    PWFSMBZGT
    WPMVNFP⒎
    .Z42-
    PWFSMBZGT
    WPMVNFPO

    View full-size slide

  43. 7PMVNFͷ0/0''ͱ
    TUPSBHFESJWFSʹΑΔ
    ੑೳྼԽͳ͠

    View full-size slide

  44. 7PMVNF0''ͰมΘΒͳ͍
    3FBE*0͸ϝϞϦʹશ෦ͷΔ
    8SJUF*0͸࠷্૚͚ͩ

    View full-size slide

  45. NAPTͷߴ଎Խ

    View full-size slide

  46. -A OUTPUT ! -d 127.0.0.0/8
    -m addrtype --dst-type LOCAL
    -j DOCKER

    View full-size slide

  47. -A OUTPUT E
    -m addrtype --dst-type LOCAL
    -j DOCKER

    View full-size slide

  48. 127.0.0.1Ѽͩͱiptablesͷ
    DockerνΣʔϯʹೖΒͳ͍

    View full-size slide

  49. docker-proxy
    EPDLFSQSPYZQSPUPUDQIPTUJQ
    IPTUQPSU
    DPOUBJOFSJQDPOUBJOFSQPSUA

    View full-size slide

  50. docker-proxy
    EPDLFSQSPYZQSPUPUDQIPTUJQ
    IPTUQPSU
    DPOUBJOFSJQDPOUBJOFSQPSUA
    CPU 50%

    View full-size slide

  51. benchmarker bench
    --host 172.31.8.235
    --init ~/init.sh
    --workload 8

    View full-size slide

  52. ͳͥ userland ͷ
    proxy ͕͍Δͷ͔

    View full-size slide

  53. IUUQTHJUIVCDPNEPDLFSEPDLFSQVMM

    View full-size slide

  54. )PTU $POUBJOFS
    EPDLFS

    -*45&/
    FUI
    FUI
    MP
    FUI
    FUI

    View full-size slide

  55. ETU
    MP
    FUI
    FUI
    )PTU $POUBJOFS
    EPDLFS

    -*45&/
    FUI
    FUI

    View full-size slide

  56. ETU
    "%0$,&3JEPDLFSQUDQNUDQEQPSU
    K%/"5UPEFTUJOBUJPO
    )PTU $POUBJOFS
    MP
    FUI
    FUI
    EPDLFS

    -*45&/
    FUI
    FUI

    View full-size slide

  57. ETU
    "%0$,&3JEPDLFSQUDQNUDQEQPSU
    K%/"5UPEFTUJOBUJPO
    )PTU $POUBJOFS
    MP
    FUI
    FUI
    EPDLFS

    -*45&/
    -*45&/
    Ծ૝bridge(docker0)Λ
    Hairpin NAT modeʹ͠ͳ͍ͱ
    NAT͞Εͳ͍

    View full-size slide

  58. FDIPTZTDMBTTOFUCSCSJGFUIIBJSQJO@NPEF

    View full-size slide

  59. IUUQTHJUIVCDPNEPDLFSEPDLFSQVMM

    View full-size slide

  60. 3)&-BOE$FOU04
    SFBEPOMZTZT

    View full-size slide

  61. 1. DockerͷύϑΥʔϚϯεʹ͍ͭͯ
    ॏཁͳࣄ͸ͳʹ͔
    2. DockerԽͨ͠ISUCONΞϓϦέʔ
    γϣϯͷϕϯνϚʔΫ

    View full-size slide

  62. Linux Containers
    Docker Filesystem
    UNION Filesystem
    Storage drivers
    Volume
    Docker Network
    Portmapper
    Host Networking

    View full-size slide

  63. Linux Containers
    Docker Filesystem
    UNION Filesystem
    Storage drivers
    Volume
    Docker Network
    Portmapper (Performance issue)
    Host Networking

    View full-size slide

  64. IUUQIBUFOBDPSQKQSFDSVJUDBSFFSTBMFTFOHJOFFS
    ηʔϧεΤϯδχΞ৬

    View full-size slide

  65. !
    8FCΦϖϨʔγϣϯΤϯδχΞ
    Λืू͓ͯ͠Γ·͢

    View full-size slide