'D'evelopment and 'D'eployment with 'D'ocker at 'D'wango

Transcript

1. Development and Deployment with Docker at Dwango @ixixi

2. %8"/(0
   $P -UE :VSJ
   0EBHJSJ
   
   !JYJYJ
   3FDPNNFOEBUJPO
   TZTUFN
   BSDIJUFDU

3. Development with Docker

4. /JDPOJDP
   "QQMJDBUJPO
   4FSWFS )551
   SFRVFTU 8FC
   4FSWFS $BTTBOESB 3FDPNNFOE
 "1*
   4FSWFS Recommendation API

   3FDPNNFOE
   4FSWFS 0UIFS
   4FSWJDFT MPHT $BDIF
   4FSWFS MPHT w%PDLFSҎલ͔ΒՔಇ͍ͯ͠ΔϨίϝϯυ"1*
   wχίχίಈըͷʮؔ࿈ಈըʯͳͲΛฦ͢ɻ

5. Ϩίϝϯυͷ֬ೝͱςετ wϨίϝϯυͷγεςϜʹखΛՃ͑ͨ৔߹ʹɺϨίϝϯυ݁Ռ͕Ͳ͏
 มԽ͢Δ͔ͷ֬ೝ͸ɺίʔυϨϏϡʔɾ6OJU5FTUͰ͸อূͮ͠Β͍ɻ
   ! wQVMM
   SFRΛNBTUFSʹऔΓࠐΉຊ൪σϓϩΠͳͷͰɺ֬ೝແ͠Ͱͷ౤ ೖ͸ͨ͘͠ͳ͍ɻ
   ! wͲ͏ʹ͔ͯ͠ɺखܰʹʮ࣮ࡍͷ؀ڥΛ֬ೝʯग़དྷΔΑ͏ʹ͍ͨ͠ɻ

6. ΍Γ͍ͨ͜ͱ wQVMM
   SFRग़ͨ͠ΒɺϨίϝϯυʹؔΘΔ%PDLFSͷ
 ίϯςφҰἧ͍ lܥz ͕উखʹཱͭɻ
   wࣗ෼Ͱ؀ڥΛ࡞ͬͯࢼ͞ͳͯ͘΋ɺ
 ʮ͜͜Ͱطʹಈ͍͍ͯΔΑʯͱ͍͏ঢ়ଶ͕࠷ߴɻ
   ! wϨίϝϯυ"1*Ͱɺ࣮ࡍʹԿ͕Ϩίϝϯυ͞ΕΔ͔
 දࣔग़དྷΔ3FDPNNFOE
   7JFXFS

   ಺෦πʔϧ Ͱݟ͍ͨɻ طʹಈ͍͍ͯΔ

7. /JDPOJDP
   "QQMJDBUJPO
   4FSWFS )551
   SFRVFTU 8FC
   4FSWFS $BTTBOESB 3FDPNNFOE

   "1*
   4FSWFS Containerize! 3FDPNNFOE
   

   4FSWFS 0UIFS
   4FSWJDFT MPHT $BDIF
   4FSWFS MPHT 3FDPNNFOE
   7JFXFS )551
   SFRVFTU

8. pull request builderͰҰ্ཱࣜͪ͛&௨஌ wKFOLJOTͷQVMM
   SFRVFTU
   CVJMEFSΛར༻ɻ
   wKPC
   DIBJOͰςετ͔Βίϯςφ࡞੒ɺίϝϯτॻ͖ࠐΈ·Ͱ࣮ߦɻ QVMM
   SFRVFTU
   CVJMEFS QVMM
   SFRVFTU ௨ৗͷςετ QVMM
   SFRVFTUΛ൓өͨ͠
   EPDLFSίϯςφ࡞੒

   EPDLFSίϯςφͰ
   ܥΛҰࣜىಈ ઀ଓઌϙʔτ৘ใΛ
   QVMM
   SFRίϝϯτʹॻࠐ ίϝϯτʹॻ͖ࠐΈ

9. ίϯςφੜ੒ wQVMMSFRVFTUʹରԠͨ͠ίϯςφ໊ͷQSFpYΛ࡞Δɻ
 wྫ QVMMSFR൪߸͕Ͱ࣌ࠁ͕ͳΒ
 ɹɹQVMMSFR@SFDPNNFOE@BQJ@@U
 wಉҰQVMMSFR಺Ͱ΋ɺߋ৽͞ΕΔ౓ʹίϯςφੜ੒͍ͨ͠ͷͰɺ೔͕࣌ඞཁɻ
 wQVMMSFRऴྃ࣌ ޙड़ ʹ͜ͷQSFpYΛར༻ɻ

10. ίϯςφͰʮܥʯΛཱͯΔ wHJU
    QVMM͔ͯ͠ΒpH
    CVJME͢ΔSBLF
    UBTLΛ࣮ߦ
    wQVMM
    SFRʹରԠͨ͠CSBODIΛ࢖ͬͯCVJME͕૸Δɻ
    wΞϓϦߋ৽෦෼Ҏલ͸Ωϟογϡ͕ޮ͘ͷͰCVJME͸ߴ଎
    ! wੜ੒ͨ͠QSFpYΛ෇Ճͨ͠ίϯςφ໊ͰpH
    VQ͢ΔSBLF
    UBTL࣮ߦ
    ! wDBTTBOESBSFEJTͷαʔό͸ɺμϛʔσʔλ͕
 ೖͬͨίϯςφ্ཱ͕͕ͪΔɻ
    !

    wQVMM
    SFRͷίϝϯτͰɺىಈͤ͞Δ֤ίϯςφͷ
 CSBODIΛ໌ࣔࢦఆ͢Δ͜ͱ΋Մೳɻ
 ผͷSFQPTJUPSZͷQVMM
    SFR΋Ұॹʹద༻͢Δඞཁ͕͋Δ৔߹౳ $BTTBOESB 3FDPNNFE

    "1*
    4FSWFS 3FDPNNFOE 4FSWFS $BDIF
    4FSWFS 3FDPNNFOE
    7JFXFS MJOL MJOL MJOL MJOL

11. PullReqίϝϯτ΁ͷॻ͖ࠐΈ w֤ίϯςφׂ͕Γ౰ͯΒΕͨϙʔτΛऔಘɻ
    ! w઀ଓ৘ใΛ()FͷQVMM
    SFRίϝϯτʹॻ͘ɻ

12. Garbage Collection (daily) wQVMM
    SFRVFTU͕DMPTF͞ΕͨΒίϯςφΛࢭΊΔ࡟আ͢Δɻ
 wKFOLJOTͰQVMM
    SFRVFTUͷDMPTF
    IPPL͢Δ͜ͱ͕೉͔ͬͨͨ͠Ίɺ೔࣍Ͱ($ɻ DSPO 
 wEPDLFSͷίϯςφ໊ͷQSFpYΛQBSTFͯ͠ɺSFQPTJUPSZͱQVMM
    SFRVFTU൪߸Λऔಘɻ
 w౰֘QVMM
    SFRVFTU͕DMPTF͞Ε͍ͯΔ͔֬ೝ͠ɺด͍ͯͨ͡Βίϯςφఀࢭ࡟আɻ
    !

    wίϯςφ໊ͷQSFpYͰ͸ͳ͘FUDE΍SFEJTʹରԠΛอ࣋ͯ͠΋ྑ͔͕ͬͨɺߏ੒ͷγϯϓ ϧ͞Λॏࢹɻ

13. developmentฤ Ή͢ͼ wطଘͷϓϩμΫτΛ%PDLFSԽ͢Δͷ͸ਏ͍৔߹͕͋ͬͨɻ
    w؀ڥߏஙΛDIFGͰߦ͏લఏͷϓϩμΫτΛޙ͔Β%PDLFSԽ͢Δͷ͸େมʹײͨ͡ɻ
    ! wطଘ෦෼Λ%PDLFSԽ͢ΔΑΓɺ৽نͰ࡞Δ΋ͷΛ%PDLFSԽલఏͰߏங͢Δํ͕ྑ͍ɻ w1VMM
    3FRVFTUͰʮͦͷ৔Ͱ࣮ࡍʹ৮ΕΔ؀ڥ͕طʹಈ͍͍ͯΔʯͷ͸࠷ߴɻ

14. Deployment with Docker

15. Personalize API /JDPOJDP
    "QQMJDBUJPO
    4FSWFS )551
    SFRVFTU 8FC
    4FSWFS $BDIF %BUBCBTF 3FWFSTF
    

    1SPYZ 1FSTPOBMJ[F
    "1* • ύʔιφϥΠζʹར༻͢ΔAPI (golang࣮૷) • DockerલఏͰ৽نʹߏஙɻ

16. Containerize! • ύʔιφϥΠζAPI • ϩάసૹ&ू໿(fluentd) • CEPΤϯδϯ(Norikra) • ϩάอଘ (ES&InfluxDB)

    • ՄࢹԽ (Kibana&Grafana) • ϦόʔεϓϩΩγ 1FSTPOBMJ[F
    "1* /JDPOJDP
    "QQMJDBUJPO
    4FSWFS )551
    SFRVFTU $POUBJOFSJ[F 8FC
    4FSWFS $BDIF %BUBCBTF MPHTUPSF .FUSJDT
    7JTVBMJ[FS $&1 -PH
    $PMMFDUPS 3FWFSTF
    1SPYZ

17. Dataflow DP
    4FSWFS )551
    SFRVFTU $BDIF %BUBCBTF MPHTUPSF .FUSJDT
    7JTVBMJ[FS $&1 3FWFSTF
    

    1SPYZ -PBE
    #BMBODFS -PH
    "HHSFHBUPS MPH
    pMF %PDLFS
    )PTU %PDLFS
    )PTU %PDLFS
    )PTU %PDLFS
    )PTU "1* -PH
    $PMMFDUPS w ֤IPTU಺ͷϦόʔεϓϩΩγ OHJOY ܦ༝Ͱ"1*ʹϦΫΤετɻ
    w ϩάϑΝΠϧ͸ϗετଆϘϦϡʔϜΛϚ΢ϯτͯ͠ॻ͖ग़͠ɻ
    w qVFOUEίϯςφଆͰ΋ϩάσΟϨΫτϦΛϚ΢ϯτͯ͠ɺಡΈࠐΈɻ
    w ϩάू໿ίϯςφͰϩάΛूΊͯɺ$&1 /PSJLSB Ͱूܭͭͭ͠ʹอଘˠՄࢹԽ

18. Docker Build Architecture MPDBM ()F +FOLJOT 3FHJTUSZ
    $PNNPO QVTI
    TPVSDFT )PPL

    w %PDLFS
    *NBHFϏϧυ
    w ίϯςφͰܥΛ্ཱͪ͛ͯ݁߹ςετ
    w υΩϡϝϯτͷੜ੒
    w %PDLFS
    3FHJTUSZʹΠϝʔδొ࿥ "1* qVFOUE /HJOY SFHJTUFS
    "1*
    DPOUBJOFS
    *NBHF VCVOUV CVJMEͷى఺ͱͳΔJNBHF͸ɺ1SJWBUF
    SFHJTUSZʹஔ͍͓͖ͯɺ%PDLFS
    IVC
    SFHJTUSZͷ౎౓ར༻͸͠ͳ͍ɻ

19. Deploy Manager WebUI • WebUIσϓϩΠπʔϧ • DockerԽҎલ͔Β࢖͍ͬͯͨ಺੡πʔϧ • Capistrano task

    ࣮ߦ • Ϣʔβೝূ • σϓϩΠϩάͷϦΞϧλΠϜදࣔ&อଘ
 • DockerσϓϩΠ΋capistrano taskԽͯ͠ར༻
 σϓϩΠઌαʔόΛબ୒ͯ͠ɺϙνοͱ
 1click͚ͩͰແఀࢭσϓϩΠɻ • σϓϩΠͷrollbackλεΫ΋༻ҙɻ

20. Docker Deployment Architecture %FQMPZ
    .BOBHFS
    8FC6* 3FHJTUSZ
    ڞ௨ 3FHJTUSZ
 QSPEVDUJPOઃఆ "1*

    qVFOUE /HJOY "1*
    TFSWFS
    DPOpH 0QFSBUF QVMM qVFOUE 3FWFSTF
    1SPYZ "1* QVMM "1*
    TFSWFS
    DPOpH %PDLFS
    )PTU 0UIFS
    4FSWJDFT %FQMPZ
    $BQJTUSBOP
    UBTL )551
    SFRVFTU NPVOU • Capistrano taskͰɺίϯςφͷ
 ΠϝʔδΛ֤ϗετ͕pullͯ͠σϓϩΠɻ ! • ࣮ߦόΠφϦͷίϯςφΠϝʔδ͸։ൃऀ શһ͕pullՄೳɻ ! • production؀ڥͰ࢖͏configίϯςφ͸ɺ registry͝ͱ෼཭(ೝূ͕ҟͳΔ)ɻ

21. Deployment ~ step.1 1.APIίϯςφΠϝʔδͱɺઃఆΠϝʔδ ΛɺDockerHostଆͰpull͢Δ %PDLFS
    )PTU 3FHJTUSZ
    ڞ௨ 3FHJTUSZ
    ઃఆ

    "1* qVFOUE /HJOY "1*W EPDLFS
    JNBHFT "1*
    TFSWFS
    DPOpH "1*
    TFSWFS
    DPOpH "1*
    TFSWFS
    DPOpH NPVOU NPVOU 3FWFSTF
    1SPYZ "1*
    W qVFOUEMBUFTU

22. Deployment ~ step.1 1.APIίϯςφΠϝʔδͱɺઃఆΠϝʔδ ΛɺDockerHostଆͰpull͢Δ %PDLFS
    )PTU 3FHJTUSZ
    ڞ௨ 3FHJTUSZ
    ઃఆ

    "1* qVFOUE /HJOY EPDLFS
    QVMM
    BQJMBUFTU EPDLFS
    QVMM
    BQJ@DPOpHMBUFTU "1*W EPDLFS
    JNBHFT "1*W
    MBUFTU "1*
    TFSWFS
    DPOpH "1*
    TFSWFS
    DPOpH "1*
    TFSWFS
    DPOpH NPVOU NPVOU 3FWFSTF
    1SPYZ "1*
    W qVFOUEMBUFTU

23. Deployment ~ step.2 1.৽͍͠imageͷίϯςφΛىಈ 2.API͕ਖ਼͘͠ϦΫΤετΛฦͤΔ·Ͱ଴ͭ
 ɹ(࣮ࡍʹϨεϙϯεΛ֬ೝ) %PDLFS
    )PTU "1*
    W qVFOUEMBUFTU DPOpH

    3FWFSTF
    1SPYZ

24. Deployment ~ step.2 1.৽͍͠imageͷίϯςφΛىಈ 2.API͕ਖ਼͘͠ϦΫΤετΛฦͤΔ·Ͱ଴ͭ
 ɹ(࣮ࡍʹϨεϙϯεΛ֬ೝ) %PDLFS
    )PTU "1*
    W "1*
    W qVFOUEMBUFTU

    qVFOUEMBUFTU DPOpH DPOpH 3FWFSTF
    1SPYZ

25. Deployment ~ step.2 1.৽͍͠imageͷίϯςφΛىಈ 2.API͕ਖ਼͘͠ϦΫΤετΛฦͤΔ·Ͱ଴ͭ
 ɹ(࣮ࡍʹϨεϙϯεΛ֬ೝ) %PDLFS
    )PTU "1*
    W "1*
    W qVFOUEMBUFTU

    qVFOUEMBUFTU DPOpH DPOpH IFBMUIDIFDL EVNNZ
    SFRVFTU 3FWFSTF
    1SPYZ

26. Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$

    kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS
    )PTU "1*
    W "1*
    W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF
    1SPYZ DPOpH DPOpH OHJOYDPOG
      OHJOY 
    QPSU
    

27. Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$

    kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS
    )PTU "1*
    W "1*
    W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF
    1SPYZ DPOpH DPOpH OHJOYDPOG
    
    
    MPDBUJPO
    
    \
    
    
    
    
    QSPYZ@QBTT
    IUUQ
    
    
    ^   OHJOY 
    QPSU
    

28. Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$

    kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS
    )PTU "1*
    W "1*
    W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF
    1SPYZ DPOpH DPOpH OHJOYDPOG
    
    
    MPDBUJPO
    
    \
    
    
    
    
    QSPYZ@QBTT
    IUUQ
    
    
    ^  
    
    MPDBUJPO
    
    \
    
    
    
    
    QSPYZ@QBTT
    IUUQ
    
    
    ^ OHJOY 
    QPSU
    

29. Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$

    kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS
    )PTU "1*
    W "1*
    W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF
    1SPYZ DPOpH DPOpH OHJOYDPOG
      OHJOY 
    QPSU
    

30. Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$

    kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS
    )PTU "1*
    W "1*
    W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF
    1SPYZ DPOpH DPOpH OHJOYDPOG
      OHJOY OHJOY 
    QPSU
    

31. Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$

    kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS
    )PTU "1*
    W "1*
    W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF
    1SPYZ DPOpH DPOpH OHJOYDPOG
      OHJOY 
    QPSU
    

32. Deployment ~ step.4 1.ݹ͍ίϯςφͷఀࢭ&࡟আ 2.ݹ͍ίϯςφΠϝʔδͷ࡟আ
 
 ˞ߴ଎ʹrollbackग़དྷΔΑ͏ɺ࣮ࡍʹ͸
 ɹ5ੈ୅લͷΠϝʔδΛ࡟আ %PDLFS
    )PTU "1*
    W

    "1*
    W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF
    1SPYZ DPOpH DPOpH OHJOY "1*W EPDLFS
    JNBHFT "1*W
    MBUFTU "1*
    TFSWFS
    DPOpH

33. Deployment ~ step.4 1.ݹ͍ίϯςφͷఀࢭ&࡟আ 2.ݹ͍ίϯςφΠϝʔδͷ࡟আ
 
 ˞ߴ଎ʹrollbackग़དྷΔΑ͏ɺ࣮ࡍʹ͸
 ɹ5ੈ୅લͷΠϝʔδΛ࡟আ %PDLFS
    )PTU "1*
    W

    qVFOUEMBUFTU 3FWFSTF
    1SPYZ DPOpH OHJOY "1*W EPDLFS
    JNBHFT "1*W
    MBUFTU "1*
    TFSWFS
    DPOpH

34. Deployment ~ step.4 1.ݹ͍ίϯςφͷఀࢭ&࡟আ 2.ݹ͍ίϯςφΠϝʔδͷ࡟আ
 
 ˞ߴ଎ʹrollbackग़དྷΔΑ͏ɺ࣮ࡍʹ͸
 ɹ5ੈ୅લͷΠϝʔδΛ࡟আ %PDLFS
    )PTU "1*
    W

    qVFOUEMBUFTU 3FWFSTF
    1SPYZ DPOpH OHJOY EPDLFS
    JNBHFT "1*W
    MBUFTU "1*
    TFSWFS
    DPOpH

35. deploymentฤ Ή͢ͼ wμ΢ϯλΠϜແ͠ͷσϓϩΠ͸Մೳ͕ͩɺݱঢ়ɺࣗલͰগ͠࡞Γ͜Ήඞཁ͕͋Δɻ
    wࣗલͷίϯςφ੾ସલϔϧενΣοΫɻ
    ! wϗετଆͷ؀ڥ͕ԚΕͳ͍ͷ͸ૉ੖Β͍͠ɻ
    ! w௕ظӡ༻͢ΔલఏͳΒΰϛΛย෇͚ΔػߏΛ࣋ͱ͏ɻ
 wLTΛ࢖͏ʹ͸ɺNBTUFSͷ৑௕Խ΍ߴՄ༻ੑͳFUDEΫϥελͷߏஙɾӡ༻౳ɺ
 ن໛ͷׂʹ߹Θͳ͔ͬͨͷͰʮϦιʔεͷந৅Խʯ·Ͱ͸౿Έࠐ·ͳͳ͔ͬͨɻ


    wαʔϏεԣஅͰར༻Մೳͳେن໛σϓϩΠઌαʔόࢿݯͱͯ͠ந৅Խ͢Δ৔߹ʹ͸ɺ
 LT౳͕ඞཁʹͳΔͱࢥ͏ɻن໛ͱ໨తʹԠͯ͡ΞʔΩςΫνϟઃܭ͠Α͏ɻ

36. υϫϯΰ޿ࠂ 
 ͜ͷهࣄ͸υϫϯΰۈ຿࣌ؒதʹॻ͔Εͨɻ ! υϫϯΰ͸ຊ෺ͷ
 (C++|Elrang|ݕࡧ|Ϩίϝϯυ|σʔλ෼ੳ|σʔλ࿈ܞ|FPGA|Πϯϑϥ|WEBΞϓϦ)
 ΤϯδχΞΛืू͍ͯ͠·͢ɻ ࠾༻৘ใ | גࣜձࣾυϫϯΰ