Tweet
Tweet

Slide 1

Slide 1 text

Development and Deployment with Docker at Dwango @ixixi

Slide 2

Slide 2 text

%8"/(0$P -UE :VSJ0EBHJSJ !JYJYJ 3FDPNNFOEBUJPOTZTUFNBSDIJUFDU

Slide 3

Slide 3 text

Development with Docker

Slide 4

Slide 4 text

/JDPOJDP "QQMJDBUJPO4FSWFS )551SFRVFTU 8FC 4FSWFS $BTTBOESB 3FDPNNFOE
 "1*4FSWFS Recommendation API 3FDPNNFOE 4FSWFS 0UIFS4FSWJDFT MPHT $BDIF4FSWFS MPHT w%PDLFSҎલ͔ΒՔಇ͍ͯ͠ΔϨίϝϯυ"1* wχίχίಈըͷʮؔ࿈ಈըʯͳͲΛฦ͢ɻ

Slide 5

Slide 5 text

Ϩίϝϯυͷ֬ೝͱςετ wϨίϝϯυͷγεςϜʹखΛՃ͑ͨ৔߹ʹɺϨίϝϯυ݁Ռ͕Ͳ͏
 มԽ͢Δ͔ͷ֬ೝ͸ɺίʔυϨϏϡʔɾ6OJU5FTUͰ͸อূͮ͠Β͍ɻ ! wQVMMSFRΛNBTUFSʹऔΓࠐΉຊ൪σϓϩΠͳͷͰɺ֬ೝແ͠Ͱͷ౤ ೖ͸ͨ͘͠ͳ͍ɻ ! wͲ͏ʹ͔ͯ͠ɺखܰʹʮ࣮ࡍͷ؀ڥΛ֬ೝʯग़དྷΔΑ͏ʹ͍ͨ͠ɻ

Slide 6

Slide 6 text

΍Γ͍ͨ͜ͱ wQVMMSFRग़ͨ͠ΒɺϨίϝϯυʹؔΘΔ%PDLFSͷ
 ίϯςφҰἧ͍ lܥz ͕উखʹཱͭɻ wࣗ෼Ͱ؀ڥΛ࡞ͬͯࢼ͞ͳͯ͘΋ɺ
 ʮ͜͜Ͱطʹಈ͍͍ͯΔΑʯͱ͍͏ঢ়ଶ͕࠷ߴɻ ! wϨίϝϯυ"1*Ͱɺ࣮ࡍʹԿ͕Ϩίϝϯυ͞ΕΔ͔
 දࣔग़དྷΔ3FDPNNFOE7JFXFS ಺෦πʔϧ Ͱݟ͍ͨɻ طʹಈ͍͍ͯΔ

Slide 7

Slide 7 text

/JDPOJDP "QQMJDBUJPO4FSWFS )551SFRVFTU 8FC 4FSWFS $BTTBOESB 3FDPNNFOE
 "1*4FSWFS Containerize! 3FDPNNFOE 4FSWFS 0UIFS4FSWJDFT MPHT $BDIF4FSWFS MPHT 3FDPNNFOE 7JFXFS )551SFRVFTU

Slide 8

Slide 8 text

pull request builderͰҰ্ཱࣜͪ͛&௨஌ wKFOLJOTͷQVMMSFRVFTUCVJMEFSΛར༻ɻ wKPCDIBJOͰςετ͔Βίϯςφ࡞੒ɺίϝϯτॻ͖ࠐΈ·Ͱ࣮ߦɻ QVMMSFRVFTUCVJMEFS QVMMSFRVFTU ௨ৗͷςετ QVMMSFRVFTUΛ൓өͨ͠ EPDLFSίϯςφ࡞੒ EPDLFSίϯςφͰ ܥΛҰࣜىಈ ઀ଓઌϙʔτ৘ใΛ QVMMSFRίϝϯτʹॻࠐ ίϝϯτʹॻ͖ࠐΈ

Slide 9

Slide 9 text

ίϯςφੜ੒ wQVMMSFRVFTUʹରԠͨ͠ίϯςφ໊ͷQSFpYΛ࡞Δɻ
 wྫ QVMMSFR൪߸͕Ͱ࣌ࠁ͕ͳΒ
 ɹɹQVMMSFR@SFDPNNFOE@BQJ@@U
 wಉҰQVMMSFR಺Ͱ΋ɺߋ৽͞ΕΔ౓ʹίϯςφੜ੒͍ͨ͠ͷͰɺ೔͕࣌ඞཁɻ
 wQVMMSFRऴྃ࣌ ޙड़ ʹ͜ͷQSFpYΛར༻ɻ

Slide 10

Slide 10 text

ίϯςφͰʮܥʯΛཱͯΔ wHJUQVMM͔ͯ͠ΒpHCVJME͢ΔSBLFUBTLΛ࣮ߦ wQVMMSFRʹରԠͨ͠CSBODIΛ࢖ͬͯCVJME͕૸Δɻ wΞϓϦߋ৽෦෼Ҏલ͸Ωϟογϡ͕ޮ͘ͷͰCVJME͸ߴ଎ ! wੜ੒ͨ͠QSFpYΛ෇Ճͨ͠ίϯςφ໊ͰpHVQ͢ΔSBLFUBTL࣮ߦ ! wDBTTBOESBSFEJTͷαʔό͸ɺμϛʔσʔλ͕
 ೖͬͨίϯςφ্ཱ͕͕ͪΔɻ ! wQVMMSFRͷίϝϯτͰɺىಈͤ͞Δ֤ίϯςφͷ
 CSBODIΛ໌ࣔࢦఆ͢Δ͜ͱ΋Մೳɻ
 ผͷSFQPTJUPSZͷQVMMSFR΋Ұॹʹద༻͢Δඞཁ͕͋Δ৔߹౳ $BTTBOESB 3FDPNNFE
 "1*4FSWFS 3FDPNNFOE 4FSWFS $BDIF 4FSWFS 3FDPNNFOE 7JFXFS MJOL MJOL MJOL MJOL

Slide 11

Slide 11 text

PullReqίϝϯτ΁ͷॻ͖ࠐΈ w֤ίϯςφׂ͕Γ౰ͯΒΕͨϙʔτΛऔಘɻ ! w઀ଓ৘ใΛ()FͷQVMMSFRίϝϯτʹॻ͘ɻ

Slide 12

Slide 12 text

Garbage Collection (daily) wQVMMSFRVFTU͕DMPTF͞ΕͨΒίϯςφΛࢭΊΔ࡟আ͢Δɻ
 wKFOLJOTͰQVMMSFRVFTUͷDMPTFIPPL͢Δ͜ͱ͕೉͔ͬͨͨ͠Ίɺ೔࣍Ͱ($ɻ DSPO 
 wEPDLFSͷίϯςφ໊ͷQSFpYΛQBSTFͯ͠ɺSFQPTJUPSZͱQVMMSFRVFTU൪߸Λऔಘɻ
 w౰֘QVMMSFRVFTU͕DMPTF͞Ε͍ͯΔ͔֬ೝ͠ɺด͍ͯͨ͡Βίϯςφఀࢭ࡟আɻ ! wίϯςφ໊ͷQSFpYͰ͸ͳ͘FUDE΍SFEJTʹରԠΛอ࣋ͯ͠΋ྑ͔͕ͬͨɺߏ੒ͷγϯϓ ϧ͞Λॏࢹɻ

Slide 13

Slide 13 text

developmentฤ Ή͢ͼ wطଘͷϓϩμΫτΛ%PDLFSԽ͢Δͷ͸ਏ͍৔߹͕͋ͬͨɻ w؀ڥߏஙΛDIFGͰߦ͏લఏͷϓϩμΫτΛޙ͔Β%PDLFSԽ͢Δͷ͸େมʹײͨ͡ɻ ! wطଘ෦෼Λ%PDLFSԽ͢ΔΑΓɺ৽نͰ࡞Δ΋ͷΛ%PDLFSԽલఏͰߏங͢Δํ͕ྑ͍ɻ w1VMM3FRVFTUͰʮͦͷ৔Ͱ࣮ࡍʹ৮ΕΔ؀ڥ͕طʹಈ͍͍ͯΔʯͷ͸࠷ߴɻ

Slide 14

Slide 14 text

Deployment with Docker

Slide 15

Slide 15 text

Personalize API /JDPOJDP "QQMJDBUJPO4FSWFS )551SFRVFTU 8FC 4FSWFS $BDIF %BUBCBTF 3FWFSTF 1SPYZ 1FSTPOBMJ[F"1* • ύʔιφϥΠζʹར༻͢ΔAPI (golang࣮૷) • DockerલఏͰ৽نʹߏஙɻ

Slide 16

Slide 16 text

Containerize! • ύʔιφϥΠζAPI • ϩάసૹ&ू໿(fluentd) • CEPΤϯδϯ(Norikra) • ϩάอଘ (ES&InfluxDB) • ՄࢹԽ (Kibana&Grafana) • ϦόʔεϓϩΩγ 1FSTPOBMJ[F"1* /JDPOJDP "QQMJDBUJPO4FSWFS )551SFRVFTU $POUBJOFSJ[F 8FC 4FSWFS $BDIF %BUBCBTF MPHTUPSF .FUSJDT7JTVBMJ[FS $&1 -PH$PMMFDUPS 3FWFSTF 1SPYZ

Slide 17

Slide 17 text

Dataflow DP 4FSWFS )551SFRVFTU $BDIF %BUBCBTF MPHTUPSF .FUSJDT7JTVBMJ[FS $&1 3FWFSTF 1SPYZ -PBE #BMBODFS -PH"HHSFHBUPS MPHpMF %PDLFS)PTU %PDLFS)PTU %PDLFS)PTU %PDLFS)PTU "1* -PH $PMMFDUPS w ֤IPTU಺ͷϦόʔεϓϩΩγ OHJOY ܦ༝Ͱ"1*ʹϦΫΤετɻ w ϩάϑΝΠϧ͸ϗετଆϘϦϡʔϜΛϚ΢ϯτͯ͠ॻ͖ग़͠ɻ w qVFOUEίϯςφଆͰ΋ϩάσΟϨΫτϦΛϚ΢ϯτͯ͠ɺಡΈࠐΈɻ w ϩάू໿ίϯςφͰϩάΛूΊͯɺ$&1 /PSJLSB Ͱूܭͭͭ͠ʹอଘˠՄࢹԽ

Slide 18

Slide 18 text

Docker Build Architecture MPDBM ()F +FOLJOT 3FHJTUSZ $PNNPO QVTITPVSDFT )PPL w %PDLFS*NBHFϏϧυ w ίϯςφͰܥΛ্ཱͪ͛ͯ݁߹ςετ w υΩϡϝϯτͷੜ੒ w %PDLFS3FHJTUSZʹΠϝʔδొ࿥ "1* qVFOUE /HJOY SFHJTUFS"1*DPOUBJOFS*NBHF VCVOUV CVJMEͷى఺ͱͳΔJNBHF͸ɺ1SJWBUF SFHJTUSZʹஔ͍͓͖ͯɺ%PDLFSIVC SFHJTUSZͷ౎౓ར༻͸͠ͳ͍ɻ

Slide 19

Slide 19 text

Deploy Manager WebUI • WebUIσϓϩΠπʔϧ • DockerԽҎલ͔Β࢖͍ͬͯͨ಺੡πʔϧ • Capistrano task ࣮ߦ • Ϣʔβೝূ • σϓϩΠϩάͷϦΞϧλΠϜදࣔ&อଘ
 • DockerσϓϩΠ΋capistrano taskԽͯ͠ར༻
 σϓϩΠઌαʔόΛબ୒ͯ͠ɺϙνοͱ
 1click͚ͩͰແఀࢭσϓϩΠɻ • σϓϩΠͷrollbackλεΫ΋༻ҙɻ

Slide 20

Slide 20 text

Docker Deployment Architecture %FQMPZ.BOBHFS 8FC6* 3FHJTUSZ ڞ௨ 3FHJTUSZ
 QSPEVDUJPOઃఆ "1* qVFOUE /HJOY "1*TFSWFSDPOpH 0QFSBUF QVMM qVFOUE 3FWFSTF1SPYZ "1* QVMM "1*TFSWFSDPOpH %PDLFS)PTU 0UIFS4FSWJDFT %FQMPZ $BQJTUSBOPUBTL )551SFRVFTU NPVOU • Capistrano taskͰɺίϯςφͷ
 ΠϝʔδΛ֤ϗετ͕pullͯ͠σϓϩΠɻ ! • ࣮ߦόΠφϦͷίϯςφΠϝʔδ͸։ൃऀ શһ͕pullՄೳɻ ! • production؀ڥͰ࢖͏configίϯςφ͸ɺ registry͝ͱ෼཭(ೝূ͕ҟͳΔ)ɻ

Slide 21

Slide 21 text

Deployment ~ step.1 1.APIίϯςφΠϝʔδͱɺઃఆΠϝʔδ ΛɺDockerHostଆͰpull͢Δ %PDLFS)PTU 3FHJTUSZ ڞ௨ 3FHJTUSZ ઃఆ "1* qVFOUE /HJOY "1*W EPDLFSJNBHFT "1*TFSWFSDPOpH "1*TFSWFSDPOpH "1*TFSWFS DPOpH NPVOU NPVOU 3FWFSTF1SPYZ "1*W qVFOUEMBUFTU

Slide 22

Slide 22 text

Deployment ~ step.1 1.APIίϯςφΠϝʔδͱɺઃఆΠϝʔδ ΛɺDockerHostଆͰpull͢Δ %PDLFS)PTU 3FHJTUSZ ڞ௨ 3FHJTUSZ ઃఆ "1* qVFOUE /HJOY EPDLFSQVMMBQJMBUFTU EPDLFSQVMMBQJ@DPOpHMBUFTU "1*W EPDLFSJNBHFT "1*W MBUFTU "1*TFSWFSDPOpH "1*TFSWFSDPOpH "1*TFSWFS DPOpH NPVOU NPVOU 3FWFSTF1SPYZ "1*W qVFOUEMBUFTU

Slide 23

Slide 23 text

Deployment ~ step.2 1.৽͍͠imageͷίϯςφΛىಈ 2.API͕ਖ਼͘͠ϦΫΤετΛฦͤΔ·Ͱ଴ͭ
 ɹ(࣮ࡍʹϨεϙϯεΛ֬ೝ) %PDLFS)PTU "1*W qVFOUEMBUFTU DPOpH 3FWFSTF1SPYZ

Slide 24

Slide 24 text

Deployment ~ step.2 1.৽͍͠imageͷίϯςφΛىಈ 2.API͕ਖ਼͘͠ϦΫΤετΛฦͤΔ·Ͱ଴ͭ
 ɹ(࣮ࡍʹϨεϙϯεΛ֬ೝ) %PDLFS)PTU "1*W "1*W qVFOUEMBUFTU qVFOUEMBUFTU DPOpH DPOpH 3FWFSTF1SPYZ

Slide 25

Slide 25 text

Deployment ~ step.2 1.৽͍͠imageͷίϯςφΛىಈ 2.API͕ਖ਼͘͠ϦΫΤετΛฦͤΔ·Ͱ଴ͭ
 ɹ(࣮ࡍʹϨεϙϯεΛ֬ೝ) %PDLFS)PTU "1*W "1*W qVFOUEMBUFTU qVFOUEMBUFTU DPOpH DPOpH IFBMUIDIFDL EVNNZSFRVFTU 3FWFSTF1SPYZ

Slide 26

Slide 26 text

Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$ kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS)PTU "1*W "1*W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF1SPYZ DPOpH DPOpH OHJOYDPOG OHJOY QPSU

Slide 27

Slide 27 text

Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$ kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS)PTU "1*W "1*W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF1SPYZ DPOpH DPOpH OHJOYDPOG MPDBUJPO\ QSPYZ@QBTTIUUQ ^ OHJOY QPSU

Slide 28

Slide 28 text

Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$ kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS)PTU "1*W "1*W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF1SPYZ DPOpH DPOpH OHJOYDPOG MPDBUJPO\ QSPYZ@QBTTIUUQ ^ MPDBUJPO\ QSPYZ@QBTTIUUQ ^ OHJOY QPSU

Slide 29

Slide 29 text

Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$ kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS)PTU "1*W "1*W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF1SPYZ DPOpH DPOpH OHJOYDPOG OHJOY QPSU

Slide 30

Slide 30 text

Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$ kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS)PTU "1*W "1*W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF1SPYZ DPOpH DPOpH OHJOYDPOG OHJOY OHJOY QPSU

Slide 31

Slide 31 text

Deployment ~ step.3 1.৽͍͠IPΛnginxͷconfʹॻ͖ࠐΉɻ 2.NginxΛgraceful restartͯ͠ɺແఀࢭͰίϯςφ ͷ޲͚ઌΛมߋ͢Δɻ
 Reverse Proxy(nginx)ͷDockerίϯςφ಺Ͱ 1.$ kill -USR2 `cat /var/run/nginx.pid` 2.$ kill -WINCH `cat /var/run/nginx.pid.oldbin` 3.$ kill -QUIT `cat /var/run/nginx.pid.oldbin`
 
 
 
 ※࣮ࡍʹ͸ɺgracefull restart෦෼͸
 γΣϧεΫϦϓτԽͯ͠ɺίϯςφ಺ʹಉࠝɻ %PDLFS)PTU "1*W "1*W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF1SPYZ DPOpH DPOpH OHJOYDPOG OHJOY QPSU

Slide 32

Slide 32 text

Deployment ~ step.4 1.ݹ͍ίϯςφͷఀࢭ&࡟আ 2.ݹ͍ίϯςφΠϝʔδͷ࡟আ
 
 ˞ߴ଎ʹrollbackग़དྷΔΑ͏ɺ࣮ࡍʹ͸
 ɹ5ੈ୅લͷΠϝʔδΛ࡟আ %PDLFS)PTU "1*W "1*W qVFOUEMBUFTU qVFOUEMBUFTU 3FWFSTF1SPYZ DPOpH DPOpH OHJOY "1*W EPDLFSJNBHFT "1*W MBUFTU "1*TFSWFSDPOpH

Slide 33

Slide 33 text

Deployment ~ step.4 1.ݹ͍ίϯςφͷఀࢭ&࡟আ 2.ݹ͍ίϯςφΠϝʔδͷ࡟আ
 
 ˞ߴ଎ʹrollbackग़དྷΔΑ͏ɺ࣮ࡍʹ͸
 ɹ5ੈ୅લͷΠϝʔδΛ࡟আ %PDLFS)PTU "1*W qVFOUEMBUFTU 3FWFSTF1SPYZ DPOpH OHJOY "1*W EPDLFSJNBHFT "1*W MBUFTU "1*TFSWFSDPOpH

Slide 34

Slide 34 text

Deployment ~ step.4 1.ݹ͍ίϯςφͷఀࢭ&࡟আ 2.ݹ͍ίϯςφΠϝʔδͷ࡟আ
 
 ˞ߴ଎ʹrollbackग़དྷΔΑ͏ɺ࣮ࡍʹ͸
 ɹ5ੈ୅લͷΠϝʔδΛ࡟আ %PDLFS)PTU "1*W qVFOUEMBUFTU 3FWFSTF1SPYZ DPOpH OHJOY EPDLFSJNBHFT "1*W MBUFTU "1*TFSWFSDPOpH

Slide 35

Slide 35 text

deploymentฤ Ή͢ͼ wμ΢ϯλΠϜແ͠ͷσϓϩΠ͸Մೳ͕ͩɺݱঢ়ɺࣗલͰগ͠࡞Γ͜Ήඞཁ͕͋Δɻ wࣗલͷίϯςφ੾ସલϔϧενΣοΫɻ ! wϗετଆͷ؀ڥ͕ԚΕͳ͍ͷ͸ૉ੖Β͍͠ɻ ! w௕ظӡ༻͢ΔલఏͳΒΰϛΛย෇͚ΔػߏΛ࣋ͱ͏ɻ
 wLTΛ࢖͏ʹ͸ɺNBTUFSͷ৑௕Խ΍ߴՄ༻ੑͳFUDEΫϥελͷߏஙɾӡ༻౳ɺ
 ن໛ͷׂʹ߹Θͳ͔ͬͨͷͰʮϦιʔεͷந৅Խʯ·Ͱ͸౿Έࠐ·ͳͳ͔ͬͨɻ
 wαʔϏεԣஅͰར༻Մೳͳେن໛σϓϩΠઌαʔόࢿݯͱͯ͠ந৅Խ͢Δ৔߹ʹ͸ɺ
 LT౳͕ඞཁʹͳΔͱࢥ͏ɻن໛ͱ໨తʹԠͯ͡ΞʔΩςΫνϟઃܭ͠Α͏ɻ

Slide 36

Slide 36 text

υϫϯΰ޿ࠂ 
 ͜ͷهࣄ͸υϫϯΰۈ຿࣌ؒதʹॻ͔Εͨɻ ! υϫϯΰ͸ຊ෺ͷ
 (C++|Elrang|ݕࡧ|Ϩίϝϯυ|σʔλ෼ੳ|σʔλ࿈ܞ|FPGA|Πϯϑϥ|WEBΞϓϦ)
 ΤϯδχΞΛืू͍ͯ͠·͢ɻ ࠾༻৘ใ | גࣜձࣾυϫϯΰ