Slide 1

Slide 1 text

TensorFlow.jsͰ όʔνϟϧഎܠΛ࡞Δ ּݪ ޺ (@kasacchiful) 2020/08/08 (౔) TensorFlow UG Niigata #3

Slide 2

Slide 2 text

Software Developer Favorite: Community: • JAWS-UG Niigata • JaSST Niigata • ASTER • SWANII • etc. Hiroshi Kasahara @kasacchiful @kasacchiful 2

Slide 3

Slide 3 text

࠷ۙ͸ϦϞʔτϫʔΫશ੝ • ձٞதͷഎܠΛผͷࣸਅʹͯ͠͠·͏ʮόʔνϟϧഎܠʯΛઃఆͰ͖ Δπʔϧ͕૿͍͑ͯΔ • ͜ΕɺTensorFlow.jsͰ࣮૷Ͱ͖ΔΜ͡Όͳ͍ʁ • ͔֬ɺਓͷηάϝϯςʔγϣϯͰ͖ΔϞσϧެ։͞Ε͍ͯΔ͸ͣ 3

Slide 4

Slide 4 text

ຊ೔ͷ಺༰ • TensorFlow.js Λ࢖ͬͯɺόʔνϟϧഎܠΛ࣮૷ͯ͠Έͨ 4

Slide 5

Slide 5 text

2೔ ੍࡞ظؒ 5

Slide 6

Slide 6 text

BodyPix Webϒϥ΢βͱTensorFlow.jsΛ ࢖ͬͨϦΞϧλΠϜͷਓ෺ηάϝϯ ςʔγϣϯϞσϧ https://github.com/tensorflow/tfjs-models/tree/master/body-pix Ҿ༻: https://developers-jp.googleblog.com/2019/04/bodypix-tensorflowjs.html 6

Slide 7

Slide 7 text

ॲཧͷྲྀΕ • WebΧϝϥͷಈըΛऔಘ • BodyPixͰਓͷηάϝϯςʔγϣϯΛ࣮ࢪ • ਓҎ֎ͷ෦෼ʢഎܠʣΛଞͷը૾ʹஔ͖׵͑Δ • ʢ͓·͚ʣOBS Virtual CameraͰ֤छWebձٞπʔϧʹ౤Ө͢Δ ιʔείʔυ͸ɺBodyPixͷσϞιʔεΛࢀߟʹվมͨ͠ 7

Slide 8

Slide 8 text

ߏ੒ਤ 8 ηάϝϯςʔγϣϯ ݁Ռ Χϝϥը૾ͱ എܠը૾Λ ߹੒ 8FCΧϝϥ ਓ͕өͬͨ Χϝϥը૾ എܠը૾ ϒϥ΢βදࣔ 0#47JSUVBM$BN ֤छπʔϧʹදࣔ

Slide 9

Slide 9 text

σϞ 9

Slide 10

Slide 10 text

࣮૷ͨ͠΋ͷ • σϞαΠτ https://tfug-niigata-bodypix-demo.netlify.app/ • ιʔείʔυ https://github.com/kasacchiful/bodypix-sample 10

Slide 11

Slide 11 text

WebΧϝϥͷಈ ըΛऔಘ
• videoλάΛඇදࣔʹͯ͠Χϝϥ ىಈ&ಈը࠶ੜ • canvasλάʹදࣔ͢Δ͜ͱͰɺ ͍ΖΜͳը૾Λඳ͚Δ • എܠը૾͸imgλάʹ࢓ࠐ·ͤͯ ͓͘ 11

Slide 12

Slide 12 text

ਓͷηάϝϯ ςʔγϣϯ async function estimateSegmentation() { return await state.net.segmentPerson(state.video, { internalResolution: segmentationOption.internalResolution, segmentationThreshold: segmentationOption.segmentationThreshold, maxDetections: segmentationOption.multiDecodingMaxDetections, scoreThreshold: segmentationOption.multiDecodingScoreThreshold, nmsRadius: segmentationOption.multiDecodingNmsRadius, }); } function segmentBodyInRealTime() { const canvas = document.getElementById('output'); const img = document.getElementById('bg_img'); async function bodySegmentationFrame() { // segmentation const personSegmentation = await estimateSegmentation(); toMaskImage(canvas, personSegmentation, state.video, img); requestAnimationFrame(bodySegmentationFrame); } bodySegmentationFrame(); } • segmentPerson()Ͱηάϝ ϯςʔγϣϯऔಘ • ֤ϐΫηϧʹɺਓͳΒ”1” ਓҎ֎ͳΒ”0”͕෇༩͞Ε Δ 12

Slide 13

Slide 13 text

Χϝϥը૾ͱഎ ܠը૾ͷ߹੒ function toMaskImage(canvas, segmentation, video, img) { // লུ… // ඳը const ctx = canvas.getContext('2d'); let ctxImageData = ctx.getImageData(0, 0, canvas.width, canvas.height); let bytes = ctxImageData.data; for (let i = 0; i < canvas.height; i++) { for (let j = 0; j < canvas.width; j++) { const n = i * canvas.width + j; if (segmentation.data[n] === 1) { // for foreground (ਓ) bytes[4 * n + 0] = fgImg.data[4 * n + 0]; bytes[4 * n + 1] = fgImg.data[4 * n + 1]; bytes[4 * n + 2] = fgImg.data[4 * n + 2]; bytes[4 * n + 3] = fgImg.data[4 * n + 3]; } else { // for background (എܠ) bytes[4 * n + 0] = bgImg.data[4 * n + 0]; bytes[4 * n + 1] = bgImg.data[4 * n + 1]; bytes[4 * n + 2] = bgImg.data[4 * n + 2]; bytes[4 * n + 3] = bgImg.data[4 * n + 3]; } } } ctx.putImageData(ctxImageData, 0, 0); } • ֤ϐΫηϧͷηάϝϯςʔγϣ ϯ൑ఆʹԠͯ͡ɺΧϝϥը૾͔ എܠը૾Λcanvasʹॻ͖ࠐΉ • BodyPixͷtoMask()Ͱ΋ಉ͡Α ͏ͳख๏Ͱߦ͍ͬͯͨͷͰ࠾༻ 13 IUUQTHJUIVCDPNUFOTPSqPXUGKTNPEFMTCMPCNBTUFSCPEZQJYTSDPVUQVU@SFOEFSJOH@VUJMUT--

Slide 14

Slide 14 text

OBS VirtualCam • OBS͸ಈը഑৴ͷࡍʹ֤छฤू͕Ͱ͖Δɺແྉͷπʔϧ • OBS VirtualCam͸OBSͷϓϥάΠϯͱͯ͠ΠϯετʔϧͰ͖Δ • ࠓճ͸ϒϥ΢βΛೖྗιʔεͱͯ͠OBSʹऔΓࠐΈɺදࣔαΠζΛద ٓमਖ਼ͯ͠ɺOBS VirtualCamͷԾ૝Χϝϥʹೖྗιʔεͱͯ͠ઃఆ • OBS VirtualCam͸Χϝϥͷ1ͭͱͯ͠ೝࣝ͞ΕΔͷͰɺZoom΍ Google Meet౳ͷೖྗΧϝϥͱͯ͠ઃఆͰ͖Δ 14

Slide 15

Slide 15 text

՝୊ • ը૾ͷ੾Γସ͑ػೳͳ͠ ʗ Χϝϥͷ੾Γସ͑ػೳͳ͠ • ࠨӈ൓సͰ͖ͳ͍ • canvasΛࠨӈ൓సͤ͞Ε͹ɺഎܠ͝ͱ൓సͰ͖ͦ͏ • CPUύϫʔ͕ඞཁ • ಛʹOBS VirtualCamΛซ༻͢ΔͱɺCPUϑΝϯ͕ᄬΔ • OBS VirtualCamͷઃఆ͕໘౗ • Chrome Extension౳Ͱ࡞ͬͯ΋͍͍͔΋ 15

Slide 16

Slide 16 text

·ͱΊ • TensorFlow.js ͱ BodyPix ϞσϧͰ؆୯ʹ࣮૷Ͱ͖ͨ • PoseNetͱ૊Έ߹ΘͤΔͱɺΑΓ໘ന͍͔΋ • ՆٳΈͷࣗ༝ݚڀωλͱͯ͠΋ɺ͓࢖͍͍ͩ͘͞ 16

Slide 17

Slide 17 text

એ఻ 17

Slide 18

Slide 18 text

JAWS SONIC 2020 & MIDNIGHT JAWS 2020 JAWS-UGͷ24hΦϯϥΠϯΠϕϯτ 9/12(౔) 16:50 - 9/13(೔) 17:20 JAWS-UG৽ׁ࿮Ͱ஻Δ͔΋ IUUQTKBXTTPOJDKBXTVHKQ 18

Slide 19

Slide 19 text

JaSST’20 Niigata ιϑτ΢ΣΞςετγϯϙδ΢Ϝ ΦϯϥΠϯ։࠵ 9/28(݄) 13:00 - 17:00 (༧ఆ) ςʔϚ: ςετࣗಈԽ IUUQKBTTUKQTZNQPTJVNKBTTUOJJHBUBIUNM 19

Slide 20

Slide 20 text

͓ΘΓ 20

Slide 21

Slide 21 text

࣮૷ͨ͠΋ͷʢ࠶ܝʣ • σϞαΠτ https://tfug-niigata-bodypix-demo.netlify.app/ • ιʔείʔυ https://github.com/kasacchiful/bodypix-sample 21

Slide 22

Slide 22 text

ࢀߟ • https://developers-jp.googleblog.com/2019/04/bodypix- tensorflowjs.html • https://github.com/tensorflow/tfjs-models/tree/master/body-pix • https://note.com/prty/n/ne2d2090148dc 22