Upgrade to Pro — share decks privately, control downloads, hide ads and more …

TensorFlow.js で バーチャル背景を作る / virtual background...

TensorFlow.js で バーチャル背景を作る / virtual background with tfjs

2020/08/08 (土) に TensorFlow UG Niigata #3 にて発表した資料です。
TensorFlow.js を使ってバーチャル背景を作ってみました。

kasacchiful

August 08, 2020
Tweet

More Decks by kasacchiful

Other Decks in Programming

Transcript

  1. Software Developer Favorite: Community: • JAWS-UG Niigata • JaSST Niigata

    • ASTER • SWANII • etc. Hiroshi Kasahara @kasacchiful @kasacchiful 2
  2. WebΧϝϥͷಈ ըΛऔಘ <div id='main' style='display:none'> <video id="video" playsinline style="display: none;"></video>

    <canvas id="output"></canvas> <img id="bg_img" src="./bg.jpg" style="display: none;"> </div> • videoλάΛඇදࣔʹͯ͠Χϝϥ ىಈ&ಈը࠶ੜ • canvasλάʹදࣔ͢Δ͜ͱͰɺ ͍ΖΜͳը૾Λඳ͚Δ • എܠը૾͸imgλάʹ࢓ࠐ·ͤͯ ͓͘ 11
  3. ਓͷηάϝϯ ςʔγϣϯ async function estimateSegmentation() { return await state.net.segmentPerson(state.video, {

    internalResolution: segmentationOption.internalResolution, segmentationThreshold: segmentationOption.segmentationThreshold, maxDetections: segmentationOption.multiDecodingMaxDetections, scoreThreshold: segmentationOption.multiDecodingScoreThreshold, nmsRadius: segmentationOption.multiDecodingNmsRadius, }); } function segmentBodyInRealTime() { const canvas = document.getElementById('output'); const img = document.getElementById('bg_img'); async function bodySegmentationFrame() { // segmentation const personSegmentation = await estimateSegmentation(); toMaskImage(canvas, personSegmentation, state.video, img); requestAnimationFrame(bodySegmentationFrame); } bodySegmentationFrame(); } • segmentPerson()Ͱηάϝ ϯςʔγϣϯऔಘ • ֤ϐΫηϧʹɺਓͳΒ”1” ਓҎ֎ͳΒ”0”͕෇༩͞Ε Δ 12
  4. Χϝϥը૾ͱഎ ܠը૾ͷ߹੒ function toMaskImage(canvas, segmentation, video, img) { // লུ…

    // ඳը const ctx = canvas.getContext('2d'); let ctxImageData = ctx.getImageData(0, 0, canvas.width, canvas.height); let bytes = ctxImageData.data; for (let i = 0; i < canvas.height; i++) { for (let j = 0; j < canvas.width; j++) { const n = i * canvas.width + j; if (segmentation.data[n] === 1) { // for foreground (ਓ) bytes[4 * n + 0] = fgImg.data[4 * n + 0]; bytes[4 * n + 1] = fgImg.data[4 * n + 1]; bytes[4 * n + 2] = fgImg.data[4 * n + 2]; bytes[4 * n + 3] = fgImg.data[4 * n + 3]; } else { // for background (എܠ) bytes[4 * n + 0] = bgImg.data[4 * n + 0]; bytes[4 * n + 1] = bgImg.data[4 * n + 1]; bytes[4 * n + 2] = bgImg.data[4 * n + 2]; bytes[4 * n + 3] = bgImg.data[4 * n + 3]; } } } ctx.putImageData(ctxImageData, 0, 0); } • ֤ϐΫηϧͷηάϝϯςʔγϣ ϯ൑ఆʹԠͯ͡ɺΧϝϥը૾͔ എܠը૾Λcanvasʹॻ͖ࠐΉ • BodyPixͷtoMask()Ͱ΋ಉ͡Α ͏ͳख๏Ͱߦ͍ͬͯͨͷͰ࠾༻ 13 IUUQTHJUIVCDPNUFOTPSqPXUGKTNPEFMTCMPCNBTUFSCPEZQJYTSDPVUQVU@SFOEFSJOH@VUJMUT--
  5. OBS VirtualCam • OBS͸ಈը഑৴ͷࡍʹ֤छฤू͕Ͱ͖Δɺແྉͷπʔϧ • OBS VirtualCam͸OBSͷϓϥάΠϯͱͯ͠ΠϯετʔϧͰ͖Δ • ࠓճ͸ϒϥ΢βΛೖྗιʔεͱͯ͠OBSʹऔΓࠐΈɺදࣔαΠζΛద ٓमਖ਼ͯ͠ɺOBS

    VirtualCamͷԾ૝Χϝϥʹೖྗιʔεͱͯ͠ઃఆ • OBS VirtualCam͸Χϝϥͷ1ͭͱͯ͠ೝࣝ͞ΕΔͷͰɺZoom΍ Google Meet౳ͷೖྗΧϝϥͱͯ͠ઃఆͰ͖Δ 14
  6. ՝୊ • ը૾ͷ੾Γସ͑ػೳͳ͠ ʗ Χϝϥͷ੾Γସ͑ػೳͳ͠ • ࠨӈ൓సͰ͖ͳ͍ • canvasΛࠨӈ൓సͤ͞Ε͹ɺഎܠ͝ͱ൓సͰ͖ͦ͏ •

    CPUύϫʔ͕ඞཁ • ಛʹOBS VirtualCamΛซ༻͢ΔͱɺCPUϑΝϯ͕ᄬΔ • OBS VirtualCamͷઃఆ͕໘౗ • Chrome Extension౳Ͱ࡞ͬͯ΋͍͍͔΋ 15
  7. JAWS SONIC 2020 & MIDNIGHT JAWS 2020 JAWS-UGͷ24hΦϯϥΠϯΠϕϯτ 9/12(౔) 16:50

    - 9/13(೔) 17:20 JAWS-UG৽ׁ࿮Ͱ஻Δ͔΋ IUUQTKBXTTPOJDKBXTVHKQ 18