Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
TensorFlow.js で バーチャル背景を作る / virtual background...
Search
kasacchiful
August 08, 2020
Programming
0
1.3k
TensorFlow.js で バーチャル背景を作る / virtual background with tfjs
2020/08/08 (土) に TensorFlow UG Niigata #3 にて発表した資料です。
TensorFlow.js を使ってバーチャル背景を作ってみました。
kasacchiful
August 08, 2020
Tweet
Share
More Decks by kasacchiful
See All by kasacchiful
ワイがおすすめする新潟の食 / 20250530phpconf-niigata-eve
kasacchiful
0
290
生成AIでメタデータを生成してみた / 20250525generate-metadata-using-generative-ai
kasacchiful
0
52
Strands Agents SDK で AIエージェント作成 を試してみた / 20250525strands-agents
kasacchiful
0
88
いろんな世界を見てみよう / 20250508ninno_tech_fest
kasacchiful
0
28
Amazon Q Developer for CLIのある生活 / 20250427ai_craft_hacks_niigata1
kasacchiful
1
74
AWSのコンテナサービス / jawsug-akita-aws-container-services
kasacchiful
0
67
データ基盤でのコンテナ活用事例 / jawsug-akita-data-platform-with-container
kasacchiful
0
73
データ基盤でのコンテナ活用事例 / jawsug-niigata21-data-platform-with-container
kasacchiful
0
110
AWSのコンテナサービス / jawsug_niigata21_aws_container_services
kasacchiful
1
98
Other Decks in Programming
See All in Programming
人には人それぞれのサービス層がある
shimabox
3
650
実はすごいスピードで進化しているCSS
hayato_yokoyama
0
110
Parallel::Pipesの紹介
skaji
2
900
型付きアクターモデルがもたらす分散シミュレーションの未来
piyo7
0
730
がんばりすぎないコーディングルール運用術
tsukakei
1
210
Effect の双対、Coeffect
yukikurage
4
1.3k
「兵法」から見る質とスピード
ickx
0
250
Beyond Portability: Live Migration for Evolving WebAssembly Workloads
chikuwait
0
300
プロダクト開発でも使おう 関数のオーバーロード
yoiwamoto
0
140
型安全RESTで爆速プロトタイピング – Hono RPC実践
tacke_jp
0
110
Haskell でアルゴリズムを抽象化する / 関数型言語で競技プログラミング
naoya
16
3.6k
iOSアプリ開発もLLMで自動運転する
hiragram
6
2.3k
Featured
See All Featured
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Code Reviewing Like a Champion
maltzj
524
40k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
We Have a Design System, Now What?
morganepeng
52
7.6k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Faster Mobile Websites
deanohume
307
31k
Build The Right Thing And Hit Your Dates
maggiecrowley
36
2.7k
Why You Should Never Use an ORM
jnunemaker
PRO
56
9.4k
Producing Creativity
orderedlist
PRO
346
40k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.9k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
Become a Pro
speakerdeck
PRO
28
5.4k
Transcript
TensorFlow.jsͰ όʔνϟϧഎܠΛ࡞Δ ּݪ (@kasacchiful) 2020/08/08 () TensorFlow UG Niigata
#3
Software Developer Favorite: Community: • JAWS-UG Niigata • JaSST Niigata
• ASTER • SWANII • etc. Hiroshi Kasahara @kasacchiful @kasacchiful 2
࠷ۙϦϞʔτϫʔΫશ • ձٞதͷഎܠΛผͷࣸਅʹͯ͠͠·͏ʮόʔνϟϧഎܠʯΛઃఆͰ͖ Δπʔϧ͕૿͍͑ͯΔ • ͜ΕɺTensorFlow.jsͰ࣮Ͱ͖ΔΜ͡Όͳ͍ʁ • ͔֬ɺਓͷηάϝϯςʔγϣϯͰ͖ΔϞσϧެ։͞Ε͍ͯΔͣ 3
ຊͷ༰ • TensorFlow.js ΛͬͯɺόʔνϟϧഎܠΛ࣮ͯ͠Έͨ 4
2 ੍࡞ظؒ 5
BodyPix WebϒϥβͱTensorFlow.jsΛ ͬͨϦΞϧλΠϜͷਓηάϝϯ ςʔγϣϯϞσϧ https://github.com/tensorflow/tfjs-models/tree/master/body-pix Ҿ༻: https://developers-jp.googleblog.com/2019/04/bodypix-tensorflowjs.html 6
ॲཧͷྲྀΕ • WebΧϝϥͷಈըΛऔಘ • BodyPixͰਓͷηάϝϯςʔγϣϯΛ࣮ࢪ • ਓҎ֎ͷ෦ʢഎܠʣΛଞͷը૾ʹஔ͖͑Δ • ʢ͓·͚ʣOBS Virtual
CameraͰ֤छWebձٞπʔϧʹӨ͢Δ ιʔείʔυɺBodyPixͷσϞιʔεΛࢀߟʹվมͨ͠ 7
ߏਤ 8 ηάϝϯςʔγϣϯ ݁Ռ Χϝϥը૾ͱ എܠը૾Λ ߹ 8FCΧϝϥ ਓ͕өͬͨ Χϝϥը૾
എܠը૾ ϒϥβදࣔ 0#47JSUVBM$BN ֤छπʔϧʹදࣔ
σϞ 9
࣮ͨ͠ͷ • σϞαΠτ https://tfug-niigata-bodypix-demo.netlify.app/ • ιʔείʔυ https://github.com/kasacchiful/bodypix-sample 10
WebΧϝϥͷಈ ըΛऔಘ <div id='main' style='display:none'> <video id="video" playsinline style="display: none;"></video>
<canvas id="output"></canvas> <img id="bg_img" src="./bg.jpg" style="display: none;"> </div> • videoλάΛඇදࣔʹͯ͠Χϝϥ ىಈ&ಈը࠶ੜ • canvasλάʹදࣔ͢Δ͜ͱͰɺ ͍ΖΜͳը૾Λඳ͚Δ • എܠը૾imgλάʹࠐ·ͤͯ ͓͘ 11
ਓͷηάϝϯ ςʔγϣϯ async function estimateSegmentation() { return await state.net.segmentPerson(state.video, {
internalResolution: segmentationOption.internalResolution, segmentationThreshold: segmentationOption.segmentationThreshold, maxDetections: segmentationOption.multiDecodingMaxDetections, scoreThreshold: segmentationOption.multiDecodingScoreThreshold, nmsRadius: segmentationOption.multiDecodingNmsRadius, }); } function segmentBodyInRealTime() { const canvas = document.getElementById('output'); const img = document.getElementById('bg_img'); async function bodySegmentationFrame() { // segmentation const personSegmentation = await estimateSegmentation(); toMaskImage(canvas, personSegmentation, state.video, img); requestAnimationFrame(bodySegmentationFrame); } bodySegmentationFrame(); } • segmentPerson()Ͱηάϝ ϯςʔγϣϯऔಘ • ֤ϐΫηϧʹɺਓͳΒ”1” ਓҎ֎ͳΒ”0”͕༩͞Ε Δ 12
Χϝϥը૾ͱഎ ܠը૾ͷ߹ function toMaskImage(canvas, segmentation, video, img) { // লུ…
// ඳը const ctx = canvas.getContext('2d'); let ctxImageData = ctx.getImageData(0, 0, canvas.width, canvas.height); let bytes = ctxImageData.data; for (let i = 0; i < canvas.height; i++) { for (let j = 0; j < canvas.width; j++) { const n = i * canvas.width + j; if (segmentation.data[n] === 1) { // for foreground (ਓ) bytes[4 * n + 0] = fgImg.data[4 * n + 0]; bytes[4 * n + 1] = fgImg.data[4 * n + 1]; bytes[4 * n + 2] = fgImg.data[4 * n + 2]; bytes[4 * n + 3] = fgImg.data[4 * n + 3]; } else { // for background (എܠ) bytes[4 * n + 0] = bgImg.data[4 * n + 0]; bytes[4 * n + 1] = bgImg.data[4 * n + 1]; bytes[4 * n + 2] = bgImg.data[4 * n + 2]; bytes[4 * n + 3] = bgImg.data[4 * n + 3]; } } } ctx.putImageData(ctxImageData, 0, 0); } • ֤ϐΫηϧͷηάϝϯςʔγϣ ϯఆʹԠͯ͡ɺΧϝϥը૾͔ എܠը૾Λcanvasʹॻ͖ࠐΉ • BodyPixͷtoMask()Ͱಉ͡Α ͏ͳख๏Ͱߦ͍ͬͯͨͷͰ࠾༻ 13 IUUQTHJUIVCDPNUFOTPSqPXUGKTNPEFMTCMPCNBTUFSCPEZQJYTSDPVUQVU@SFOEFSJOH@VUJMUT--
OBS VirtualCam • OBSಈը৴ͷࡍʹ֤छฤू͕Ͱ͖Δɺແྉͷπʔϧ • OBS VirtualCamOBSͷϓϥάΠϯͱͯ͠ΠϯετʔϧͰ͖Δ • ࠓճϒϥβΛೖྗιʔεͱͯ͠OBSʹऔΓࠐΈɺදࣔαΠζΛద ٓमਖ਼ͯ͠ɺOBS
VirtualCamͷԾΧϝϥʹೖྗιʔεͱͯ͠ઃఆ • OBS VirtualCamΧϝϥͷ1ͭͱͯ͠ೝࣝ͞ΕΔͷͰɺZoom Google MeetͷೖྗΧϝϥͱͯ͠ઃఆͰ͖Δ 14
՝ • ը૾ͷΓସ͑ػೳͳ͠ ʗ ΧϝϥͷΓସ͑ػೳͳ͠ • ࠨӈసͰ͖ͳ͍ • canvasΛࠨӈసͤ͞Εɺഎܠ͝ͱసͰ͖ͦ͏ •
CPUύϫʔ͕ඞཁ • ಛʹOBS VirtualCamΛซ༻͢ΔͱɺCPUϑΝϯ͕ᄬΔ • OBS VirtualCamͷઃఆ͕໘ • Chrome ExtensionͰ࡞͍͍͔ͬͯ 15
·ͱΊ • TensorFlow.js ͱ BodyPix ϞσϧͰ؆୯ʹ࣮Ͱ͖ͨ • PoseNetͱΈ߹ΘͤΔͱɺΑΓ໘ന͍͔ • ՆٳΈͷࣗ༝ݚڀωλͱͯ͠ɺ͓͍͍ͩ͘͞
16
એ 17
JAWS SONIC 2020 & MIDNIGHT JAWS 2020 JAWS-UGͷ24hΦϯϥΠϯΠϕϯτ 9/12() 16:50
- 9/13() 17:20 JAWS-UG৽ׁͰΔ͔ IUUQTKBXTTPOJDKBXTVHKQ 18
JaSST’20 Niigata ιϑτΣΞςετγϯϙδϜ ΦϯϥΠϯ։࠵ 9/28(݄) 13:00 - 17:00 (༧ఆ) ςʔϚ:
ςετࣗಈԽ IUUQKBTTUKQTZNQPTJVNKBTTUOJJHBUBIUNM 19
͓ΘΓ 20
࣮ͨ͠ͷʢ࠶ܝʣ • σϞαΠτ https://tfug-niigata-bodypix-demo.netlify.app/ • ιʔείʔυ https://github.com/kasacchiful/bodypix-sample 21
ࢀߟ • https://developers-jp.googleblog.com/2019/04/bodypix- tensorflowjs.html • https://github.com/tensorflow/tfjs-models/tree/master/body-pix • https://note.com/prty/n/ne2d2090148dc 22