Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
TensorFlow.js で バーチャル背景を作る / virtual background with tfjs
Search
kasacchiful
August 08, 2020
Programming
0
1.2k
TensorFlow.js で バーチャル背景を作る / virtual background with tfjs
2020/08/08 (土) に TensorFlow UG Niigata #3 にて発表した資料です。
TensorFlow.js を使ってバーチャル背景を作ってみました。
kasacchiful
August 08, 2020
Tweet
Share
More Decks by kasacchiful
See All by kasacchiful
AWS IoT 1-clickがサービス終了するので、SORACOMに移行した話 / 20240518-jawsug-niigata-iotlt-niigata
kasacchiful
0
110
AWS Application Composerで始める、 サーバーレスなデータ基盤構築 / 20240406-jawsug-hokuriku-shinkansen
kasacchiful
1
400
AWSの各種サービス紹介と活用方法 − AI・ML活用デモを交えて − / 20231208aws-aiml-seminar
kasacchiful
0
440
Amazon Rekognition デモ / 20231208-aws_seminar-01-rekognition-demo
kasacchiful
0
450
Amazon Lookout for Vision デモ / 20231208-aws_seminar-02-lookout-vision-demo
kasacchiful
0
440
Python機械学習勉強会in新潟のロゴが無いので、生成AIで作ってみましょう / osc2023niigata
kasacchiful
0
380
Amazon Bedrock概要と生成AIの基礎 / 20231118-jawsug-niigata-15
kasacchiful
0
93
生成AIと自然言語処理の基礎 / 20231111-pyml-niigata-18.pdf
kasacchiful
0
180
最近やってる、サーバーレスでデータ分析基盤を構築している話 / 20230916-nds63
kasacchiful
0
88
Other Decks in Programming
See All in Programming
CSC307 Lecture 10
javiergs
PRO
0
310
最古の関数型言語「Lisp」ことはじめ / lisp_in_kamiyama
uhooi
1
190
MIERUNE BBQにおけるユーザー中心設計()
mierune
PRO
1
110
Trial
cairolibrary720
1
130
CSC307 Lecture 09
javiergs
PRO
1
500
はしめてのプログラミングとロボット制御
watawatavoltage
0
290
OpenAI/Gemini APIを使って EPUBを翻訳するCLIツールをつくってみた
tomiyan
0
790
CSC307 Lecture 14
javiergs
PRO
0
220
Folding Cheat Sheet #7
philipschwarz
PRO
0
150
CSC307 Lecture 08
javiergs
PRO
0
330
データカタログ運用物語 〜令和6年夏の理想と現実〜
kuro_kurorrr
0
110
Rubyのパフォーマンスプロファイリングの改善 / Enhancing performance profiling for Ruby
osyoyu
1
410
Featured
See All Featured
Music & Morning Musume
bryan
43
5.9k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
13
430
Statistics for Hackers
jakevdp
792
220k
Robots, Beer and Maslow
schacon
PRO
157
8.1k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k
What's new in Ruby 2.0
geeforr
338
31k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
662
120k
Bootstrapping a Software Product
garrettdimon
PRO
304
110k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
277
13k
Designing for humans not robots
tammielis
247
25k
Building a Modern Day E-commerce SEO Strategy
aleyda
25
6.7k
Unsuck your backbone
ammeep
666
57k
Transcript
TensorFlow.jsͰ όʔνϟϧഎܠΛ࡞Δ ּݪ (@kasacchiful) 2020/08/08 () TensorFlow UG Niigata
#3
Software Developer Favorite: Community: • JAWS-UG Niigata • JaSST Niigata
• ASTER • SWANII • etc. Hiroshi Kasahara @kasacchiful @kasacchiful 2
࠷ۙϦϞʔτϫʔΫશ • ձٞதͷഎܠΛผͷࣸਅʹͯ͠͠·͏ʮόʔνϟϧഎܠʯΛઃఆͰ͖ Δπʔϧ͕૿͍͑ͯΔ • ͜ΕɺTensorFlow.jsͰ࣮Ͱ͖ΔΜ͡Όͳ͍ʁ • ͔֬ɺਓͷηάϝϯςʔγϣϯͰ͖ΔϞσϧެ։͞Ε͍ͯΔͣ 3
ຊͷ༰ • TensorFlow.js ΛͬͯɺόʔνϟϧഎܠΛ࣮ͯ͠Έͨ 4
2 ੍࡞ظؒ 5
BodyPix WebϒϥβͱTensorFlow.jsΛ ͬͨϦΞϧλΠϜͷਓηάϝϯ ςʔγϣϯϞσϧ https://github.com/tensorflow/tfjs-models/tree/master/body-pix Ҿ༻: https://developers-jp.googleblog.com/2019/04/bodypix-tensorflowjs.html 6
ॲཧͷྲྀΕ • WebΧϝϥͷಈըΛऔಘ • BodyPixͰਓͷηάϝϯςʔγϣϯΛ࣮ࢪ • ਓҎ֎ͷ෦ʢഎܠʣΛଞͷը૾ʹஔ͖͑Δ • ʢ͓·͚ʣOBS Virtual
CameraͰ֤छWebձٞπʔϧʹӨ͢Δ ιʔείʔυɺBodyPixͷσϞιʔεΛࢀߟʹվมͨ͠ 7
ߏਤ 8 ηάϝϯςʔγϣϯ ݁Ռ Χϝϥը૾ͱ എܠը૾Λ ߹ 8FCΧϝϥ ਓ͕өͬͨ Χϝϥը૾
എܠը૾ ϒϥβදࣔ 0#47JSUVBM$BN ֤छπʔϧʹදࣔ
σϞ 9
࣮ͨ͠ͷ • σϞαΠτ https://tfug-niigata-bodypix-demo.netlify.app/ • ιʔείʔυ https://github.com/kasacchiful/bodypix-sample 10
WebΧϝϥͷಈ ըΛऔಘ <div id='main' style='display:none'> <video id="video" playsinline style="display: none;"></video>
<canvas id="output"></canvas> <img id="bg_img" src="./bg.jpg" style="display: none;"> </div> • videoλάΛඇදࣔʹͯ͠Χϝϥ ىಈ&ಈը࠶ੜ • canvasλάʹදࣔ͢Δ͜ͱͰɺ ͍ΖΜͳը૾Λඳ͚Δ • എܠը૾imgλάʹࠐ·ͤͯ ͓͘ 11
ਓͷηάϝϯ ςʔγϣϯ async function estimateSegmentation() { return await state.net.segmentPerson(state.video, {
internalResolution: segmentationOption.internalResolution, segmentationThreshold: segmentationOption.segmentationThreshold, maxDetections: segmentationOption.multiDecodingMaxDetections, scoreThreshold: segmentationOption.multiDecodingScoreThreshold, nmsRadius: segmentationOption.multiDecodingNmsRadius, }); } function segmentBodyInRealTime() { const canvas = document.getElementById('output'); const img = document.getElementById('bg_img'); async function bodySegmentationFrame() { // segmentation const personSegmentation = await estimateSegmentation(); toMaskImage(canvas, personSegmentation, state.video, img); requestAnimationFrame(bodySegmentationFrame); } bodySegmentationFrame(); } • segmentPerson()Ͱηάϝ ϯςʔγϣϯऔಘ • ֤ϐΫηϧʹɺਓͳΒ”1” ਓҎ֎ͳΒ”0”͕༩͞Ε Δ 12
Χϝϥը૾ͱഎ ܠը૾ͷ߹ function toMaskImage(canvas, segmentation, video, img) { // লུ…
// ඳը const ctx = canvas.getContext('2d'); let ctxImageData = ctx.getImageData(0, 0, canvas.width, canvas.height); let bytes = ctxImageData.data; for (let i = 0; i < canvas.height; i++) { for (let j = 0; j < canvas.width; j++) { const n = i * canvas.width + j; if (segmentation.data[n] === 1) { // for foreground (ਓ) bytes[4 * n + 0] = fgImg.data[4 * n + 0]; bytes[4 * n + 1] = fgImg.data[4 * n + 1]; bytes[4 * n + 2] = fgImg.data[4 * n + 2]; bytes[4 * n + 3] = fgImg.data[4 * n + 3]; } else { // for background (എܠ) bytes[4 * n + 0] = bgImg.data[4 * n + 0]; bytes[4 * n + 1] = bgImg.data[4 * n + 1]; bytes[4 * n + 2] = bgImg.data[4 * n + 2]; bytes[4 * n + 3] = bgImg.data[4 * n + 3]; } } } ctx.putImageData(ctxImageData, 0, 0); } • ֤ϐΫηϧͷηάϝϯςʔγϣ ϯఆʹԠͯ͡ɺΧϝϥը૾͔ എܠը૾Λcanvasʹॻ͖ࠐΉ • BodyPixͷtoMask()Ͱಉ͡Α ͏ͳख๏Ͱߦ͍ͬͯͨͷͰ࠾༻ 13 IUUQTHJUIVCDPNUFOTPSqPXUGKTNPEFMTCMPCNBTUFSCPEZQJYTSDPVUQVU@SFOEFSJOH@VUJMUT--
OBS VirtualCam • OBSಈը৴ͷࡍʹ֤छฤू͕Ͱ͖Δɺແྉͷπʔϧ • OBS VirtualCamOBSͷϓϥάΠϯͱͯ͠ΠϯετʔϧͰ͖Δ • ࠓճϒϥβΛೖྗιʔεͱͯ͠OBSʹऔΓࠐΈɺදࣔαΠζΛద ٓमਖ਼ͯ͠ɺOBS
VirtualCamͷԾΧϝϥʹೖྗιʔεͱͯ͠ઃఆ • OBS VirtualCamΧϝϥͷ1ͭͱͯ͠ೝࣝ͞ΕΔͷͰɺZoom Google MeetͷೖྗΧϝϥͱͯ͠ઃఆͰ͖Δ 14
՝ • ը૾ͷΓସ͑ػೳͳ͠ ʗ ΧϝϥͷΓସ͑ػೳͳ͠ • ࠨӈసͰ͖ͳ͍ • canvasΛࠨӈసͤ͞Εɺഎܠ͝ͱసͰ͖ͦ͏ •
CPUύϫʔ͕ඞཁ • ಛʹOBS VirtualCamΛซ༻͢ΔͱɺCPUϑΝϯ͕ᄬΔ • OBS VirtualCamͷઃఆ͕໘ • Chrome ExtensionͰ࡞͍͍͔ͬͯ 15
·ͱΊ • TensorFlow.js ͱ BodyPix ϞσϧͰ؆୯ʹ࣮Ͱ͖ͨ • PoseNetͱΈ߹ΘͤΔͱɺΑΓ໘ന͍͔ • ՆٳΈͷࣗ༝ݚڀωλͱͯ͠ɺ͓͍͍ͩ͘͞
16
એ 17
JAWS SONIC 2020 & MIDNIGHT JAWS 2020 JAWS-UGͷ24hΦϯϥΠϯΠϕϯτ 9/12() 16:50
- 9/13() 17:20 JAWS-UG৽ׁͰΔ͔ IUUQTKBXTTPOJDKBXTVHKQ 18
JaSST’20 Niigata ιϑτΣΞςετγϯϙδϜ ΦϯϥΠϯ։࠵ 9/28(݄) 13:00 - 17:00 (༧ఆ) ςʔϚ:
ςετࣗಈԽ IUUQKBTTUKQTZNQPTJVNKBTTUOJJHBUBIUNM 19
͓ΘΓ 20
࣮ͨ͠ͷʢ࠶ܝʣ • σϞαΠτ https://tfug-niigata-bodypix-demo.netlify.app/ • ιʔείʔυ https://github.com/kasacchiful/bodypix-sample 21
ࢀߟ • https://developers-jp.googleblog.com/2019/04/bodypix- tensorflowjs.html • https://github.com/tensorflow/tfjs-models/tree/master/body-pix • https://note.com/prty/n/ne2d2090148dc 22