Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
TensorFlow.js で バーチャル背景を作る / virtual background...
Search
kasacchiful
August 08, 2020
Programming
0
1.2k
TensorFlow.js で バーチャル背景を作る / virtual background with tfjs
2020/08/08 (土) に TensorFlow UG Niigata #3 にて発表した資料です。
TensorFlow.js を使ってバーチャル背景を作ってみました。
kasacchiful
August 08, 2020
Tweet
Share
More Decks by kasacchiful
See All by kasacchiful
dbt coreとFargateでデータ変換 / 20240928-jawsug-toyama-hokuriku-shinkansen
kasacchiful
1
13
What we keep in mind when migrating from Serverless Framework to AWS CDK and AWS SAM
kasacchiful
1
170
AWSでIcebergを使ってデータウェアハウスを構築してみる / 20240810-jawsug-akita
kasacchiful
0
21
サーバーレスパターンを元にAWS CDKでデータ基盤を構築する / 20240731_classmethod_odyssey_online_build_a_data_infrastructures_using_aws_cdk_based_on_serverless_patterns
kasacchiful
0
340
AWS IoT 1-clickがサービス終了するので、SORACOMに移行した話 / 20240518-jawsug-niigata-iotlt-niigata
kasacchiful
0
170
AWS Application Composerで始める、 サーバーレスなデータ基盤構築 / 20240406-jawsug-hokuriku-shinkansen
kasacchiful
1
480
AWSの各種サービス紹介と活用方法 − AI・ML活用デモを交えて − / 20231208aws-aiml-seminar
kasacchiful
0
470
Amazon Rekognition デモ / 20231208-aws_seminar-01-rekognition-demo
kasacchiful
0
480
Amazon Lookout for Vision デモ / 20231208-aws_seminar-02-lookout-vision-demo
kasacchiful
0
480
Other Decks in Programming
See All in Programming
文化が生産性を作る
jimpei
3
500
複数プロダクトの技術改善・クラウド移行に向き合うチームのフレキシブルなペア・モブプログラミングの実践 / Flexible Pair Programming And Mob Programming
honyanya
0
150
AWS CDKを用いたセキュアなCI/CDパイプラインの構築 / Build a secure CI/CD pipeline using AWS CDK
seike460
PRO
3
500
Removing Corepack
yosuke_furukawa
PRO
9
1.1k
Делим тесты между QA и разработчиком
mariyasaygina
0
480
pytest プラグインを開発して DRY に自動テストを書こう
inuatsu
2
240
コードレビューと私の過去と未来
jxmtst
0
180
ファーストペンギンBot @Qiita Hackathon 2024 予選
dyson_web
0
210
型付きで行うVSCode拡張機能開発 / VSCode Meetup #31
mazrean
0
230
dbt-ga4パッケージを実業務に導入してみた話
t_tokumaru_feedcorp
0
120
[KR] Server Driven Compose With Firebase
skydoves
2
130
全方位強化 Python 服務可觀測性:以 FastAPI 和 Grafana Stack 為例
blueswen
1
360
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
In The Pink: A Labor of Love
frogandcode
139
22k
Thoughts on Productivity
jonyablonski
67
4.2k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
6
230
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
191
16k
No one is an island. Learnings from fostering a developers community.
thoeni
19
2.9k
Visualization
eitanlees
143
15k
Building a Modern Day E-commerce SEO Strategy
aleyda
37
6.8k
Rails Girls Zürich Keynote
gr2m
93
13k
Intergalactic Javascript Robots from Outer Space
tanoku
268
27k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9k
Agile that works and the tools we love
rasmusluckow
327
21k
Transcript
TensorFlow.jsͰ όʔνϟϧഎܠΛ࡞Δ ּݪ (@kasacchiful) 2020/08/08 () TensorFlow UG Niigata
#3
Software Developer Favorite: Community: • JAWS-UG Niigata • JaSST Niigata
• ASTER • SWANII • etc. Hiroshi Kasahara @kasacchiful @kasacchiful 2
࠷ۙϦϞʔτϫʔΫશ • ձٞதͷഎܠΛผͷࣸਅʹͯ͠͠·͏ʮόʔνϟϧഎܠʯΛઃఆͰ͖ Δπʔϧ͕૿͍͑ͯΔ • ͜ΕɺTensorFlow.jsͰ࣮Ͱ͖ΔΜ͡Όͳ͍ʁ • ͔֬ɺਓͷηάϝϯςʔγϣϯͰ͖ΔϞσϧެ։͞Ε͍ͯΔͣ 3
ຊͷ༰ • TensorFlow.js ΛͬͯɺόʔνϟϧഎܠΛ࣮ͯ͠Έͨ 4
2 ੍࡞ظؒ 5
BodyPix WebϒϥβͱTensorFlow.jsΛ ͬͨϦΞϧλΠϜͷਓηάϝϯ ςʔγϣϯϞσϧ https://github.com/tensorflow/tfjs-models/tree/master/body-pix Ҿ༻: https://developers-jp.googleblog.com/2019/04/bodypix-tensorflowjs.html 6
ॲཧͷྲྀΕ • WebΧϝϥͷಈըΛऔಘ • BodyPixͰਓͷηάϝϯςʔγϣϯΛ࣮ࢪ • ਓҎ֎ͷ෦ʢഎܠʣΛଞͷը૾ʹஔ͖͑Δ • ʢ͓·͚ʣOBS Virtual
CameraͰ֤छWebձٞπʔϧʹӨ͢Δ ιʔείʔυɺBodyPixͷσϞιʔεΛࢀߟʹվมͨ͠ 7
ߏਤ 8 ηάϝϯςʔγϣϯ ݁Ռ Χϝϥը૾ͱ എܠը૾Λ ߹ 8FCΧϝϥ ਓ͕өͬͨ Χϝϥը૾
എܠը૾ ϒϥβදࣔ 0#47JSUVBM$BN ֤छπʔϧʹදࣔ
σϞ 9
࣮ͨ͠ͷ • σϞαΠτ https://tfug-niigata-bodypix-demo.netlify.app/ • ιʔείʔυ https://github.com/kasacchiful/bodypix-sample 10
WebΧϝϥͷಈ ըΛऔಘ <div id='main' style='display:none'> <video id="video" playsinline style="display: none;"></video>
<canvas id="output"></canvas> <img id="bg_img" src="./bg.jpg" style="display: none;"> </div> • videoλάΛඇදࣔʹͯ͠Χϝϥ ىಈ&ಈը࠶ੜ • canvasλάʹදࣔ͢Δ͜ͱͰɺ ͍ΖΜͳը૾Λඳ͚Δ • എܠը૾imgλάʹࠐ·ͤͯ ͓͘ 11
ਓͷηάϝϯ ςʔγϣϯ async function estimateSegmentation() { return await state.net.segmentPerson(state.video, {
internalResolution: segmentationOption.internalResolution, segmentationThreshold: segmentationOption.segmentationThreshold, maxDetections: segmentationOption.multiDecodingMaxDetections, scoreThreshold: segmentationOption.multiDecodingScoreThreshold, nmsRadius: segmentationOption.multiDecodingNmsRadius, }); } function segmentBodyInRealTime() { const canvas = document.getElementById('output'); const img = document.getElementById('bg_img'); async function bodySegmentationFrame() { // segmentation const personSegmentation = await estimateSegmentation(); toMaskImage(canvas, personSegmentation, state.video, img); requestAnimationFrame(bodySegmentationFrame); } bodySegmentationFrame(); } • segmentPerson()Ͱηάϝ ϯςʔγϣϯऔಘ • ֤ϐΫηϧʹɺਓͳΒ”1” ਓҎ֎ͳΒ”0”͕༩͞Ε Δ 12
Χϝϥը૾ͱഎ ܠը૾ͷ߹ function toMaskImage(canvas, segmentation, video, img) { // লུ…
// ඳը const ctx = canvas.getContext('2d'); let ctxImageData = ctx.getImageData(0, 0, canvas.width, canvas.height); let bytes = ctxImageData.data; for (let i = 0; i < canvas.height; i++) { for (let j = 0; j < canvas.width; j++) { const n = i * canvas.width + j; if (segmentation.data[n] === 1) { // for foreground (ਓ) bytes[4 * n + 0] = fgImg.data[4 * n + 0]; bytes[4 * n + 1] = fgImg.data[4 * n + 1]; bytes[4 * n + 2] = fgImg.data[4 * n + 2]; bytes[4 * n + 3] = fgImg.data[4 * n + 3]; } else { // for background (എܠ) bytes[4 * n + 0] = bgImg.data[4 * n + 0]; bytes[4 * n + 1] = bgImg.data[4 * n + 1]; bytes[4 * n + 2] = bgImg.data[4 * n + 2]; bytes[4 * n + 3] = bgImg.data[4 * n + 3]; } } } ctx.putImageData(ctxImageData, 0, 0); } • ֤ϐΫηϧͷηάϝϯςʔγϣ ϯఆʹԠͯ͡ɺΧϝϥը૾͔ എܠը૾Λcanvasʹॻ͖ࠐΉ • BodyPixͷtoMask()Ͱಉ͡Α ͏ͳख๏Ͱߦ͍ͬͯͨͷͰ࠾༻ 13 IUUQTHJUIVCDPNUFOTPSqPXUGKTNPEFMTCMPCNBTUFSCPEZQJYTSDPVUQVU@SFOEFSJOH@VUJMUT--
OBS VirtualCam • OBSಈը৴ͷࡍʹ֤छฤू͕Ͱ͖Δɺແྉͷπʔϧ • OBS VirtualCamOBSͷϓϥάΠϯͱͯ͠ΠϯετʔϧͰ͖Δ • ࠓճϒϥβΛೖྗιʔεͱͯ͠OBSʹऔΓࠐΈɺදࣔαΠζΛద ٓमਖ਼ͯ͠ɺOBS
VirtualCamͷԾΧϝϥʹೖྗιʔεͱͯ͠ઃఆ • OBS VirtualCamΧϝϥͷ1ͭͱͯ͠ೝࣝ͞ΕΔͷͰɺZoom Google MeetͷೖྗΧϝϥͱͯ͠ઃఆͰ͖Δ 14
՝ • ը૾ͷΓସ͑ػೳͳ͠ ʗ ΧϝϥͷΓସ͑ػೳͳ͠ • ࠨӈసͰ͖ͳ͍ • canvasΛࠨӈసͤ͞Εɺഎܠ͝ͱసͰ͖ͦ͏ •
CPUύϫʔ͕ඞཁ • ಛʹOBS VirtualCamΛซ༻͢ΔͱɺCPUϑΝϯ͕ᄬΔ • OBS VirtualCamͷઃఆ͕໘ • Chrome ExtensionͰ࡞͍͍͔ͬͯ 15
·ͱΊ • TensorFlow.js ͱ BodyPix ϞσϧͰ؆୯ʹ࣮Ͱ͖ͨ • PoseNetͱΈ߹ΘͤΔͱɺΑΓ໘ന͍͔ • ՆٳΈͷࣗ༝ݚڀωλͱͯ͠ɺ͓͍͍ͩ͘͞
16
એ 17
JAWS SONIC 2020 & MIDNIGHT JAWS 2020 JAWS-UGͷ24hΦϯϥΠϯΠϕϯτ 9/12() 16:50
- 9/13() 17:20 JAWS-UG৽ׁͰΔ͔ IUUQTKBXTTPOJDKBXTVHKQ 18
JaSST’20 Niigata ιϑτΣΞςετγϯϙδϜ ΦϯϥΠϯ։࠵ 9/28(݄) 13:00 - 17:00 (༧ఆ) ςʔϚ:
ςετࣗಈԽ IUUQKBTTUKQTZNQPTJVNKBTTUOJJHBUBIUNM 19
͓ΘΓ 20
࣮ͨ͠ͷʢ࠶ܝʣ • σϞαΠτ https://tfug-niigata-bodypix-demo.netlify.app/ • ιʔείʔυ https://github.com/kasacchiful/bodypix-sample 21
ࢀߟ • https://developers-jp.googleblog.com/2019/04/bodypix- tensorflowjs.html • https://github.com/tensorflow/tfjs-models/tree/master/body-pix • https://note.com/prty/n/ne2d2090148dc 22