Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
TensorFlow.js で バーチャル背景を作る / virtual background...
Search
kasacchiful
August 08, 2020
Programming
0
1.3k
TensorFlow.js で バーチャル背景を作る / virtual background with tfjs
2020/08/08 (土) に TensorFlow UG Niigata #3 にて発表した資料です。
TensorFlow.js を使ってバーチャル背景を作ってみました。
kasacchiful
August 08, 2020
Tweet
Share
More Decks by kasacchiful
See All by kasacchiful
Amazon S3 TablesとAmazon S3 Metadataを触ってみた / 20250201-jawsug-tochigi-s3tables-s3metadata
kasacchiful
0
190
Amazon S3 TablesとAmazon S3 Metadataを動かしてみた / 20250125-niigata-5min-tech-lt
kasacchiful
0
20
dbt coreとFargateでデータ変換 / 20240928-jawsug-toyama-hokuriku-shinkansen
kasacchiful
1
97
What we keep in mind when migrating from Serverless Framework to AWS CDK and AWS SAM
kasacchiful
1
350
AWSでIcebergを使ってデータウェアハウスを構築してみる / 20240810-jawsug-akita
kasacchiful
0
43
サーバーレスパターンを元にAWS CDKでデータ基盤を構築する / 20240731_classmethod_odyssey_online_build_a_data_infrastructures_using_aws_cdk_based_on_serverless_patterns
kasacchiful
0
500
AWS IoT 1-clickがサービス終了するので、SORACOMに移行した話 / 20240518-jawsug-niigata-iotlt-niigata
kasacchiful
0
270
AWS Application Composerで始める、 サーバーレスなデータ基盤構築 / 20240406-jawsug-hokuriku-shinkansen
kasacchiful
1
590
AWSの各種サービス紹介と活用方法 − AI・ML活用デモを交えて − / 20231208aws-aiml-seminar
kasacchiful
0
540
Other Decks in Programming
See All in Programming
Honoのおもしろいミドルウェアをみてみよう
yusukebe
1
230
color-scheme: light dark; を完全に理解する
uhyo
7
490
Ça bouge du côté des animations CSS !
goetter
2
150
Open source software: how to live long and go far
gaelvaroquaux
0
660
生成AIで加速するテスト実装 - ロリポップ for Gamersの事例と 生成AIエディタの活用
kinosuke01
0
110
仕様変更に耐えるための"今の"DRY原則を考える
mkmk884
9
3.2k
Generating OpenAPI schema from serializers throughout the Rails stack - Kyobashi.rb #5
envek
1
370
ソフトウェアエンジニアの成長
masuda220
PRO
12
2.1k
パスキーのすべて ── 導入・UX設計・実装の紹介 / 20250213 パスキー開発者の集い
kuralab
3
890
Formの複雑さに立ち向かう
bmthd
1
930
Rubyで始める関数型ドメインモデリング
shogo_tksk
0
140
はじめての Go * WASM *OCR
sgash708
1
100
Featured
See All Featured
The Power of CSS Pseudo Elements
geoffreycrofte
75
5.5k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
30
4.6k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
12
990
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Navigating Team Friction
lara
183
15k
Scaling GitHub
holman
459
140k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
27
1.6k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
49
2.3k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2.1k
Transcript
TensorFlow.jsͰ όʔνϟϧഎܠΛ࡞Δ ּݪ (@kasacchiful) 2020/08/08 () TensorFlow UG Niigata
#3
Software Developer Favorite: Community: • JAWS-UG Niigata • JaSST Niigata
• ASTER • SWANII • etc. Hiroshi Kasahara @kasacchiful @kasacchiful 2
࠷ۙϦϞʔτϫʔΫશ • ձٞதͷഎܠΛผͷࣸਅʹͯ͠͠·͏ʮόʔνϟϧഎܠʯΛઃఆͰ͖ Δπʔϧ͕૿͍͑ͯΔ • ͜ΕɺTensorFlow.jsͰ࣮Ͱ͖ΔΜ͡Όͳ͍ʁ • ͔֬ɺਓͷηάϝϯςʔγϣϯͰ͖ΔϞσϧެ։͞Ε͍ͯΔͣ 3
ຊͷ༰ • TensorFlow.js ΛͬͯɺόʔνϟϧഎܠΛ࣮ͯ͠Έͨ 4
2 ੍࡞ظؒ 5
BodyPix WebϒϥβͱTensorFlow.jsΛ ͬͨϦΞϧλΠϜͷਓηάϝϯ ςʔγϣϯϞσϧ https://github.com/tensorflow/tfjs-models/tree/master/body-pix Ҿ༻: https://developers-jp.googleblog.com/2019/04/bodypix-tensorflowjs.html 6
ॲཧͷྲྀΕ • WebΧϝϥͷಈըΛऔಘ • BodyPixͰਓͷηάϝϯςʔγϣϯΛ࣮ࢪ • ਓҎ֎ͷ෦ʢഎܠʣΛଞͷը૾ʹஔ͖͑Δ • ʢ͓·͚ʣOBS Virtual
CameraͰ֤छWebձٞπʔϧʹӨ͢Δ ιʔείʔυɺBodyPixͷσϞιʔεΛࢀߟʹվมͨ͠ 7
ߏਤ 8 ηάϝϯςʔγϣϯ ݁Ռ Χϝϥը૾ͱ എܠը૾Λ ߹ 8FCΧϝϥ ਓ͕өͬͨ Χϝϥը૾
എܠը૾ ϒϥβදࣔ 0#47JSUVBM$BN ֤छπʔϧʹදࣔ
σϞ 9
࣮ͨ͠ͷ • σϞαΠτ https://tfug-niigata-bodypix-demo.netlify.app/ • ιʔείʔυ https://github.com/kasacchiful/bodypix-sample 10
WebΧϝϥͷಈ ըΛऔಘ <div id='main' style='display:none'> <video id="video" playsinline style="display: none;"></video>
<canvas id="output"></canvas> <img id="bg_img" src="./bg.jpg" style="display: none;"> </div> • videoλάΛඇදࣔʹͯ͠Χϝϥ ىಈ&ಈը࠶ੜ • canvasλάʹදࣔ͢Δ͜ͱͰɺ ͍ΖΜͳը૾Λඳ͚Δ • എܠը૾imgλάʹࠐ·ͤͯ ͓͘ 11
ਓͷηάϝϯ ςʔγϣϯ async function estimateSegmentation() { return await state.net.segmentPerson(state.video, {
internalResolution: segmentationOption.internalResolution, segmentationThreshold: segmentationOption.segmentationThreshold, maxDetections: segmentationOption.multiDecodingMaxDetections, scoreThreshold: segmentationOption.multiDecodingScoreThreshold, nmsRadius: segmentationOption.multiDecodingNmsRadius, }); } function segmentBodyInRealTime() { const canvas = document.getElementById('output'); const img = document.getElementById('bg_img'); async function bodySegmentationFrame() { // segmentation const personSegmentation = await estimateSegmentation(); toMaskImage(canvas, personSegmentation, state.video, img); requestAnimationFrame(bodySegmentationFrame); } bodySegmentationFrame(); } • segmentPerson()Ͱηάϝ ϯςʔγϣϯऔಘ • ֤ϐΫηϧʹɺਓͳΒ”1” ਓҎ֎ͳΒ”0”͕༩͞Ε Δ 12
Χϝϥը૾ͱഎ ܠը૾ͷ߹ function toMaskImage(canvas, segmentation, video, img) { // লུ…
// ඳը const ctx = canvas.getContext('2d'); let ctxImageData = ctx.getImageData(0, 0, canvas.width, canvas.height); let bytes = ctxImageData.data; for (let i = 0; i < canvas.height; i++) { for (let j = 0; j < canvas.width; j++) { const n = i * canvas.width + j; if (segmentation.data[n] === 1) { // for foreground (ਓ) bytes[4 * n + 0] = fgImg.data[4 * n + 0]; bytes[4 * n + 1] = fgImg.data[4 * n + 1]; bytes[4 * n + 2] = fgImg.data[4 * n + 2]; bytes[4 * n + 3] = fgImg.data[4 * n + 3]; } else { // for background (എܠ) bytes[4 * n + 0] = bgImg.data[4 * n + 0]; bytes[4 * n + 1] = bgImg.data[4 * n + 1]; bytes[4 * n + 2] = bgImg.data[4 * n + 2]; bytes[4 * n + 3] = bgImg.data[4 * n + 3]; } } } ctx.putImageData(ctxImageData, 0, 0); } • ֤ϐΫηϧͷηάϝϯςʔγϣ ϯఆʹԠͯ͡ɺΧϝϥը૾͔ എܠը૾Λcanvasʹॻ͖ࠐΉ • BodyPixͷtoMask()Ͱಉ͡Α ͏ͳख๏Ͱߦ͍ͬͯͨͷͰ࠾༻ 13 IUUQTHJUIVCDPNUFOTPSqPXUGKTNPEFMTCMPCNBTUFSCPEZQJYTSDPVUQVU@SFOEFSJOH@VUJMUT--
OBS VirtualCam • OBSಈը৴ͷࡍʹ֤छฤू͕Ͱ͖Δɺແྉͷπʔϧ • OBS VirtualCamOBSͷϓϥάΠϯͱͯ͠ΠϯετʔϧͰ͖Δ • ࠓճϒϥβΛೖྗιʔεͱͯ͠OBSʹऔΓࠐΈɺදࣔαΠζΛద ٓमਖ਼ͯ͠ɺOBS
VirtualCamͷԾΧϝϥʹೖྗιʔεͱͯ͠ઃఆ • OBS VirtualCamΧϝϥͷ1ͭͱͯ͠ೝࣝ͞ΕΔͷͰɺZoom Google MeetͷೖྗΧϝϥͱͯ͠ઃఆͰ͖Δ 14
՝ • ը૾ͷΓସ͑ػೳͳ͠ ʗ ΧϝϥͷΓସ͑ػೳͳ͠ • ࠨӈసͰ͖ͳ͍ • canvasΛࠨӈసͤ͞Εɺഎܠ͝ͱసͰ͖ͦ͏ •
CPUύϫʔ͕ඞཁ • ಛʹOBS VirtualCamΛซ༻͢ΔͱɺCPUϑΝϯ͕ᄬΔ • OBS VirtualCamͷઃఆ͕໘ • Chrome ExtensionͰ࡞͍͍͔ͬͯ 15
·ͱΊ • TensorFlow.js ͱ BodyPix ϞσϧͰ؆୯ʹ࣮Ͱ͖ͨ • PoseNetͱΈ߹ΘͤΔͱɺΑΓ໘ന͍͔ • ՆٳΈͷࣗ༝ݚڀωλͱͯ͠ɺ͓͍͍ͩ͘͞
16
એ 17
JAWS SONIC 2020 & MIDNIGHT JAWS 2020 JAWS-UGͷ24hΦϯϥΠϯΠϕϯτ 9/12() 16:50
- 9/13() 17:20 JAWS-UG৽ׁͰΔ͔ IUUQTKBXTTPOJDKBXTVHKQ 18
JaSST’20 Niigata ιϑτΣΞςετγϯϙδϜ ΦϯϥΠϯ։࠵ 9/28(݄) 13:00 - 17:00 (༧ఆ) ςʔϚ:
ςετࣗಈԽ IUUQKBTTUKQTZNQPTJVNKBTTUOJJHBUBIUNM 19
͓ΘΓ 20
࣮ͨ͠ͷʢ࠶ܝʣ • σϞαΠτ https://tfug-niigata-bodypix-demo.netlify.app/ • ιʔείʔυ https://github.com/kasacchiful/bodypix-sample 21
ࢀߟ • https://developers-jp.googleblog.com/2019/04/bodypix- tensorflowjs.html • https://github.com/tensorflow/tfjs-models/tree/master/body-pix • https://note.com/prty/n/ne2d2090148dc 22