Share
Embed
Image Processing with Vision Framework
- Vision
https://developer.apple.com/documentation/vision
- Understanding Images in Vision Framework
https://developer.apple.com/videos/play/wwdc2019/222
Shohei Yokoyama
June 19, 2019
Transcript
49 slides
-
CA.swift CyberaAgent Presents iOS Developer’s Meetup Image Processing with Vision
Framework shoheiyokoyama CA.swift CyberaAgent Presents iOS Developer’s Meetup -
CA.swift CyberaAgent Presents iOS Developer’s Meetup What’s Vision? ɾiOS11͔Βಋೖ͞ΕͨϑϨʔϜϫʔΫ ɾίϯϐϡʔλϏδϣϯΞϧΰϦζϜΛద༻
ɾը૾ಈըͷಛɾςΩετɾΦϒδΣΫτݕ ɾτϥοΩϯά -
CA.swift CyberaAgent Presents iOS Developer’s Meetup ɾSaliency ɾImage classfication ɾImage
Similarity ɾOther New API -
CA.swift CyberaAgent Presents iOS Developer’s Meetup ɾSaliency ɾImage classfication ɾImage
Similarity ɾOther New API -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Saliency Saliency refers to
what’s noticeable or important in an image. ը૾Ͱཱͭͷॏཁͳͷ https://developer.apple.com/documentation/vision/cropping_images_using_saliency. Cropping Images Using Saliency, Apple Developer Document by Apple Inc, -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Saliency ɾ object-based saliency
ɾ attention-based saliency -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Saliency ɾ object-based saliency
ɾ attention-based saliency ਓʑ͕ݟ͍ͯΔՄೳੑ͕ߴ͍ / ͍ ඃࣸମલܠͷΦϒδΣΫτͷϋΠϥΠτ߹͍ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Heatmap and Bounding Box
object-based attention-based -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Attention-based saliency ɾίϯτϥετ ɾإ
ɾඃࣸମ ɾฏઢ ɾޫ ɾ… ༷ʑͳཁҼ͕ݦஶੑΛܾఆ͚ͮΔͨΊෳࡶͰ͋Δ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup MFUSFRVFTU)BOEMFS7/*NBHF3FRVFTU)BOEMFS DW1JYFM#V⒎FSTBNQMF#V⒎FS PSJFOUBUJPOSJHIU PQUJPOT<>
MFUSFRVFTU7/3FRVFTU7/(FOFSBUF"UUFOUJPO#BTFE4BMJFODZ*NBHF3FRVFTU USZ SFRVFTU)BOEMFSQFSGPSN <SFRVFTU> MFUPCTFSWBUJPOSFRVFTUSFTVMUT pSTUBT 7/4BMJFODZ*NBHF0CTFSWBUJPO MFUQJYFM#V⒎FSPCTFSWBUJPO QJYFM#V⒎FS Attention-based saliency -
CA.swift CyberaAgent Presents iOS Developer’s Meetup MFUSFRVFTU)BOEMFS7/*NBHF3FRVFTU)BOEMFS DW1JYFM#V⒎FSTBNQMF#V⒎FS PSJFOUBUJPOSJHIU PQUJPOT<>
MFUSFRVFTU7/3FRVFTU7/(FOFSBUF0CKFDUOFTT#BTFE4BMJFODZ*NBHF3FRVFTU USZ SFRVFTU)BOEMFSQFSGPSN <SFRVFTU> MFUPCTFSWBUJPOSFRVFTUSFTVMUT pSTUBT 7/4BMJFODZ*NBHF0CTFSWBUJPO MFUQJYFM#V⒎FSPCTFSWBUJPO QJYFM#V⒎FS Object-based saliency -
CA.swift CyberaAgent Presents iOS Developer’s Meetup MFUSFRVFTU)BOEMFS7/*NBHF3FRVFTU)BOEMFS VSMJNBHF63- NBLFSFRVFTUXJUIJNBHF63- MFUSFRVFTU7/3FRVFTU7/(FOFSBUF0CKFDUOFTT#BTFE4BMJFODZ*NBHF3FRVFTU
USZ SFRVFTU)BOEMFSQFSGPSN <SFRVFTU> MFUPCTFSWBUJPOSFRVFTUSFTVMUT pSTUBT 7/4BMJFODZ*NBHF0CTFSWBUJPO MFUQJYFM#V⒎FSPCTFSWBUJPO QJYFM#V⒎FS Object-based saliency -
CA.swift CyberaAgent Presents iOS Developer’s Meetup ɾSaliency ɾImage classfication ɾImage
Similarity ɾOther New API -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Image classfication େنͳྨωοτϫʔΫΛθϩ͔Β։ൃ Կඦສͷը૾ɺ܇࿅͢ΔͨΊͷԿઍ࣌ؒͷ࣌ؒɺඇৗʹઐతͳઐ͕ࣝඞཁ
Apple͜ͷϓϩηεΛऴ͑ɺେنྨωοτϫʔΫΛެ։͢Δஈ֊ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup MFUPCTFSWBUJPOTUSZ7/$MBTTJGZ*NBHF3FRVFTULOPXO$MBTTJpDBUJPOT GPS3FWJTJPO7/$MBTTJGZ*NBHF3FRVFTU3FWJTJPO QSJOU
PCTFSWBUJPOTDPVOU PCTFSWBUJPOTGPS&BDI\QSJOU JEFOUJpFS ^ BCBDVT BDDPSEJPO BDPSO BDSPCBU BEVMU BEVMU@DBU ɻɻɻ ZPHVSU ZPML [FCSB [PP [VDDIJOJ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup MFUIBOEMFS7/*NBHF3FRVFTU)BOEMFS VSMJNBHF63- MFUSFRVFTU7/$MBTTJGZ*NBHF3FRVFTU
USZ IBOEMFSQFSGPSN <SFRVFTU> MFUPCTFSWBUJPOTSFRVFTUSFTVMUTBT <7/$MBTTJpDBUJPO0CTFSWBUJPO> Sample code -
CA.swift CyberaAgent Presents iOS Developer’s Meetup MFUIBOEMFS7/*NBHF3FRVFTU)BOEMFS VSMJNBHF63- MFUSFRVFTU7/$MBTTJGZ*NBHF3FRVFTU
USZ IBOEMFSQFSGPSN <SFRVFTU> MFUPCTFSWBUJPOTSFRVFTUSFTVMUTBT <7/$MBTTJpDBUJPO0CTFSWBUJPO> Sample code < BOJNBM DBU NBNBM ʜ > *EFOUJpFS$POpEFODF -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering result hasMinimumPrecision(_:forRecall:) hasMinimumRecall(_:forPrecision:)
https://developer.apple.com/documentation/vision/vnclassificationobservation -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall ɾPresision
ɾRecall ݁Ռͷ͏ͪɺͲͷ͘Β͍݁Ռ͕ਖ਼͍͔͠ ਖ਼ղͷதͰɺ݁ՌͷதʹͲΕ͘Β͍ਖ਼ղؚ͕·Ε͍ͯΔ͔ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall ݕࡧΤϯδϯͷྫ
ɾݕࡧΤϯδϯͰ30݅ώοτ ɾͦͷ͏ͪ20͕݅ؔ࿈ ɾؔ࿈͍ͯ͠Δ40͕݅ώοτ͠ͳ͔ͬͨ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall ɾPresision
ݕࡧΤϯδϯͷྫ ɾݕࡧΤϯδϯͰ30݅ώοτ ɾͦͷ͏ͪ20͕݅ؔ࿈ ɾؔ࿈͍ͯ͠Δ40͕݅ώοτ͠ͳ͔ͬͨ 20/30 = 2/3 ɾRecall 20/60 = 1/3 -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall ݕࡧΤϯδϯͷྫ
ɾݕࡧΤϯδϯͰ30݅ώοτ ɾͦͷ͏ͪ20͕݅ؔ࿈ ɾؔ࿈͍ͯ͠Δ40͕݅ώοτ͠ͳ͔ͬͨ A B C A: ݕࡧ݁Ռ B: શମͷݕࡧཁٻΛຬͨ͢ϖʔδ C: ݕࡧ݁ՌͷதͷݕࡧཁٻΛຬͨ͢ϖʔδ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall A:
ݕࡧ݁Ռ B: શମͷݕࡧཁٻΛຬͨ͢ϖʔδ C: ݕࡧ݁ՌͷதͷݕࡧཁٻΛຬͨ͢ϖʔδ ɾPresision C / A ɾRecall C / B A B C -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall A:
ݕࡧ݁Ռ B: શମͷݕࡧཁٻΛຬͨ͢ϖʔδ C: ݕࡧ݁ՌͷதͷݕࡧཁٻΛຬͨ͢ϖʔδ ɾPresision ɾRecall A B C ͲΕ͚ͩແؔͳ݁ՌΑΓ ؔ࿈ੑͷߴ͍݁ՌΛฦ͔͢ ͲΕ͚ͩΞϧΰϦζϜ͕ؔ࿈ੑ ͷ͋Δ݁ՌΛฦ͔͢ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup The PR Curve Presision
Recall 1 1 0 ɾҰํ্͕͕Εଞํ͕Լ͢Δ ɾΫϥεʹΑͬͯݻ༗ͷΧʔϒΛඳ͘ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup MFUIBOEMFS7/*NBHF3FRVFTU)BOEMFS VSMJNBHF63- MFUSFRVFTU7/$MBTTJGZ*NBHF3FRVFTU
USZ IBOEMFSQFSGPSN <SFRVFTU> MFUPCTFSWBUJPOTSFRVFTUSFTVMUTBT <7/$MBTTJpDBUJPO0CTFSWBUJPO> MFUpMUFSFEPCTFSWBUJPOT pMUFS\IBT.JOJNVN1SFDJTJPO GPS3FDBMM ^ Sample code ݁ՌΛϑΟϧλϦϯά -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering the PR Curve
Presision Recall 1 1 0 observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) } -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering the PR Curve
Presision Recall 1 1 0 observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) } Precision > 0.4 -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering the PR Curve
Presision Recall 1 1 0 observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) } Precision > 0.4 Recall = 0.5 -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering the PR Curve
Presision Recall 1 1 0 observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.9) } Precision > 0.4 Recall = 0.9 -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering the PR Curve
Presision Recall 1 1 0 observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.9) } Precision > 0.4 Recall = 0.9 ΞϓϦέʔγϣϯͷ ಛੑʹ͋ͬͨᮢΛઃఆ͢Δ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup F-measure f − measure
= 2 × precision × recall precision + recall F͕ߴ͍΄ͲɺόϥϯεΑ͘ํͷ͕ߴ͍ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Image classfication summary ɾը૾ΦϒδΣΫτΛೝࣝͯ͠ɺΫϥεͰྨ͞Εͨ݁ՌΛऔಘՄೳ
ɾpresision, recallΛཧղ͢Δ ɾΞϓέʔγϣϯಛ༗ͷᮢΛઃఆ͢Δ -
CA.swift CyberaAgent Presents iOS Developer’s Meetup ɾSaliency ɾImage classfication ɾImage
Similarity ɾOther New API -
CA.swift CyberaAgent Presents iOS Developer’s Meetup MFUIBOEMFS7/*NBHF3FRVFTU)BOEMFS VSMJNBHF63- MFUSFRVFTU7/(FOFSBUF*NBHF'FBUVSF1SJOU3FRVFTU
USZ IBOEMFSQFSGPSN <SFRVFTU> MFUDPOUFTUBOU'10SFRVFTUSFTVMUTpSTUBT 7/'FBUVSF1SJOU0CTFSWBUJPO WBSEJTUBODF'MPBU USZDPOUFTUBOU'10DPNQVUF%JTUBODF EJTUBODF UPPSJHJOBM'10 Sample code -
CA.swift CyberaAgent Presents iOS Developer’s Meetup ɾSaliency ɾImage classfication ɾImage
Similarity ɾOther New API -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Other New API ɾFace
capture quality ɾFace Landmarks Enhancements ɾHuman detector ɾCat and Dog detector ɾNew Object Tracker ɾVision and CoreML Integration Enhancements -
CA.swift CyberaAgent Presents iOS Developer’s Meetup Thank you CA.swift CyberaAgent
Presents iOS Developer’s Meetup