Slide 1
Slide 1 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Image Processing
with Vision Framework
shoheiyokoyama
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Slide 2
Slide 2 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
shoheiyokoyama
shoheiyokoyama
shoheiyokoyam
Slide 3
Slide 3 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
What’s Vision?
ɾiOS11͔Βಋೖ͞ΕͨϑϨʔϜϫʔΫ
ɾίϯϐϡʔλϏδϣϯΞϧΰϦζϜΛద༻
ɾը૾ಈըͷಛɾςΩετɾΦϒδΣΫτݕ
ɾτϥοΩϯά
Slide 4
Slide 4 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
ɾSaliency
ɾImage classfication
ɾImage Similarity
ɾOther New API
Slide 5
Slide 5 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
ɾSaliency
ɾImage classfication
ɾImage Similarity
ɾOther New API
Slide 6
Slide 6 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Saliency
ݦஶੑ
ײܹ͕֮ϘτϜΞοϓੑҙΛ༠Ҿ͢Δಛੑ
Slide 7
Slide 7 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Saliency
Saliency refers to what’s noticeable or
important in an image.
ը૾Ͱཱͭͷॏཁͳͷ
https://developer.apple.com/documentation/vision/cropping_images_using_saliency.
Cropping Images Using Saliency, Apple Developer Document by Apple Inc,
Slide 8
Slide 8 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Saliency
ɾ object-based saliency
ɾ attention-based saliency
Slide 9
Slide 9 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Saliency
ɾ object-based saliency
ɾ attention-based saliency
ਓʑ͕ݟ͍ͯΔՄೳੑ͕ߴ͍ / ͍
ඃࣸମલܠͷΦϒδΣΫτͷϋΠϥΠτ߹͍
Slide 10
Slide 10 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Saliency
Slide 11
Slide 11 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Heatmap and Bounding Box
object-based
attention-based
Slide 12
Slide 12 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Attention-based saliency
ɾίϯτϥετ
ɾإ
ɾඃࣸମ
ɾฏઢ
ɾޫ
ɾ…
༷ʑͳཁҼ͕ݦஶੑΛܾఆ͚ͮΔͨΊෳࡶͰ͋Δ
Slide 13
Slide 13 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
MFUSFRVFTU)BOEMFS7/*NBHF3FRVFTU)BOEMFS DW1JYFM#V⒎FSTBNQMF#V⒎FS
PSJFOUBUJPOSJHIU
PQUJPOT<>
MFUSFRVFTU7/3FRVFTU7/(FOFSBUF"UUFOUJPO#BTFE4BMJFODZ*NBHF3FRVFTU
USZ SFRVFTU)BOEMFSQFSGPSN
MFUPCTFSWBUJPOSFRVFTUSFTVMUT pSTUBT 7/4BMJFODZ*NBHF0CTFSWBUJPO
MFUQJYFM#V⒎FSPCTFSWBUJPO QJYFM#V⒎FS
Attention-based saliency
Slide 14
Slide 14 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
MFUSFRVFTU)BOEMFS7/*NBHF3FRVFTU)BOEMFS DW1JYFM#V⒎FSTBNQMF#V⒎FS
PSJFOUBUJPOSJHIU
PQUJPOT<>
MFUSFRVFTU7/3FRVFTU7/(FOFSBUF0CKFDUOFTT#BTFE4BMJFODZ*NBHF3FRVFTU
USZ SFRVFTU)BOEMFSQFSGPSN
MFUPCTFSWBUJPOSFRVFTUSFTVMUT pSTUBT 7/4BMJFODZ*NBHF0CTFSWBUJPO
MFUQJYFM#V⒎FSPCTFSWBUJPO QJYFM#V⒎FS
Object-based saliency
Slide 15
Slide 15 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
MFUSFRVFTU)BOEMFS7/*NBHF3FRVFTU)BOEMFS VSMJNBHF63-
NBLFSFRVFTUXJUIJNBHF63-
MFUSFRVFTU7/3FRVFTU7/(FOFSBUF0CKFDUOFTT#BTFE4BMJFODZ*NBHF3FRVFTU
USZ SFRVFTU)BOEMFSQFSGPSN
MFUPCTFSWBUJPOSFRVFTUSFTVMUT pSTUBT 7/4BMJFODZ*NBHF0CTFSWBUJPO
MFUQJYFM#V⒎FSPCTFSWBUJPO QJYFM#V⒎FS
Object-based saliency
Slide 16
Slide 16 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Use Saliency
ɾը૾ͷશܠཱͭՕॴΛڧௐ
ɾࣸਅϏϡΞʔͳͲͰϥϯμϜʹݟ͑Δ෦Λ֦େ
Slide 17
Slide 17 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
ɾSaliency
ɾImage classfication
ɾImage Similarity
ɾOther New API
Slide 18
Slide 18 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Image classfication
Cat
Dog
Pug
Animal
Mamal
Slide 19
Slide 19 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Image classfication
େنͳྨωοτϫʔΫΛθϩ͔Β։ൃ
Կඦສͷը૾ɺ܇࿅͢ΔͨΊͷԿઍ࣌ؒͷ࣌ؒɺඇৗʹઐతͳઐ͕ࣝඞཁ
Slide 20
Slide 20 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Image classfication
େنͳྨωοτϫʔΫΛθϩ͔Β։ൃ
Կඦສͷը૾ɺ܇࿅͢ΔͨΊͷԿઍ࣌ؒͷ࣌ؒɺඇৗʹઐతͳઐ͕ࣝඞཁ
Apple͜ͷϓϩηεΛऴ͑ɺେنྨωοτϫʔΫΛެ։͢Δஈ֊
Slide 21
Slide 21 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Image classfication
ެ։͍ͯ͠ΔྨωοτϫʔΫ
ɹɹɹࣸਅݕࡧͳͲʹ׆༻
Slide 22
Slide 22 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Taxonomy
ɾ֊ߏʹͳͬͨ1000Ϋϥεͷྨ
ɾҙຯΛ࣋ͭΫϥεͰάϧʔϓԽ
ɾํؔΛ࣋ͪɺࢠؔΛ࣋ͭ
ಈ
ೣ
ݘ
ᄡೕྨ
ϖϧγϟ
ύά
Slide 23
Slide 23 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
MFUPCTFSWBUJPOTUSZ7/$MBTTJGZ*NBHF3FRVFTULOPXO$MBTTJpDBUJPOT GPS3FWJTJPO7/$MBTTJGZ*NBHF3FRVFTU3FWJTJPO
QSJOU PCTFSWBUJPOTDPVOU
PCTFSWBUJPOTGPS&BDI\QSJOU JEFOUJpFS
^
BCBDVT
BDDPSEJPO
BDPSO
BDSPCBU
BEVMU
BEVMU@DBU
ɻɻɻ
ZPHVSU
ZPML
[FCSB
[PP
[VDDIJOJ
Slide 24
Slide 24 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
MFUIBOEMFS7/*NBHF3FRVFTU)BOEMFS VSMJNBHF63-
MFUSFRVFTU7/$MBTTJGZ*NBHF3FRVFTU
USZ IBOEMFSQFSGPSN
MFUPCTFSWBUJPOTSFRVFTUSFTVMUTBT <7/$MBTTJpDBUJPO0CTFSWBUJPO>
Sample code
Slide 25
Slide 25 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
MFUIBOEMFS7/*NBHF3FRVFTU)BOEMFS VSMJNBHF63-
MFUSFRVFTU7/$MBTTJGZ*NBHF3FRVFTU
USZ IBOEMFSQFSGPSN
MFUPCTFSWBUJPOTSFRVFTUSFTVMUTBT <7/$MBTTJpDBUJPO0CTFSWBUJPO>
Sample code
<
BOJNBM
DBU
NBNBM
ʜ
>
*EFOUJpFS$POpEFODF
Slide 26
Slide 26 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Filtering result
hasMinimumPrecision(_:forRecall:)
hasMinimumRecall(_:forPrecision:)
https://developer.apple.com/documentation/vision/vnclassificationobservation
Slide 27
Slide 27 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Precision and recall
ɾPresision
ɾRecall
݁Ռͷ͏ͪɺͲͷ͘Β͍݁Ռ͕ਖ਼͍͔͠
ਖ਼ղͷதͰɺ݁ՌͷதʹͲΕ͘Β͍ਖ਼ղؚ͕·Ε͍ͯΔ͔
Slide 28
Slide 28 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Precision and recall
ݕࡧΤϯδϯͷྫ
ɾݕࡧΤϯδϯͰ30݅ώοτ
ɾͦͷ͏ͪ20͕݅ؔ࿈
ɾؔ࿈͍ͯ͠Δ40͕݅ώοτ͠ͳ͔ͬͨ
Slide 29
Slide 29 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Precision and recall
ɾPresision
ݕࡧΤϯδϯͷྫ
ɾݕࡧΤϯδϯͰ30݅ώοτ
ɾͦͷ͏ͪ20͕݅ؔ࿈
ɾؔ࿈͍ͯ͠Δ40͕݅ώοτ͠ͳ͔ͬͨ
20/30 = 2/3
ɾRecall
20/60 = 1/3
Slide 30
Slide 30 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Precision and recall
ݕࡧΤϯδϯͷྫ
ɾݕࡧΤϯδϯͰ30݅ώοτ
ɾͦͷ͏ͪ20͕݅ؔ࿈
ɾؔ࿈͍ͯ͠Δ40͕݅ώοτ͠ͳ͔ͬͨ A B
C
A: ݕࡧ݁Ռ
B: શମͷݕࡧཁٻΛຬͨ͢ϖʔδ
C: ݕࡧ݁ՌͷதͷݕࡧཁٻΛຬͨ͢ϖʔδ
Slide 31
Slide 31 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Precision and recall
A: ݕࡧ݁Ռ
B: શମͷݕࡧཁٻΛຬͨ͢ϖʔδ
C: ݕࡧ݁ՌͷதͷݕࡧཁٻΛຬͨ͢ϖʔδ
ɾPresision
C / A
ɾRecall
C / B
A B
C
Slide 32
Slide 32 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Precision and recall
A: ݕࡧ݁Ռ
B: શମͷݕࡧཁٻΛຬͨ͢ϖʔδ
C: ݕࡧ݁ՌͷதͷݕࡧཁٻΛຬͨ͢ϖʔδ
ɾPresision
ɾRecall A B
C
ͲΕ͚ͩແؔͳ݁ՌΑΓ
ؔ࿈ੑͷߴ͍݁ՌΛฦ͔͢
ͲΕ͚ͩΞϧΰϦζϜ͕ؔ࿈ੑ
ͷ͋Δ݁ՌΛฦ͔͢
Slide 33
Slide 33 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
The PR Curve
Presision
Recall
1
1
0
ɾҰํ্͕͕Εଞํ͕Լ͢Δ
ɾΫϥεʹΑͬͯݻ༗ͷΧʔϒΛඳ͘
Slide 34
Slide 34 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
MFUIBOEMFS7/*NBHF3FRVFTU)BOEMFS VSMJNBHF63-
MFUSFRVFTU7/$MBTTJGZ*NBHF3FRVFTU
USZ IBOEMFSQFSGPSN
MFUPCTFSWBUJPOTSFRVFTUSFTVMUTBT <7/$MBTTJpDBUJPO0CTFSWBUJPO>
MFUpMUFSFEPCTFSWBUJPOT pMUFS\IBT.JOJNVN1SFDJTJPO
GPS3FDBMM
^
Sample code
݁ՌΛϑΟϧλϦϯά
Slide 35
Slide 35 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Filtering the PR Curve
Presision
Recall
1
1
0
observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) }
Slide 36
Slide 36 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Filtering the PR Curve
Presision
Recall
1
1
0
observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) }
Precision > 0.4
Slide 37
Slide 37 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Filtering the PR Curve
Presision
Recall
1
1
0
observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) }
Precision > 0.4
Recall = 0.5
Slide 38
Slide 38 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Filtering the PR Curve
Presision
Recall
1
1
0
observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.9) }
Precision > 0.4
Recall = 0.9
Slide 39
Slide 39 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Filtering the PR Curve
Presision
Recall
1
1
0
observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.9) }
Precision > 0.4
Recall = 0.9
ΞϓϦέʔγϣϯͷ
ಛੑʹ͋ͬͨᮢΛઃఆ͢Δ
Slide 40
Slide 40 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
F-measure
f − measure =
2 × precision × recall
precision + recall
F͕ߴ͍΄ͲɺόϥϯεΑ͘ํͷ͕ߴ͍
Slide 41
Slide 41 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Image classfication summary
ɾը૾ΦϒδΣΫτΛೝࣝͯ͠ɺΫϥεͰྨ͞Εͨ݁ՌΛऔಘՄೳ
ɾpresision, recallΛཧղ͢Δ
ɾΞϓέʔγϣϯಛ༗ͷᮢΛઃఆ͢Δ
Slide 42
Slide 42 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
ɾSaliency
ɾImage classfication
ɾImage Similarity
ɾOther New API
Slide 43
Slide 43 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Image Similarity
ɾྨωοτϫʔΫ͕ը૾ͷදݱΛ࡞
ɾݟ͚ͨͩͰͳ͘ɺը૾ͷதͰྨࣅΛఆ
ɾFeaturePrintͱ͍͏ई͕ྨࣅੑΛࣔ͢
Slide 44
Slide 44 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
FeaturePrint
ɾ୯ޠϕΫτϧͷΑ͏ͳը૾σΟεΫϦϓλ
ɾ͕খ͍͞΄Ͳɺྨࣅੑͷߴ͞Λࣔ͢
Slide 45
Slide 45 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
MFUIBOEMFS7/*NBHF3FRVFTU)BOEMFS VSMJNBHF63-
MFUSFRVFTU7/(FOFSBUF*NBHF'FBUVSF1SJOU3FRVFTU
USZ IBOEMFSQFSGPSN
MFUDPOUFTUBOU'10SFRVFTUSFTVMUTpSTUBT 7/'FBUVSF1SJOU0CTFSWBUJPO
WBSEJTUBODF'MPBU
USZDPOUFTUBOU'10DPNQVUF%JTUBODF EJTUBODF
UPPSJHJOBM'10
Sample code
Slide 46
Slide 46 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
ɾSaliency
ɾImage classfication
ɾImage Similarity
ɾOther New API
Slide 47
Slide 47 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Other New API
ɾFace capture quality
ɾFace Landmarks Enhancements
ɾHuman detector
ɾCat and Dog detector
ɾNew Object Tracker
ɾVision and CoreML Integration Enhancements
Slide 48
Slide 48 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
References
ɾhttps://developer.apple.com/documentation/vision
ɾhttps://developer.apple.com/videos/play/wwdc2019/222
Slide 49
Slide 49 text
CA.swift
CyberaAgent Presents iOS Developer’s Meetup
Thank you
CA.swift
CyberaAgent Presents iOS Developer’s Meetup