Image Processing with Vision Framework

- Vision
https://developer.apple.com/documentation/vision
- Understanding Images in Vision Framework
https://developer.apple.com/videos/play/wwdc2019/222

Transcript

1. CA.swift CyberaAgent Presents iOS Developer’s Meetup Image Processing with Vision Framework shoheiyokoyama CA.swift CyberaAgent Presents iOS Developer’s Meetup
2. CA.swift CyberaAgent Presents iOS Developer’s Meetup shoheiyokoyama shoheiyokoyama shoheiyokoyam
3. CA.swift CyberaAgent Presents iOS Developer’s Meetup What’s Vision? ɾiOS11͔Βಋೖ͞ΕͨϑϨʔϜϫʔΫ ɾίϯϐϡʔλϏδϣϯΞϧΰϦζϜΛద༻ ɾը૾΍ಈըͷಛ௃ɾςΩετɾΦϒδΣΫτݕ஌ ɾτϥοΩϯά
4. CA.swift CyberaAgent Presents iOS Developer’s Meetup ɾSaliency ɾImage classfication ɾImage Similarity ɾOther New API
5. CA.swift CyberaAgent Presents iOS Developer’s Meetup ɾSaliency ɾImage classfication ɾImage Similarity ɾOther New API
6. CA.swift CyberaAgent Presents iOS Developer’s Meetup Saliency ݦஶੑ ײܹ͕֮ࢗϘτϜΞοϓੑ஫ҙΛ༠Ҿ͢Δಛੑ
7. CA.swift CyberaAgent Presents iOS Developer’s Meetup Saliency Saliency refers to what’s noticeable or important in an image. 
 ը૾಺Ͱ໨ཱͭ΋ͷ΍ॏཁͳ΋ͷ https://developer.apple.com/documentation/vision/cropping_images_using_saliency. Cropping Images Using Saliency, Apple Developer Document by Apple Inc,
8. CA.swift CyberaAgent Presents iOS Developer’s Meetup Saliency ɾ object-based saliency ɾ attention-based saliency
9. CA.swift CyberaAgent Presents iOS Developer’s Meetup Saliency ɾ object-based saliency ɾ attention-based saliency ਓʑ͕ݟ͍ͯΔՄೳੑ͕ߴ͍ / ௿͍ ඃࣸମ΍લܠͷΦϒδΣΫτͷϋΠϥΠτ౓߹͍
10. CA.swift CyberaAgent Presents iOS Developer’s Meetup Saliency
11. CA.swift CyberaAgent Presents iOS Developer’s Meetup Heatmap and Bounding Box object-based attention-based
12. CA.swift CyberaAgent Presents iOS Developer’s Meetup Attention-based saliency ɾίϯτϥετ ɾإ ɾඃࣸମ ɾ஍ฏઢ ɾޫ ɾ… ༷ʑͳཁҼ͕ݦஶੑΛܾఆ͚ͮΔͨΊෳࡶͰ͋Δ
13. CA.swift CyberaAgent Presents iOS Developer’s Meetup MFU
    SFRVFTU)BOEMFS
    
    7/*NBHF3FRVFTU)BOEMFS DW1JYFM#V⒎FS
    TBNQMF#V⒎FS
    PSJFOUBUJPO
    SJHIU
    PQUJPOT
    <>
    MFU
    SFRVFTU
    7/3FRVFTU
    
    7/(FOFSBUF"UUFOUJPO#BTFE4BMJFODZ*NBHF3FRVFTU
    USZ
    SFRVFTU)BOEMFSQFSGPSN <SFRVFTU>
    MFU
    PCTFSWBUJPO
    
    SFRVFTUSFTVMUT pSTU
    BT
    7/4BMJFODZ*NBHF0CTFSWBUJPO
    MFU
    QJYFM#V⒎FS
    
    PCTFSWBUJPO QJYFM#V⒎FS Attention-based saliency
14. CA.swift CyberaAgent Presents iOS Developer’s Meetup MFU
    SFRVFTU)BOEMFS
    
    7/*NBHF3FRVFTU)BOEMFS DW1JYFM#V⒎FS
    TBNQMF#V⒎FS
    PSJFOUBUJPO
    SJHIU
    PQUJPOT
    <>
    MFU
    SFRVFTU
    7/3FRVFTU
    
    7/(FOFSBUF0CKFDUOFTT#BTFE4BMJFODZ*NBHF3FRVFTU
    USZ
    SFRVFTU)BOEMFSQFSGPSN <SFRVFTU>
    MFU
    PCTFSWBUJPO
    
    SFRVFTUSFTVMUT pSTU
    BT
    7/4BMJFODZ*NBHF0CTFSWBUJPO
    MFU
    QJYFM#V⒎FS
    
    PCTFSWBUJPO QJYFM#V⒎FS Object-based saliency
15. CA.swift CyberaAgent Presents iOS Developer’s Meetup MFU
    SFRVFTU)BOEMFS
    
    7/*NBHF3FRVFTU)BOEMFS VSM
    JNBHF63-
    
    NBLF
    SFRVFTU
    XJUI
    JNBHF63-
    MFU
    SFRVFTU
    7/3FRVFTU
    
    7/(FOFSBUF0CKFDUOFTT#BTFE4BMJFODZ*NBHF3FRVFTU
    USZ
    SFRVFTU)BOEMFSQFSGPSN <SFRVFTU>
    MFU
    PCTFSWBUJPO
    
    SFRVFTUSFTVMUT pSTU
    BT
    7/4BMJFODZ*NBHF0CTFSWBUJPO
    MFU
    QJYFM#V⒎FS
    
    PCTFSWBUJPO QJYFM#V⒎FS Object-based saliency
16. CA.swift CyberaAgent Presents iOS Developer’s Meetup Use Saliency ɾը૾಺ͷશܠ΍໨ཱͭՕॴΛڧௐ ɾࣸਅϏϡΞʔͳͲͰϥϯμϜʹݟ͑Δ෦෼Λ֦େ
17. CA.swift CyberaAgent Presents iOS Developer’s Meetup ɾSaliency ɾImage classfication ɾImage Similarity ɾOther New API
18. CA.swift CyberaAgent Presents iOS Developer’s Meetup Image classfication Cat Dog Pug Animal Mamal
19. CA.swift CyberaAgent Presents iOS Developer’s Meetup Image classfication େن໛ͳ෼ྨωοτϫʔΫΛθϩ͔Β։ൃ Կඦສ΋ͷը૾ɺ܇࿅͢ΔͨΊͷԿઍ࣌ؒ΋ͷ࣌ؒɺඇৗʹઐ໳తͳઐ໳஌͕ࣝඞཁ
20. CA.swift CyberaAgent Presents iOS Developer’s Meetup Image classfication େن໛ͳ෼ྨωοτϫʔΫΛθϩ͔Β։ൃ Կඦສ΋ͷը૾ɺ܇࿅͢ΔͨΊͷԿઍ࣌ؒ΋ͷ࣌ؒɺඇৗʹઐ໳తͳઐ໳஌͕ࣝඞཁ Apple͸͜ͷϓϩηεΛऴ͑ɺେن໛෼ྨωοτϫʔΫΛެ։͢Δஈ֊΁
21. CA.swift CyberaAgent Presents iOS Developer’s Meetup Image classfication ެ։͍ͯ͠Δ෼ྨωοτϫʔΫ ɹɹɹࣸਅݕࡧͳͲʹ׆༻
22. CA.swift CyberaAgent Presents iOS Developer’s Meetup Taxonomy ɾ֊૚ߏ଄ʹͳͬͨ໿1000Ϋϥεͷ෼ྨ ɾҙຯΛ࣋ͭΫϥεͰάϧʔϓԽ ɾํ޲ؔ܎Λ࣋ͪɺ਌ࢠؔ܎Λ࣋ͭ ಈ෺ ೣ ݘ ᄡೕྨ ϖϧγϟ ύά
23. CA.swift CyberaAgent Presents iOS Developer’s Meetup MFU
    PCTFSWBUJPOT
    
    USZ
    7/$MBTTJGZ*NBHF3FRVFTULOPXO$MBTTJpDBUJPOT GPS3FWJTJPO
    7/$MBTTJGZ*NBHF3FRVFTU3FWJTJPO
    QSJOU PCTFSWBUJPOTDPVOU
    
    
    PCTFSWBUJPOTGPS&BDI
    \
    QSJOU JEFOUJpFS
    ^ BCBDVT
    BDDPSEJPO
    BDPSO
    BDSPCBU
    BEVMU
    BEVMU@DBU
    ɻɻɻ
    ZPHVSU
    ZPML
    [FCSB
    [PP
    [VDDIJOJ
24. CA.swift CyberaAgent Presents iOS Developer’s Meetup MFU
    IBOEMFS
    
    7/*NBHF3FRVFTU)BOEMFS VSM
    JNBHF63-
    MFU
    SFRVFTU
    
    7/$MBTTJGZ*NBHF3FRVFTU
    USZ
    IBOEMFSQFSGPSN <SFRVFTU> 
 MFU
    PCTFSWBUJPOT
    
    SFRVFTUSFTVMUT
    BT
    <7/$MBTTJpDBUJPO0CTFSWBUJPO> Sample code
25. CA.swift CyberaAgent Presents iOS Developer’s Meetup MFU
    IBOEMFS
    
    7/*NBHF3FRVFTU)BOEMFS VSM
    JNBHF63-
    MFU
    SFRVFTU
    
    7/$MBTTJGZ*NBHF3FRVFTU
    USZ
    IBOEMFSQFSGPSN <SFRVFTU> 
 MFU
    PCTFSWBUJPOT
    
    SFRVFTUSFTVMUT
    BT
    <7/$MBTTJpDBUJPO0CTFSWBUJPO> Sample code <
    BOJNBM
    
    DBU
    
    NBNBM
    
    ʜ
    > *EFOUJpFS
    $POpEFODF
26. CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering result hasMinimumPrecision(_:forRecall:) hasMinimumRecall(_:forPrecision:) https://developer.apple.com/documentation/vision/vnclassificationobservation
27. CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall ɾPresision ɾRecall ݁Ռͷ͏ͪɺͲͷ͘Β͍݁Ռ͕ਖ਼͍͔͠ ਖ਼ղͷதͰɺ݁ՌͷதʹͲΕ͘Β͍ਖ਼ղؚ͕·Ε͍ͯΔ͔
28. CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall ݕࡧΤϯδϯͷྫ ɾݕࡧΤϯδϯͰ30݅ώοτ ɾͦͷ͏ͪ20͕݅ؔ࿈ ɾؔ࿈͍ͯ͠Δ40͕݅ώοτ͠ͳ͔ͬͨ
29. CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall ɾPresision ݕࡧΤϯδϯͷྫ ɾݕࡧΤϯδϯͰ30݅ώοτ ɾͦͷ͏ͪ20͕݅ؔ࿈ ɾؔ࿈͍ͯ͠Δ40͕݅ώοτ͠ͳ͔ͬͨ 20/30 = 2/3 ɾRecall 20/60 = 1/3
30. CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall ݕࡧΤϯδϯͷྫ ɾݕࡧΤϯδϯͰ30݅ώοτ ɾͦͷ͏ͪ20͕݅ؔ࿈ ɾؔ࿈͍ͯ͠Δ40͕݅ώοτ͠ͳ͔ͬͨ A B C A: ݕࡧ݁Ռ B: શମͷݕࡧཁٻΛຬͨ͢ϖʔδ C: ݕࡧ݁ՌͷதͷݕࡧཁٻΛຬͨ͢ϖʔδ
31. CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall A: ݕࡧ݁Ռ B: શମͷݕࡧཁٻΛຬͨ͢ϖʔδ C: ݕࡧ݁ՌͷதͷݕࡧཁٻΛຬͨ͢ϖʔδ ɾPresision C / A ɾRecall C / B A B C
32. CA.swift CyberaAgent Presents iOS Developer’s Meetup Precision and recall A: ݕࡧ݁Ռ B: શମͷݕࡧཁٻΛຬͨ͢ϖʔδ C: ݕࡧ݁ՌͷதͷݕࡧཁٻΛຬͨ͢ϖʔδ ɾPresision ɾRecall A B C ͲΕ͚ͩແؔ܎ͳ݁ՌΑΓ΋ ؔ࿈ੑͷߴ͍݁ՌΛฦ͔͢ ͲΕ͚ͩΞϧΰϦζϜ͕ؔ࿈ੑ ͷ͋Δ݁ՌΛฦ͔͢
33. CA.swift CyberaAgent Presents iOS Developer’s Meetup The PR Curve Presision Recall 1 1 0 ɾҰํ্͕͕Ε͹ଞํ͕௿Լ͢Δ ɾΫϥεʹΑͬͯݻ༗ͷΧʔϒΛඳ͘
34. CA.swift CyberaAgent Presents iOS Developer’s Meetup MFU
    IBOEMFS
    
    7/*NBHF3FRVFTU)BOEMFS VSM
    JNBHF63-
    MFU
    SFRVFTU
    
    7/$MBTTJGZ*NBHF3FRVFTU
    USZ
    IBOEMFSQFSGPSN <SFRVFTU> 
 MFU
    PCTFSWBUJPOT
    
    SFRVFTUSFTVMUT
    BT
    <7/$MBTTJpDBUJPO0CTFSWBUJPO>
    MFU
    pMUFSFE
    
    PCTFSWBUJPOT pMUFS
    \
    IBT.JOJNVN1SFDJTJPO 
    GPS3FDBMM
    
    ^
    Sample code ݁ՌΛϑΟϧλϦϯά
35. CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering the PR Curve Presision Recall 1 1 0 observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) }
36. CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering the PR Curve Presision Recall 1 1 0 observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) } Precision > 0.4
37. CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering the PR Curve Presision Recall 1 1 0 observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) } Precision > 0.4 Recall = 0.5
38. CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering the PR Curve Presision Recall 1 1 0 observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.9) } Precision > 0.4 Recall = 0.9
39. CA.swift CyberaAgent Presents iOS Developer’s Meetup Filtering the PR Curve Presision Recall 1 1 0 observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.9) } Precision > 0.4 Recall = 0.9 ΞϓϦέʔγϣϯͷ ಛੑʹ͋ͬͨᮢ஋Λઃఆ͢Δ
40. CA.swift CyberaAgent Presents iOS Developer’s Meetup F-measure f − measure = 2 × precision × recall precision + recall F஋͕ߴ͍΄ͲɺόϥϯεΑ͘૒ํͷ஋͕ߴ͍
41. CA.swift CyberaAgent Presents iOS Developer’s Meetup Image classfication summary ɾը૾಺ΦϒδΣΫτΛೝࣝͯ͠ɺΫϥεͰ෼ྨ͞Εͨ݁ՌΛऔಘՄೳ ɾpresision, recallΛཧղ͢Δ ɾΞϓέʔγϣϯಛ༗ͷᮢ஋Λઃఆ͢Δ
42. CA.swift CyberaAgent Presents iOS Developer’s Meetup ɾSaliency ɾImage classfication ɾImage Similarity ɾOther New API
43. CA.swift CyberaAgent Presents iOS Developer’s Meetup Image Similarity ɾ෼ྨωοτϫʔΫ͕ը૾ͷදݱΛ࡞੒ ɾݟͨ໨͚ͩͰ͸ͳ͘ɺը૾ͷத਎Ͱྨࣅ౓Λ൑ఆ ɾFeaturePrintͱ͍͏ई౓͕ྨࣅੑΛࣔ͢
44. CA.swift CyberaAgent Presents iOS Developer’s Meetup FeaturePrint ɾ୯ޠϕΫτϧͷΑ͏ͳը૾σΟεΫϦϓλ ɾ஋͕খ͍͞΄Ͳɺྨࣅੑͷߴ͞Λࣔ͢  
45. CA.swift CyberaAgent Presents iOS Developer’s Meetup MFU
    IBOEMFS
    
    7/*NBHF3FRVFTU)BOEMFS VSM
    JNBHF63-
    MFU
    SFRVFTU
    
    7/(FOFSBUF*NBHF'FBUVSF1SJOU3FRVFTU
    USZ
    IBOEMFSQFSGPSN <SFRVFTU> 
 MFU
    DPOUFTUBOU'10
    
    SFRVFTUSFTVMUTpSTU
    BT
    7/'FBUVSF1SJOU0CTFSWBUJPO
    WBS
    EJTUBODF
    
    'MPBU 
    USZ
    DPOUFTUBOU'10DPNQVUF%JTUBODF EJTUBODF
    UP
    PSJHJOBM'10 Sample code
46. CA.swift CyberaAgent Presents iOS Developer’s Meetup ɾSaliency ɾImage classfication ɾImage Similarity ɾOther New API
47. CA.swift CyberaAgent Presents iOS Developer’s Meetup Other New API ɾFace capture quality ɾFace Landmarks Enhancements ɾHuman detector ɾCat and Dog detector ɾNew Object Tracker ɾVision and CoreML Integration Enhancements
48. CA.swift CyberaAgent Presents iOS Developer’s Meetup References ɾhttps://developer.apple.com/documentation/vision ɾhttps://developer.apple.com/videos/play/wwdc2019/222
49. CA.swift CyberaAgent Presents iOS Developer’s Meetup Thank you CA.swift CyberaAgent Presents iOS Developer’s Meetup