Vision.framework - 商品画像からのテキスト検出と並列化実装への試み

2023.05.22 DeNA×STORES×ラクマ iOS Meetup!!

Transcript

1. ʙ Vision.framework ͷ׆༻ࣄྫ ʙ
   
   
   ঎඼ը૾͔ΒͷςΩετݕग़ͱ
   
   
   ฒྻԽ࣮૷΁ͷࢼΈ
   2023.05.22 DeNA×STORES×ϥΫϚ iOS Meetup!!
   
   Hiromu Tsuruta @ Rakuten Rakuma
   

   View Slide

2. 　　　　　　　　　　　　　　　　　　
   
   　　　　　　　　　　　　　　　　　　
   ・Rakuten Group, Inc
   　　　　　　　　　　
   　　　　　　　　　　・Rakuma
   　　　
   　　　・iOS Developer
   　　　・Developer Relations
   　　　　　　　　　　　　　　　　　
   
   　　　　　　　　　　　　　　　　　　
   ・Twitter：@hcrane14
   

   View Slide

3. Vision.framework ͱ͸ 🤔
   

   View Slide

4. Quote: https://developer.apple.com/documentation/vision
   

   View Slide

5. Apply computer vision algorithms to perform a variety
   of tasks on input images and video.
   
   The Vision framework performs face and face landmark
   detection, text detection, barcode recognition, image
   registration, and general feature tracking. Vision also
   allows the use of custom Core ML models for tasks like
   classi
   fi
   cation or object detection.
   

   View Slide

6. View Slide

7. Vision.framework ͷྺ࢙
   

   View Slide

8. iOS11: Vision.framework ͷొ৔ɻCore ML ͱ૊Έ߹Θͤͯจࣈೝࣝɻ
   
   iOS13: จࣈೝ͕ࣝՄೳʹͳΔɻͨͩ͠ɺӳޠͷΈɻ
   
   iOS14: 7ͭͷݴޠʹରԠ͢Δɻ೔ຊޠ͸ະରԠɻ
   
   iOS16: 14ͭͷݴޠʹରԠ͢Δɻ೔ຊޠʹ΋ରԠɻ
   ※จࣈೝࣝʹؔͯ͠ͷΈ
   

   View Slide

9. Vision.framework ͷ׆༻ྫ
   

   View Slide

10. PayPayϑϦϚ
    Quote:
    fi
    ndy-code.io/events/dtOzzlRgwookF?fr=event_20230208
    

    View Slide

11. ϥΫϚͰͷ׆༻ྫ
    

    View Slide

12. ※·ͩϦϦʔε͸͞Ε͍ͯ·ͤΜ
    

    View Slide

13. Vision.framework ͷ࣮૷
    

    View Slide

14. VNDetectFaceRectanglesRequest
    VNDetectFaceLandmarksRequest
    VNDetectRectanglesRequest
    VNDetectBarcodesRequest
    VNDetectTextRectanglesRequest

    …
    Request
    VNImageRequestHandler

    VNSequenceRequestHandler

    …
    VNFaceObservation

    VNImageAlignmentObservation

    VNRectangleObservation

    VNBarcodeObservation

    …
    RequestHandler Observation
    

    View Slide

15. Example
    ֤ Observation ͷ਌ΫϥεͰ͋Δ VNObservation ͷ഑ྻ͔͠औಘͰ͖ͳ͍
    

    View Slide

16. Helper Extension
    ը૾͔Βೝࣝͨ͠ςΩετͷ݁Ռ ؍ଌͨ͠ςΩετͷީิΛऔಘ
    

    View Slide

17. .topCandidates ͷڍಈ
    

    View Slide

18. Image Model
    Quote: https://www.rakuteneagles.jp/news/detail/202300285435.html
    

    View Slide

19. .topCandidates(1)
    

    View Slide

20. .topCandidates(4)
    

    View Slide

21. .topCandidates(4)
    ߴ
    ௿
    ਫ਼౓
    

    View Slide

22. Vision.framework ͷ
    
    
    ࣮૷ʹ͓͍ͯͷ՝୊ 👀
    

    View Slide

23. ϥΫϚͰ͸
    ࠷େ̍̌ຕͷ
    ը૾Λग़඼Ͱ͖Δ
    ը૾ʹΑͬͯ͸ɺςΩ
    ετ͕ฦͬͯ͘Δͷʹ
    0.5ඵ͔͔ͬͨΓ͢Δ
    ͜ͱ΋͋Δ
    

    View Slide

24. ॲཧΛฒྻԽ͢Δ
    

    View Slide

25. Recognizer
    
    
    TaskGroup
    
    
    Task
    Task
    Finish
    ɾ
    ɾ
    ɾ
    AsyncStream
    Task
    Start
    

    View Slide

26. ※ίʔυ͸Ұ෦Λলུ͍ͯ͠·͢
    Request ͱ RequestHandler Λ AsyncThrowingStream Λ࢖ͬͯɺasync ʹରԠͤ͞Δ
    

    View Slide

27. ฦ͖ͬͯͨ஋Λ AsyncThrowingStream ʹྲྀ͢
    ※ίʔυ͸Ұ෦Λলུ͍ͯ͠·͢
    

    View Slide

28. ※ίʔυ͸Ұ෦Λলུ͍ͯ͠·͢
    TaskGroup ͰॲཧΛฒྻ࣮ߦ͢Δ
    

    View Slide

29. ※ίʔυ͸Ұ෦Λলུ͍ͯ͠·͢
    ݺͼग़͠ݩͷؔ਺͸εοΩϦʂ
    

    View Slide

30. ϦϦʔεָ͓͠Έʹʂ
    

    View Slide

31. Vision.framework ͸͍͍ͧʂʂ
    

    View Slide

32. Fin
    

    View Slide