顔写真メイクアップアプリの開発におけるプライバシー保護とコスト削減のための手法

iOSDC Japan 2023

https://fortee.jp/iosdc-japan-2023/proposal/6aa80fd7-ec73-4261-8ea0-563cbfa52df5

Transcript

1. akkey / AKIO ITAYA
   ϓϥΠόγʔอޢͱίετ࡟ݮͷͨΊͷख๏
   AppBrew, Inc.
   إࣸਅϝΠΫΞοϓΞϓϦͷ։ൃʹ͓͚Δ
   

   View Slide

2. Խহ඼ߪೖͷ͝ܦݧ͸ʁ
   

   View Slide

3. Խহ඼ͷ৭બͼ͸೉͍͠
   ஶ࡞ऀɿFreepik

   https://jp.freepik.com/free-photo/red-lipsticks-arrangement-
   fl
   at-lay_14604143.htm
   

   View Slide

4. όʔνϟϧϝΠΫʹ௅ઓ
   ஶ࡞ऀɿFreepik

   https://jp.freepik.com/free-photo/medium-shot-girl-taking-sel
   fi
   e_42085215.htm
   ʮόʔνϟϧϝΠΫςελʔʯ͸גࣜձࣾࢿੜಊͷొ࿥঎ඪͰ͢
   

   View Slide

5. ৭Λ
   બ୒
   όʔνϟϧϝΠΫʹ௅ઓ
   

   View Slide

6. ৭Λ
   બ୒
   📸
   ࡱӨ
   όʔνϟϧϝΠΫʹ௅ઓ
   

   View Slide

7. ৭Λ
   બ୒
   📸
   ࡱӨ
   إͷ
   ෼ੳ
   όʔνϟϧϝΠΫʹ௅ઓ
   

   View Slide

8. ৭Λ
   બ୒
   📸
   ࡱӨ
   إͷ
   ෼ੳ
   🔨
   ฤू
   όʔνϟϧϝΠΫʹ௅ઓ
   

   View Slide

9. ৭Λ
   બ୒
   📸
   ࡱӨ
   إͷ
   ෼ੳ
   🔨
   ฤू
   όʔνϟϧϝΠΫʹ௅ઓ
   

   View Slide

10. ৭Λ
    બ୒
    📸
    ࡱӨ
    إͷ
    ෼ੳ
    🔨
    ฤू
    όʔνϟϧϝΠΫʹ௅ઓ
    ϩ
    ʔ
    Χ
    ϧ
    ׬
    ݁
    

    View Slide

11. ݁࿦
    04
    

    View Slide

12. ৭Λ
    બ୒
    📸
    ࡱӨ
    إͷ
    ෼ੳ
    🔨
    ฤू
    ϩ
    ʔ
    Χ
    ϧ
    ׬
    ݁
    إͷ෼ੳɾՃ޻Λ୺຤಺Ͱߦ͏
    ϓϥΠόγʔอޢ
    

    View Slide

13. ৭Λ
    બ୒
    📸
    ࡱӨ
    إͷ
    ෼ੳ
    🔨
    ฤू
    ϩ
    ʔ
    Χ
    ϧ
    ׬
    ݁
    إͷ෼ੳɾՃ޻Λ୺຤಺Ͱߦ͏
    ίετ࡟ݮ
    

    View Slide

14. ϓϥΠόγʔอޢͱίετ࡟ݮͷͨΊͷख๏
    إࣸਅϝΠΫΞοϓΞϓϦͷ։ൃʹ͓͚Δ
    

    View Slide

15. ։ൃ
    

    View Slide

16. ৭Λ
    બ୒
    📸
    ࡱӨ
    إͷ
    ෼ੳ
    🔨
    ฤू
    إͷ෼ੳ
    

    View Slide

17. Vision Framework
    Apply computer vision algorithms to perform a variety of tasks on input images and video.
    https://developer.apple.com/documentation/vision
    

    View Slide

18. ࣄલ४උ
    

    View Slide

19. Core ML models
    https://github.com/john-rocky/CoreML-Models
    

    View Slide

20. https://github.com/john-rocky/CoreML-Models#face-parsing
    

    View Slide

21. ϓϩδΣΫτʹऔΓࠐΉ
    

    View Slide

22. ϓϩδΣΫτʹऔΓࠐΉ
    

    View Slide

23. ϓϩδΣΫτʹऔΓࠐΉ
    final class ViewModel {
    
    
    typealias FaceParsing = face_parsing
    
    
    init() {
    
    
    let mlModel = try! FaceParsing().model
    
    
    }
    

    }
    

    View Slide

24. Core ML Helpers
    https://github.com/hollance/CoreMLHelpers
    

    View Slide

25. ϓϩδΣΫτʹऔΓࠐΉ
    

    View Slide

26. ίʔσΟϯά
    08
    

    View Slide

27. إͷ෼ੳ
    final class ViewModel {
    
    
    init() {
    
    
    let mlModel = try! FaceParsing().model
    
    
    }
    

    }
    MLModel
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    

    View Slide

28. إͷ෼ੳ
    final class ViewModel {
    
    
    init() {
    
    
    let mlModel = try! FaceParsing().model
    
    
    let coreMLModel = try! VNCoreMLModel(for: mlModel)
    
    
    }
    

    }
    MLModel VNCoreMLModel
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    

    View Slide

29. إͷ෼ੳ
    final class ViewModel {
    
    
    init() {
    
    
    let mlModel = try! FaceParsing().model
    
    
    let coreMLModel = try! VNCoreMLModel(for: mlModel)
    
    
    let request = VNCoreMLRequest(model: coreMLModel)
    
    
    request.imageCropAndScaleOption = .scaleFill
    
    
    }
    

    }
    MLModel VNCoreMLModel VNCoreMLRequest
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    

    View Slide

30. إͷ෼ੳ
    final class ViewModel {
    
    
    private let faceImage = CIImage(image: UIImage(named: “face")!)!
    
    
    init() {
    

    …
    
    
    }
    

    }
    CIImage
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    

    View Slide

31. إͷ෼ੳ
    final class ViewModel {
    
    
    private let faceImage = CIImage(image: UIImage(named: “face")!)!
    
    
    init() {
    

    …
    
    
    let requestHandler = VNImageRequestHandler(ciImage: faceImage)
    
    
    }
    

    }
    CIImage VNImageRequestHandler
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    

    View Slide

32. إͷ෼ੳ
    final class ViewModel {
    
    
    private let faceImage = CIImage(image: UIImage(named: “face")!)!
    
    
    init() {
    

    …
    
    
    let requestHandler = VNImageRequestHandler(ciImage: faceImage)
    
    
    try! requestHandler.perform([request])
    
    
    }
    

    }
    CIImage VNImageRequestHandler
    VNCoreMLRequest
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    

    View Slide

33. VNCoreMLRequest
    إͷ෼ੳ
    final class ViewModel {
    
    
    private let faceImage = CIImage(image: UIImage(named: "face")!)!
    
    
    init() {
    
    
    …
    
    
    let requestHandler = VNImageRequestHandler(ciImage: faceImage)
    
    
    try! requestHandler.perform([request])
    
    
    let result = request.results!.first as! VNCoreMLFeatureValueObservation
    
    
    }
    

    }
    VNCoreMLFeatureValueObservation
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    

    View Slide

34. إͷ෼ੳ
    final class ViewModel {
    
    
    private let faceImage = CIImage(image: UIImage(named: "face")!)!
    
    
    init() {
    
    
    …
    
    
    let requestHandler = VNImageRequestHandler(ciImage: faceImage)
    
    
    try! requestHandler.perform([request])
    
    
    let result = request.results!.first as! VNCoreMLFeatureValueObservation
    
    
    let multiArray = result.featureValue.multiArrayValue!
    
    
    }
    

    }
    MLMultiArray
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    VNCoreMLRequest VNCoreMLFeatureValueObservation
    

    View Slide

35. إͷ෼ੳ
    final class ViewModel {
    
    
    private let multiArray: MLMultiArray
    
    
    private let faceImage = CIImage(image: UIImage(named: "face")!)!
    
    
    init() {
    
    
    …
    
    
    let requestHandler = VNImageRequestHandler(ciImage: faceImage)
    
    
    try! requestHandler.perform([request])
    
    
    let result = request.results!.first as! VNCoreMLFeatureValueObservation
    
    
    multiArray = result.featureValue.multiArrayValue!
    
    
    }
    

    }
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    MLMultiArray
    VNCoreMLRequest VNCoreMLFeatureValueObservation
    

    View Slide

36. ৭Λ
    બ୒
    📸
    ࡱӨ
    إͷ
    ෼ੳ
    🔨
    ฤू
    🔨 ฤू
    10
    

    View Slide

37. Ϧοϓͷදݱ💄
    

    View Slide

38. Ϧοϓͷදݱ
    Normal
    

    View Slide

39. Ϧοϓͷදݱ
    Normal
    Overlay
    

    View Slide

40. Ϧοϓͷදݱ
    Normal
    Overlay
    Color Change
    

    View Slide

41. Ϧοϓͷදݱ
    Color Change
    Overlay
    Real
    

    View Slide

42. ੒෼Ͱߟ͑ͯΈΔ
    

    View Slide

43. إྉͱછྉ
    https://www.toyo-color.com/ja/products/organic_pigments/about.html
    إྉ છྉ
    

    View Slide

44. إྉͱછྉ
    https://www.toyo-color.com/ja/products/organic_pigments/about.html
    إྉ છྉ
    

    View Slide

45. https://lipscosme.com/products/337039
    

    View Slide

46. https://lipscosme.com/products/337039ʛhttps://ameblo.jp/rik01194/entry-12309635144.html
    إྉ છྉ
    

    View Slide

47. ৶ͷ৭Λิਖ਼͢Δͱྑͦ͞͏ʁ
    ※ ঺հࣄྫ͸͋͘·Ͱ΋ҰྫͰ͢
    

    View Slide

48. إྉͱછྉ
    https://www.toyo-color.com/ja/products/organic_pigments/about.html
    إྉ છྉ
    

    View Slide

49. CIColorCrossPolynomial
    The properties you use to con
    fi
    gure a color cross-polynomial
    fi
    lter
    https://developer.apple.com/documentation/coreimage/cicolorcrosspolynomial
    14
    

    View Slide

50. CIColorCrossPolynomial
    https://note.com/yogox/n/n7aa72d268144#aXmbf
    ɹ: ܎਺

    ɹ: ೖྗըૉͷ੺੒෼஋

    ɹ: ೖྗըૉͷ྘੒෼஋

    ɹ: ೖྗըૉͷ੨੒෼஋
    f(r,g,b)=k_0r+k_1g+k_2b+k_3r^2+k_4g^2+k_5b^2+k_6rg+k_7gb+k_8br+k_9

    https://sundryst.com/convenienttool_math/drawexpression.html
    ग़ྗըૉͷ֤৭੒෼஋
    

    View Slide

51. CIColorCrossPolynomial
    let redArray: [CGFloat] = [2, 2, 2, 0, 0, 0, 0, 0, 0, 0.2]
    

    View Slide

52. CIColorCrossPolynomial
    let redArray: [CGFloat] = [2, 2, 2, 0, 0, 0, 0, 0, 0, 0.2]
    
    
    let redVector = CIVector(values: redArray, count: redArray.count)
    

    View Slide

53. CIColorCrossPolynomial
    let redArray: [CGFloat] = [2, 2, 2, 0, 0, 0, 0, 0, 0, 0.2]
    
    
    let redVector = CIVector(values: redArray, count: redArray.count)
    
    
    let editedCIImage = CIFilter(
    
    
    name: "CIColorCrossPolynomial",
    
    
    parameters: ["inputRedCoefficients": redVector]
    
    
    )!.outputImage!
    
    
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    

    View Slide

54. CIColorCrossPolynomial
    let redArray: [CGFloat] = [2, 2, 2, 0, 0, 0, 0, 0, 0, 0.2]
    
    
    let redVector = CIVector(values: redArray, count: redArray.count)
    
    
    

    let image = CIImage(image: UIImage(named: "color_test")!)!
    
    
    let editedCIImage = CIFilter(
    
    
    name: "CIColorCrossPolynomial",
    
    
    parameters: [
    
    
    kCIInputImageKey: ciImage,
    
    
    "inputRedCoefficients": redVector
    
    
    ]
    
    
    )!.outputImage!
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    

    View Slide

55. CIColorCrossPolynomial
    let redArray: [CGFloat] = [2, 2, 2, 0, 0, 0, 0, 0, 0, 0.2]
    
    
    let redVector = CIVector(values: redArray, count: redArray.count)
    
    
    

    let image = CIImage(image: UIImage(named: "color_test")!)!
    
    
    let editedCIImage = CIFilter(
    
    
    name: "CIColorCrossPolynomial",
    
    
    parameters: [
    
    
    kCIInputImageKey: ciImage,
    
    
    "inputRedCoefficients": redVector
    
    
    ]
    
    
    )!.outputImage!
    
    
    let context = CIContext()
    
    
    let cgImage = context.createCGImage(editedCIImage, from: editedCIImage.extent)!
    
    
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    

    View Slide

56. όʔνϟϧϝΠΫʹ௅ઓ
    ৭Λ
    બ୒
    📸
    ࡱӨ
    إͷ
    ෼ੳ
    🔨
    ฤू
    17
    

    View Slide

57. let editedCIImage = CIFilter(name: "CIColorCrossPolynomial",…)!.outputImage!
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    όʔνϟϧϝΠΫʹ௅ઓ
    

    View Slide

58. ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    όʔνϟϧϝΠΫʹ௅ઓ
    let editedCIImage = CIFilter(name: "CIColorCrossPolynomial",…)!.outputImage!
    

    

    let upperLipCGImage = multiArray.cgImage(…, outputType: FaceType.upperLip)!
    

    View Slide

59. let editedCIImage = CIFilter(name: "CIColorCrossPolynomial",…)!.outputImage!
    

    

    let upperLipCGImage = multiArray.cgImage(…, outputType: FaceType.upperLip)!
    

    let upperLipCIImage = CIImage(cgImage: …).resize(as: faceImage.extent.size)
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    όʔνϟϧϝΠΫʹ௅ઓ
    

    View Slide

60. let editedCIImage = CIFilter(name: "CIColorCrossPolynomial",…)!.outputImage!
    

    

    let upperLipCGImage = multiArray.cgImage(…, outputType: FaceType.upperLip)!
    

    let upperLipCIImage = CIImage(cgImage: …).resize(as: faceImage.extent.size)
    

    

    let compositedUpperLipImage = CIFilter(
    
    
    name: "CIBlendWithMask",
    
    
    parameters: [
    
    
    kCIInputImageKey: editedCIImage,
    
    
    kCIInputBackgroundImageKey: faceImage,
    
    
    kCIInputMaskImageKey: upperLipCIImage
    
    
    ]
    
    
    )!.outputImage!
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    όʔνϟϧϝΠΫʹ௅ઓ
    

    View Slide

61. let editedCIImage = CIFilter(name: "CIColorCrossPolynomial",…)!.outputImage!
    

    

    let upperLipCGImage = multiArray.cgImage(…, outputType: FaceType.upperLip)!
    

    let upperLipCIImage = CIImage(cgImage: …).resize(as: faceImage.extent.size)
    

    

    let compositedUpperLipImage = CIFilter(
    
    
    name: "CIBlendWithMask",
    
    
    parameters: [
    
    
    kCIInputImageKey: editedCIImage,
    
    
    kCIInputBackgroundImageKey: faceImage,
    
    
    kCIInputMaskImageKey: upperLipCIImage
    
    
    ]
    
    
    )!.outputImage!
    

    

    let context = CIContext()
    
    
    let cgImage = context.createCGImage(
    

    compositedUpperLipImage,
    

    from: compositedUpperLipImage.extent
    

    )!
    ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
    όʔνϟϧϝΠΫʹ௅ઓ
    

    View Slide

62. Sample
    https://github.com/AkkeyLab/vision-makeup
    

    View Slide

63. ࣗݾ঺հ
    

    View Slide

64. akkey
    ஥ؒืूத डߨੜืूத
    

    View Slide

65. Thank you !!
    

    View Slide