Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CoreML3のオンデバイストレーニングでつくる母音推定

naru-jpn
January 25, 2020

 CoreML3のオンデバイストレーニングでつくる母音推定

naru-jpn

January 25, 2020
Tweet

More Decks by naru-jpn

Other Decks in Technology

Transcript

  1. CoreML3ͷΦϯσόΠετϨʔχϯάͰͭ͘Δ฼Իਪఆ
    Inference of vowel by on-device training of CoreML3
    Naruki Chigira @ Mirrativ, inc.
    _naru_jpn

    View full-size slide

  2. σϞͰ࢖͏ΞϓϦͷίʔυ
    https://github.com/naru-jpn/InferenceVowel

    View full-size slide

  3. ฼Իਪఆ
    /a/, /i/, /u/, /e/, /o/ ͷਪఆ

    View full-size slide

  4. ΦϯσόΠετϨʔχϯά
    https://developer.apple.com/jp/machine-learning/core-ml/
    …ϢʔβʔσʔλΛ࢖༻ͨ͠ϞσϧͷΞοϓσʔτΛΦϯσό
    ΠεͰ࣮ߦͰ͖ΔΑ͏ʹͳͬͨͨΊɺϓϥΠόγʔΛ৵֐͢Δ
    ͜ͱͳ͘ɺϢʔβʔͷߦಈʹ߹ΘͤͯϞσϧΛΞοϓσʔτ͠
    ͍ͯ͘͜ͱ͕Ͱ͖·͢ɻ

    View full-size slide

  5. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio

    View full-size slide

  6. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio
    let captureSession = AVCaptureSession()

    captureSession.startSession()

    View full-size slide

  7. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio
    ϑʔϦΤม׵
    Fourier Transformation
    ϝϧϑΟϧλόϯΫ
    Mel Filter Bank
    ཭ࢄίαΠϯม׵
    Discrete Cosine Transformation

    View full-size slide

  8. ϑʔϦΤม׵
    Fourier Transformation
    (ϩά)ϝϧϑΟϧλόϯΫ
    (Log) Mel Filter Bank
    ཭ࢄίαΠϯม׵
    Discrete Cosine Transformation
    Import Accelerate
    // ԻΛप೾਺͝ͱʹ෼ղ
    vDSP_fft_zrip(fftsetup, …
    // ਓ͕ฉ͖෼͚΍͍͢௿ԻΛΑΓৄࡉʹ
    vDSP_mmul(melFilerBank, …
    // ΑΓগͳ͍σʔλ਺Ͱදݱ
    vDSP_DCT_Execute(dctsetup, …

    View full-size slide

  9. ϑʔϦΤม׵
    Fourier Transformation
    (ϩά)ϝϧϑΟϧλόϯΫ
    (Log) Mel Filter Bank
    ཭ࢄίαΠϯม׵
    Discrete Cosine Transformation
    Import Accelerate
    // ԻΛप೾਺͝ͱʹ෼ղ
    vDSP_fft_zrip(fftsetup, …
    // ਓ͕ฉ͖෼͚΍͍͢௿ԻΛΑΓৄࡉʹ
    vDSP_mmul(melFilerBank, …
    // ΑΓগͳ͍σʔλ਺Ͱදݱ
    vDSP_DCT_Execute(dctsetup, …
    1024ݸͷσʔλ
    16ݸͷσʔλ
    ~25μs

    View full-size slide

  10. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio
    https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb
    apple/coremltools
    updatable_nearest_neighbor_classifier.ipynb
    Jupyter notebook ্Ͱkۙ๣๏Λར༻ͯ͠෼
    ྨΛ͢ΔϞσϧΛ࡞੒͢Δαϯϓϧ

    View full-size slide

  11. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio
    https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb
    // ֶशʹ࢖͏σʔλͷ४උ
    let inputValue = snapshot.featureValue
    let outputValue = MLFeatureValue(
    string: snapshot.vowel.identifier
    )
    let dataPointFeatures: [String: MLFeatureValue] = [
    "input": inputValue,
    "label": outputValue
    ]
    let provider = try! MLDictionaryFeatureProvider(
    dictionary: dataPointFeatures
    )

    View full-size slide

  12. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio
    https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb
    // ֶशͷ࣮ߦ
    let batchProvider = MLArrayBatchProvider(
    array: featureProviders
    )
    let updateTask = try! MLUpdateTask(
    forModelAt: UpdatableKNN.urlOfModelInThisBundle,
    trainingData: batchProvider,
    configuration: nil,
    completionHandler: completion
    )
    updateTask.resume()

    View full-size slide

  13. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio
    https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb
    // ਪ࿦ͷ࣮ߦ
    let inputs = snapshots.compactMap({
    $0.featureValue.multiArrayValue
    }).map({
    UpdatableKNNInput(input: $0)
    })
    let model = try! UpdatableKNN(contentsOf: updatedModelUrl)
    // ฼Ի͝ͱͷ֬཰͕ग़ྗͱͯ͠ฦͬͯ͘Δ
    let outputs = try model.predictions(inputs: inputs)
    100~200μs

    View full-size slide