Upgrade to Pro — share decks privately, control downloads, hide ads and more …

無理をしない機械学習プロジェクト2/step_or_not2

 無理をしない機械学習プロジェクト2/step_or_not2

Machine Learning 15minutes! というイベントで登壇したときの資料です。
https://machine-learning15minutes.connpass.com/event/127412/

開発室Graph

May 25, 2019
Tweet

More Decks by 開発室Graph

Other Decks in Technology

Transcript

  1. キーワード抽出はうまくいかない •うまくいかない例 ‣ 「できあがり」があると非手順に • 例: 上に三つ葉を散らしたらできあがり ‣ アドバイスを手順としてしまう •

    ◦◦さんがマヨネーズを足して作ってくれました •正解率 (Accuracy) は51.7% ‣ アノテーターに作業依頼し小さい評価用データを作成 !15
  2. 構成図 !28 ECR Jenkins GitHub Hako (ECS) S3 Redshift Dockerfileを元に


    ビルド Jenkins: CIのための自動化ツール
  3. 構成図 !29 ECR Jenkins GitHub Hako (ECS) S3 Redshift ECRにプッシュ

    EC2 Container Registry
 Dockerコンテナイメージを保存し ているもの
  4. 構成図 !30 ECR Jenkins GitHub Hako (ECS) S3 Redshift ジョブ実行命令

    Kuroko2
 クックパッド製のジョブ
 スケジューラー・管理ツール
  5. 構成図 !31 ECR Jenkins GitHub Hako (ECS) S3 Redshift ECRから


    イメージ取得 Hako (ECS)
 コンテナのデプロイ・管理
 ツール
  6. 構成図 !32 ECR Jenkins GitHub Hako (ECS) S3 Redshift S3から


    モデル取得 Simple Storage Service
 拡張性と堅牢性に優れた
 ストレージサービス
  7. 構成図 !33 ECR Jenkins GitHub Hako (ECS) S3 Redshift 予測結果を


    S3に保存 Redshiftから
 予測用データ取得 Redshift
 大量データを扱える
 データウェアハウス
  8. !49 Step or Not: Discriminator for The Real Instructions in

    User-generated Recipes ▪ cookpad: Most viewed recipe sharing service in the world •Largest recipe sharing service in Japan •Users can upload and search original, user-generated recipes •Most viewed recipe sharing service in the world •Community platform for people to share 
 recipe ideas and cooking tips •Available in 68 countries 22 languages ▪ Step or Not: Automatic step validation •Recipe steps •Main part of recipes (besides title, ingredients) •Include text and/or photos •Used by authors for communication (fake steps) •Fake steps •Some of the “steps” are not actually part of the cooking process. •Advertisements of recipe (e.g., “introduced on TV”) •Comments (e.g., “Thanks for many messages”) •Advice (e.g. “Also good with rice” ) •Arrangements (e.g., “Using as pasta sauce is good” ) •Fake steps cause problems •Spoken by devices such as smart speakers •Recipe search indexing •Task setting •Distinguish between fake steps and the steps actually used for cooking Fake Steps 41% True Steps 59% Example of recipe steps Fake step Monthly users Recipes Japan 55.6 million 2.95 million Global 36.3 million 1.65 million This recipe became hot recipe on Oct. 9, 2010. Thank you for all who cookedˑ Put all seasonings and mix. Type of steps
  9. !50 The 4th Workshop on Noisy User-generated Text, Nov 1,

    2018, Brussels, Belgium (at EMNLP 2018) Method Accuracy Precision Recall Dict 51.2 52.2 53.3 XGB 84.8 87.4 79.5 RForest 89.3 86.1 92.3 LogReg 92.4 91.1 93.1 LSTM 95.1 94.6 95.0 AVG 93.3 93.1 92.9 ▪ LSTM with pre-trained word vectors ▪ Experiments • Data • Collected 20,000 distinct steps from cookpad • Manually annotated by human • Pos / Neg rate is balanced • Only using text information • Without position, timestamp, etc. • Methods • Dict: Manually selected clue words • Non-neural models (Input: TF-IDF vector) • LogReg: LogisticRegression • RForest: RandomForest • XGB: XGBoost • Neural models(Input: pre-trained skip-gram) • LSTM: LSTM • AVG: Average of word vectors •Distinguish between fake steps and the steps actually used for cooking Word Embedding LSTM # # # # # # # Dropout # # # # # # # LeakyReLu Sigmoid Model Architecture • Preprocessing • Split sentences into words • Input: “ௐຯྉΛશͯೖΕͯࠞͥ߹Θͤ·͢” • Ouput: “ௐຯྉ”, “Λ”, “શͯ”, “ೖΕ”, “ͯ”, “ࠞ ͥ߹Θ”, “ͤ”, “·͢” • Not apply normalization • e.g. “ೖΕ” → “ೖΕΔ” • Word Embedding • Skip-Gram Word2vec • Pre-trained with our whole recipe steps dataset (over 20 million steps) • (218408, 100) vector • Results and Discussion • Over 95% accuracy in neural models • Non-neural models are easy to deploy but insufficient for our service • LSTM-based model misclassifies • Steps that include special words such as “finished”, “completed”, etc. • Steps that are advice including cooking vocabularies • Future Work • Get higher score • Not only textual features • Deploy and apply in our services cookedˑ • False Positive • JP: “ίʔώʔʹೖΕͯɾɾɾ؁͓͍͍ͯ͘͠ ϛϧΫίʔώʔͷग़དྷ্Γ<unk>” • EN: “Put in Coffee. You will get sweet delicious milk coffee. That’s all.” • False Negative • JP: “'ˎΦʔϒϯͷ৔߹̎̏̌℃ʹ༧೤ͨ͠Φʔ ϒϯͰ̍̑෼͘Β͍ম͘ɻ'” • EN: “Bake for about 15 minutes in an oven preheated to 230℃.” Example of results