Small Mislabeling is fine: How LINE catches label noise

Transcript

1. 2019 DevDay Small Mislabeling Is Fine: How LINE Catches Label

   Noise > Jaewook Kang (Marco) > NAVER Chatbot Model

2. What is Label Noise?

3. Dog Dog Dog Dog ? Dog ? Dog ? https://www.freecodecam

   p.org/news/chihuahua- or-muffin-my-search-for- the-best-computer- vision-api- cbda4d6b425d/

4. Dog Dog Dog Dog ? Dog ? Dog ? Clean

   Dog Noisy Dog https://www.freecodecam p.org/news/chihuahua- or-muffin-my-search-for- the-best-computer- vision-api- cbda4d6b425d/

5. “Unintentional” mislabeling which incorrectly describes data in the same category

6. Label Noise is everywhere !!

7. What’s wrong with Label Noise?

8. Impact of Label Noise on Model Training vWithout Label Noise

   Feature extractor Classifier Input X Prediction Score Optimizer “Dog” Sally features 1) Brown Skin 2) Black Eyes Low Loss Label Y Model

9. Impact of Label Noise on Model Training vWith Label Noise

   Feature extractor Classifier Prediction Score Label Y Optimizer “Dog” Sally feature 1) Brown Skin 2) Black Eyes 3) Chocolate balls 4) Raisin 5) Sugar Input X High Loss Noisy dog Model

10. Label noise makes feature extraction of model difficult

11. The difficult feature extraction degrades the model’s performance

12. https://www.trustinsights.ai/blog/2019/08/ 5-ways-your-ai-projects-fail-part-3-data- related-ai-failures/ http://www.ciokorea.com/news/127988 https://medium.com/@kanaugust/ai

13. Then, how can we resolve the Label Noise problem ?

14. Trained Model = Method( Data, Model Structure)

15. Approach 1: Model Structure v Label Noise Robust Model

16. Heavy Model == Heavy Serving + Training Cost Global warming

    is coming

17. Approach 2-1:Curriculum Learning [Y. Bengio: 2009]

18. Approach 2-2: MentorNet [Lu Jiang, et al.:2018],[Bo Han, et al.:

    2019]

19. Heavy training cost + additional data Global warming is coming

20. Approach 3: Data Cleaning Method

21. Active Learning for Label Cleaning [Konyushkova’2017],[Yoo’2019] Noisy Label Data ReLabeled

    Data Noise Label Filtering Model Inference ReLabeling By Human

22. How can we remove Label Noise without human assistance?

23. Our Plan Noisy Label Data ReLabeled Data Model Inference Relabeling

    By Model

24. Today’s Talk How does Line correct the Label noise ?

    - Label-correcting AutoML: PICO - Application to FAQ/Chat dataset

25. Label-correcting AutoML: PICO

26. Idea • Problem: A white duck is mislabeled. Can you

    look around and fix it? Labeled as the Same Class yellow! yellow! yellow! yellow! yellow!

27. Idea • Problem: A white duck is mislabeled. Can you

    look around and fix it? The Mislabeled (Label Noise) The Well- labeled Labeled as the Same Class yellow! yellow! yellow! yellow! yellow!

28. How to let AI do the same task?

29. Split – Train – Check Algorithm

30. Split - Train - Check Algorithm Split: Train Set Valid

    Set Class1-Labeled Class1- Mislabeled Class2-Labeled

31. Split - Train - Check Algorithm Split: • Train set:

    Ref. dataset for correction • Valid set: Checking targets (detaching label from the targets) Train Set Valid Set

32. Split - Train - Check Algorithm Train: • Training the

    checker model + Train Set Training “Checker”

33. Split - Train - Check Algorithm Check: • Checking the

    label of the valid set Valid Set + Trained Checker

34. Split - Train - Check Algorithm • Mislabeling Correction! Mislabeled

    Corrected Yellow? White!

35. Split - Train - Check Algorithm Split Orig. Dataset +

    Check Train Label Update

36. The current method cannot check entire data set

37. Split - Train - Check Algorithm Split Orig. Dataset +

    Check Train Label Update

38. Is there anyway to completely check the entire data ?

39. MultiSplit – Train – Check - Vote

40. MultiSplit: Preparing a variety version of the split branches Vote:

    Collecting the results from each “Split-Train-Check” for label update

41. MultiSplit - Train – Check -Vote Orig. Dataset

42. MultiSplit - Train – Check -Vote MultiSplit Orig. Dataset k

    branches

43. MultiSplit - Train – Check -Vote + MultiSplit Orig. Dataset

    Train k branches

44. MultiSplit - Train – Check -Vote MultiSplit Orig. Dataset +

    + Train k branches

45. MultiSplit - Train – Check -Vote Train MultiSplit Orig. Dataset

    + + + k branches

46. MultiSplit - Train – Check -Vote Train MultiSplit Orig. Dataset

    + + + + … k branches k checkers

47. MultiSplit - Train – Check -Vote Train MultiSplit Orig. Dataset

    + + + + … Check … k branches k checkers Single segment is checked n time

48. MultiSplit - Train – Check -Vote Train MultiSplit Orig. Dataset

    + + + + … Check … Vote & update k branches k checkers Single segment is checked n time

49. MultiSplit - Train – Check -Vote Vote & update HOW?

50. How to Vote ?

51. MultiSplit - Train – Check -Vote Majority Vote:

52. MultiSplit - Train – Check -Vote Algo Majority Vote: •

    The most naïve method • white: yellow = 3 : 0 + + + + + You are white Sure, you are white You m ust be white The checking target

53. Utilize that the checker results are classification-score (probability)

54. PICO: Probabilistic Iterative COrrection + + + + + W:0.7,

    Y: 0.3 흰 :0.85, 노 : 0.15 흰 :0.62, 노 : 0.38

55. PICO’s Three Keys Bayesian Combining of the branch results Thomas

    Bayes 1701-1761 Hidden Markov Modeling of labeling history over iterations Andrey A. Markov 1856–1922 Iterative Probabilistic correction for labeling update Robert G. Gallager 1931-present

56. 0) PICO Vote Starts! + + + + … Trained

    k Checkers

57. 1) Soft-Voting of Branch Results + + + + …

    Likelihood 1 Likelihood 2 Joint Likelihood p(q |C t+1 ) p(q |C t+1 ) ∝ check k (q) k∈ne(q) ∏ Take n related check results from k q Data

58. 2) Labeling Prior Update Joint Likelihood Prev posterior Class transition

    probability + + + + … Likelihood 1 Likelihood 2 Update Prior By Hidden Markov Model p(C t | q) = p(C t |C t−1 )p( C t−1 ∑ C t−1 | q) p(q |C t+1 ) Prior p(C t | q) q Data

59. 3) Compute Posterior by Bayes’ Rule + + + +

    … Likelihood 1 Likelihood 2 Posterior Joint Likelihood p(q |C t+1 ) p(C t+1 | q) Update Posterior By Bayes Rule Prior p(C t | q) p(C t+1 | q) = 1 Z p(C t | q) × check k (q) k∈ne(q) ∏ q Data

60. 4) Label Update Posterior p(C t+1 | q) Update Label

    + + + + … C ! t+1 = argmax p(C t+1 | q) Data p(C t+1 |C t ) ≈ 1 N p(C t+1 |C t ,q) ∀q ∑ Update Class trans. prob.

61. 5) Passing Probability Map to Next Iter. Posterior + +

    + + … Retraining k Checkers Based on Updated Labels C ! t+1 Data PICO is iterative! Update Label Update Class trans prob.

62. 0) Next Iteration Starts! Posterior + + + + …

    PICO is iterative! Trained k Checkers for next iteration Retraining k Checkers Based on Updated Labels Data Update Label Update Class trans prob.

63. Gradually remove label noise through iterative probabilistic voting !

64. All in One!

65. PICO Architecture Branch 0 Data Splitter Train Split-Train-Check branches Branch

    1 branchN Train Train … The original dataset Relabeled dataset PICO is iterative! Checker Checker Checker PICO

66. PICO Implementation Branch 0 Data Splitter Train Split-Train-Check branches Branch

    1 Branch N Train Train … The original dataset Updated dataset PICO is iterative! Checker Checker Checker PICO

67. Our Plan Noisy Label Data ReLabeled Data Model Inference Relabeling

    By Model

68. Our Plan became PICO! Noisy Label Data ReLabeled Data Split-Train-Check

    branches PICO

69. Application to FAQ/Chat dataset!

70. FAQ Chatbot • Providing qualified service by limiting conversation domain

    • Query-Intent Classification methods showing good performance Neural Intent Classifier LINEのパスワ
    ドがわか らないのでどうしたらい いですか？ Word embedding Sentence embedding OR Embedding Input query パスワ
    ド LINEミュ
    ジック ラインスタンプ ラインペイ

71. FAQ chatbot service wrt. LINE use • Appx 90000 data

    / 1000 intent classes • Checker model: classifier-BERT • Characteristic LINE KantanHelp FAQ ü Train “classifier” only for the checkers in PICO ü Imbalance class problem ü Query-intent mapping ambiguity problem ü Some noisy queries

72. LINE KantanHelp FAQ PICO-based Preprocessing 74.45% 2.97% gain Baseline 71.48%

    Rule-based preprocessing 72.43% Top-1 accuracy: Classifier-BERT, for single model ü 5000 validation set 2.02% 0.95%

73. Take Home Message!

74. Small Mislabeling is fine! • AI projects without data strategy

    are difficult to succeed. • AI data automatic cleaning pipeline is very competitive • CLOVA/LINE Chatbot Builder improves service quality by automatically purifying data through PICO • PICO architecture can be applied to other data sets

75. Human-free data cleaning can improve your AI performance

76. Thank you !

77. References • [Bo Han, et al.: 2019] Han, Bo et

    al.,“Co-teaching: Robust training of deep neural networks with extremely noisy labels,” in NIPS, pp. 8536-8546, 2018. • [Konyushkova’2017] K. Konyushkova, et al., “Learning active learning from real and synthetic data,” in NIPS, 2017. • [Lu Jiang, et al.:2018] Jiang, Lu et al., “Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.” in ICML, pp. 2309-2318, 2018. • [Y. Bengio: 2009] Bengio, Yoshua et al, “Curriculum learning,” in ICML, pp. 41–48, 2009 • [Yoo’2019] D. Yoo et al., “Learning loss for Active learning,” in CVPR, 2019