Teaching English Speaking Language Models to Speak Taiwanese Mandarin

An attempt to re-train EN language models to understand and generate fluent Taiwanese Mandarin (Traditional Chinese).

https://github.com/zetavg/twlm

Transcript

1. Teaching 
 English Speaking Language Models to Speak Taiwanese Mandarin Experiments on transfer learning of English- trained language models to understand and generate Taiwanese Mandarin 
 (Traditional Chinese). Pokai Chang 2023/06
2. I’m not a machine learning professional. Everything in this talk should be treated as non-scientific hobbyist content.
3. Why train a new language model?
4. Open source language models are emerging rapidly. Why train a new language model?
5. However, most open source language models can’t understand and generate Traditional Chinese well. Open source language models are emerging rapidly. Why train a new language model?
6. Most open source language models can’t understand and generate Traditional Chinese well. (Screenshot from https://chat.lmsys.org/).
7. Most open source language models can’t understand and generate Traditional Chinese well. (Screenshot from https://chat.lmsys.org/).
8. Most open source language models can’t understand and generate Traditional Chinese well. (Screenshot from https://chat.lmsys.org/). 4UBSUT
   MPPQJOH
9. Most open source language models can’t understand and generate Traditional Chinese well. (Screenshot from 
 https://chat.lmsys.org/).
10. Most open source language models can’t understand and generate Traditional Chinese well. (Screenshot from 
 https://chat.lmsys.org/). *OWBMJE
    HSBNNBS
11. Most open source language models can’t understand and generate Traditional Chinese well. (Screenshot from 
 https://chat.lmsys.org/). *OWBMJE
    HSBNNBS #SPLFO
12. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model?
13. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? (Screenshots from https://chat.lmsys.org/).
14. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? (Screenshots from https://chat.lmsys.org/). 1SPNQU 5SBEJUJPOBM
    $IJOFTF
15. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? (Screenshots from https://chat.lmsys.org/). 1SPNQU 5SBEJUJPOBM
    $IJOFTF 3FQMZ 4JNQMJGJFE
    $IJOFTF
16. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? (Screenshots from https://chat.lmsys.org/).
17. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? (Screenshots from https://chat.lmsys.org/). 1SPNQU 5SBEJUJPOBM
    $IJOFTF
18. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? (Screenshots from https://chat.lmsys.org/). 1SPNQU 5SBEJUJPOBM
    $IJOFTF 3FQMZ 4JNQMJGJFE
    $IJOFTF
19. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? Model: ikala/bloom-zh-3b-chat
20. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? Model: ikala/bloom-zh-3b-chat
21. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? • It make sense since Simpli fi ed Chinese data is more than 2× larger than Traditional Chinese in most of the corpus that is used to train those language models.
22. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? • It make sense since Simpli fi ed Chinese data is more than 2× larger than Traditional Chinese in most of the corpus that is used to train those language models. • For example, BigScience ROOTS Corpus (used to train the BLOOM language model): Simpli fi ed Chinese is 342× larger than Traditional Chinese. Unit: size in bytes. arXiv:2303.03915.
23. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? • It make sense since Simpli fi ed Chinese data is more than 2× larger than Traditional Chinese in most of the corpus that is used to train those language models. • For example, BigScience ROOTS Corpus (used to train the BLOOM language model)... Simpli fi ed Chinese is 342× larger than Traditional Chinese. Unit: size in bytes. arXiv:2303.03915. 100% Simpli fi ed Chinese Traditional Chinese
24. If one can, then it’s very likely to be “polluted” by Simplified Chinese. Most open source models can’t generate Traditional Chinese well. Why train a new language model? • It make sense since Simpli fi ed Chinese data is more than 2× larger than Traditional Chinese in most of the corpus that is used to train those language models. • For example, BigScience ROOTS Corpus (used to train the BLOOM language model)... Simpli fi ed Chinese is 342× larger than Traditional Chinese. Unit: size in bytes. arXiv:2303.03915. 5SBEJUJPOBM
    $IJOFTF
    
25. LLMs are known for their ability to do transfer learning. Why train a new language model?
26. What if we can find a way to let an English language model to learn Traditional Chinese? LLMs are known for their ability to do transfer learning. Why train a new language model?
27. What if we can find a way to let an English language model to learn Traditional Chinese? LLMs are known for their ability to do transfer learning. Why train a new language model? • We can take advantage of the emerging English language models, and turn them into Traditional Chinese language models! 🦙🦙🦙 🦙🦙🦙 LLaMA, Pythia, MPT, … Reproducible process zh-tw LLaMA, zh-tw Pythia, zh-tw MPT, …
28. The Outcome: TW-Pythia-6.9B-Chat A Traditional Chinese & English bilingual model based on pythia-6.9b
29. The Outcome: TW-Pythia-6.9B-Chat A Traditional Chinese & English bilingual model based on pythia-6.9b In the video, the model claims that it was trained by OpenAI. It’s not true. The shown model was trained based on pythia-6.9b. The model says so because it’s trained with ShareGPT data, which includes conversations that the AI introduces itself as ChatGPT trained by OpenAI.
30. Acceptable English ↔ Traditional Chinese Translation Capabilities of TW-Pythia-6.9B-Chat Sometimes it might outperform Google Translate
31. Acceptable English ↔ Traditional Chinese Translation Capabilities of TW-Pythia-6.9B-Chat Sometimes it might outperform Google Translate (PPHMF
    5SBOTMBUF 4FMGUSBJOFE )BSE
    UP
    VOEFSTUBOE #FUUFS 4JNQMJGJFE
    $IJOFTF
    XPSE 5SBEJUJPOBM
    $IJOFTF
    XPSE
32. Acceptable English ↔ Traditional Chinese Translation Capabilities of TW-Pythia-6.9B-Chat
33. Write Fluent Traditional Chinese Capabilities of TW-Pythia-6.9B-Chat （中略）
34. Speaking Funny Non-sense Capabilities of TW-Pythia-6.9B-Chat
35. Speaking Funny Non-sense Capabilities of TW-Pythia-6.9B-Chat
36. The Training Process
37. Why Based on Pythia? The Training Process
38. • Deliberately choose a model with poor Chinese. Why Based on Pythia? The Training Process
39. • Deliberately choose a model with poor Chinese. • Mitigate the impact of Simpli fi ed Chinese. Why Based on Pythia? The Training Process
40. • Deliberately choose a model with poor Chinese. • Mitigate the impact of Simpli fi ed Chinese. • Make the learning results stand out. Why Based on Pythia? The Training Process
41. • Deliberately choose a model with poor Chinese. • Mitigate the impact of Simpli fi ed Chinese. • Make the learning results stand out. • Pythia has various versions with different sizes (70M ~ 12B). • We can do experiments on small models and scale it up to larger ones. Why Based on Pythia? The Training Process
42. Three Major Steps… The Training Process 1 2 3 Expand the Vocabulary Train the Embedding Layers Instruction Tuning Add 8k new Traditional Chinese Tokens fi
43. The Training Process 2 Expand the Vocabulary 1 3
44. The Training Process 2 Expand the Vocabulary 1 3 ˜Ϙ™ kkk
45. The Training Process 2 Expand the Vocabulary 1 3 ˜Ϙ™ kkk • Hard for the model to understand the language.
46. The Training Process 2 Expand the Vocabulary 1 3 ˜Ϙ™ kkk • Hard for the model to understand the language. • Resource consuming - more tokens are needed to process the same amount of text.
47. The Training Process 2 Expand the Vocabulary 1 3 ˜Ϙ™ kkk
48. The Training Process 1 Train the Embedding Layers 2 3 • When we learn a new language, we do not need to re-learn basic logic and reasoning abilities.
49. The Training Process 1 Train the Embedding Layers 2 3 • When we learn a new language, we do not need to re-learn basic logic and reasoning abilities. • It might be the same for language models - we might not need to train all the parameters.
50. The Training Process 1 Train the Embedding Layers 2 3
51. The Training Process 1 Train the Embedding Layers 2 3 • How to know that the model is really learning?
52. • How to know that the model is really learning? • Print the model output during training! ᅼۨٙཫ಻ ͍ᆽഈࣩ
53. • How to know that the model is really learning? • Print the model output during training! ᅼۨٙཫ಻ ͍ᆽഈࣩ
54. • How to know that the model is really learning? • Print the model output during training! ᅼۨٙཫ಻ ͍ᆽഈࣩ
55. • How to know that the model is really learning? • Print the model output during training! ᅼۨٙཫ಻ ͍ᆽഈࣩ
56. • How to know that the model is really learning? • Print the model output during training! ᅼۨٙཫ಻ ͍ᆽഈࣩ
57. • How to know that the model is really learning? • Print the model output during training! ᅼۨٙཫ಻ ͍ᆽഈࣩ
58. • How to know that the model is really learning? • Print the model output during training! ᅼۨٙཫ಻ ͍ᆽഈࣩ
59. • How to know that the model is really learning? • Print the model output during training! ᅼۨٙཫ಻ ͍ᆽഈࣩ 5PLFO
    *%
    
    
    
    OFX
    XPSE
60. The Training Process 1 Instruction Fine-Tuning 3 2 • Nothing special. Just LoRA fi ne-tuning.
61. Training Framework https://github.com/zetavg/twlm
62. https://github.com/zetavg/twlm Training Framework %FGJOF
    IPX
    UP
    CVJME
    UIF
    UPLFOJ[FS %FGJOF
    IPX
    UP
    EP
    UIF
    USBJOJOH
    USBJO
    XIBU
    QBSBNFUFST
    XIJDI
    EBUBTFU
    IZQFSQBSBNFUFST
63. Future Work • A better way to measure how the model is learning by monitoring the embedding of each new token. • Train a complete model and see how it performs. • Apply the training on more open source models. • Take the trained models and have some fun! • 拿政治⼈物接受問答的內容來 train ⼀個 model ⋯⋯