Similarity metrics for Japanese kanji

An overview of my PhD work on calculating the similarity between Japanese kanji, presented to the Melbourne Maths and Science meetup.

Transcript

1. Similarity Metrics for
   Japanese Kanji
   Lars Yencken / 99designs
   Maths and Science Meetup, 30th Nov 2012
   

   View Slide

2. Linguistics
   Computer
   Science
   Computational Linguistics
   

   View Slide

3. Relative difficulty
   of languages
   

   View Slide

4. DIFFICULTY OF LEARNING LANGUAGES
   FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
   

   View Slide

5. DIFFICULTY OF LEARNING LANGUAGES
   FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
   Closely related to English
   ‭‱575-600 class hours
   Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese,
   Romanian, Spanish, Swedish
   

   View Slide

6. DIFFICULTY OF LEARNING LANGUAGES
   FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
   Closely related to English
   ‭‱575-600 class hours
   Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese,
   Romanian, Spanish, Swedish
   Significant linguistic and/or
   cultural differences
   ‭‭‭ 1100 class hours
   Albanian, Amharic, Armenian, Azerbaijani, Bengali, Bosnian, Bulgarian,
   Burmese, Croatian, Czech, Estonian, Finish, Georgian, Greek, Hebrew,
   Hindi, Hungarian, Icelandic, Khmer, Lao, Latvian, Lithuanian,
   Macedonian, Mongolian, Nepali, Pashto, Persian (Dari, Farsi, Tajik),
   Polish, Russian, Serbian, Sinhalese, Slovak, Slovenian, Tagalog, Thai,
   Turkish, Ukranian, Urdu, Uzbek, Vietnamese, Xhosa, Zulu
   

   View Slide

7. DIFFICULTY OF LEARNING LANGUAGES
   FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
   Closely related to English
   ‭‱575-600 class hours
   Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese,
   Romanian, Spanish, Swedish
   Significant linguistic and/or
   cultural differences
   ‭‭‭ 1100 class hours
   Albanian, Amharic, Armenian, Azerbaijani, Bengali, Bosnian, Bulgarian,
   Burmese, Croatian, Czech, Estonian, Finish, Georgian, Greek, Hebrew,
   Hindi, Hungarian, Icelandic, Khmer, Lao, Latvian, Lithuanian,
   Macedonian, Mongolian, Nepali, Pashto, Persian (Dari, Farsi, Tajik),
   Polish, Russian, Serbian, Sinhalese, Slovak, Slovenian, Tagalog, Thai,
   Turkish, Ukranian, Urdu, Uzbek, Vietnamese, Xhosa, Zulu
   

   View Slide

8. DIFFICULTY OF LEARNING LANGUAGES
   FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
   Closely related to English
   ‭‱575-600 class hours
   Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese,
   Romanian, Spanish, Swedish
   Significant linguistic and/or
   cultural differences
   ‭‭‭ 1100 class hours
   Albanian, Amharic, Armenian, Azerbaijani, Bengali, Bosnian, Bulgarian,
   Burmese, Croatian, Czech, Estonian, Finish, Georgian, Greek, Hebrew,
   Hindi, Hungarian, Icelandic, Khmer, Lao, Latvian, Lithuanian,
   Macedonian, Mongolian, Nepali, Pashto, Persian (Dari, Farsi, Tajik),
   Polish, Russian, Serbian, Sinhalese, Slovak, Slovenian, Tagalog, Thai,
   Turkish, Ukranian, Urdu, Uzbek, Vietnamese, Xhosa, Zulu
   

   View Slide

9. DIFFICULTY OF LEARNING LANGUAGES
   FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
   Closely related to English
   ‭‱575-600 class hours
   Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese,
   Romanian, Spanish, Swedish
   Significant linguistic and/or
   cultural differences
   ‭‭‭ 1100 class hours
   Albanian, Amharic, Armenian, Azerbaijani, Bengali, Bosnian, Bulgarian,
   Burmese, Croatian, Czech, Estonian, Finish, Georgian, Greek, Hebrew,
   Hindi, Hungarian, Icelandic, Khmer, Lao, Latvian, Lithuanian,
   Macedonian, Mongolian, Nepali, Pashto, Persian (Dari, Farsi, Tajik),
   Polish, Russian, Serbian, Sinhalese, Slovak, Slovenian, Tagalog, Thai,
   Turkish, Ukranian, Urdu, Uzbek, Vietnamese, Xhosa, Zulu
   

   View Slide

10. DIFFICULTY OF LEARNING LANGUAGES
    FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
    Closely related to English
    ‭‱575-600 class hours
    Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese,
    Romanian, Spanish, Swedish
    Significant linguistic and/or
    cultural differences
    ‭‭‭ 1100 class hours
    Albanian, Amharic, Armenian, Azerbaijani, Bengali, Bosnian, Bulgarian,
    Burmese, Croatian, Czech, Estonian, Finish, Georgian, Greek, Hebrew,
    Hindi, Hungarian, Icelandic, Khmer, Lao, Latvian, Lithuanian,
    Macedonian, Mongolian, Nepali, Pashto, Persian (Dari, Farsi, Tajik),
    Polish, Russian, Serbian, Sinhalese, Slovak, Slovenian, Tagalog, Thai,
    Turkish, Ukranian, Urdu, Uzbek, Vietnamese, Xhosa, Zulu
    

    View Slide

11. DIFFICULTY OF LEARNING LANGUAGES
    FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
    Closely related to English
    ‭‱575-600 class hours
    Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese,
    Romanian, Spanish, Swedish
    Significant linguistic and/or
    cultural differences
    ‭‭‭ 1100 class hours
    Albanian, Amharic, Armenian, Azerbaijani, Bengali, Bosnian, Bulgarian,
    Burmese, Croatian, Czech, Estonian, Finish, Georgian, Greek, Hebrew,
    Hindi, Hungarian, Icelandic, Khmer, Lao, Latvian, Lithuanian,
    Macedonian, Mongolian, Nepali, Pashto, Persian (Dari, Farsi, Tajik),
    Polish, Russian, Serbian, Sinhalese, Slovak, Slovenian, Tagalog, Thai,
    Turkish, Ukranian, Urdu, Uzbek, Vietnamese, Xhosa, Zulu
    

    View Slide

12. DIFFICULTY OF LEARNING LANGUAGES
    FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
    Closely related to English
    ‭‱575-600 class hours
    Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese,
    Romanian, Spanish, Swedish
    Significant linguistic and/or
    cultural differences
    ‭‭‭ 1100 class hours
    Albanian, Amharic, Armenian, Azerbaijani, Bengali, Bosnian, Bulgarian,
    Burmese, Croatian, Czech, Estonian, Finish, Georgian, Greek, Hebrew,
    Hindi, Hungarian, Icelandic, Khmer, Lao, Latvian, Lithuanian,
    Macedonian, Mongolian, Nepali, Pashto, Persian (Dari, Farsi, Tajik),
    Polish, Russian, Serbian, Sinhalese, Slovak, Slovenian, Tagalog, Thai,
    Turkish, Ukranian, Urdu, Uzbek, Vietnamese, Xhosa, Zulu
    

    View Slide

13. DIFFICULTY OF LEARNING LANGUAGES
    FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
    Closely related to English
    ‭‱575-600 class hours
    Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese,
    Romanian, Spanish, Swedish
    Significant linguistic and/or
    cultural differences
    ‭‭‭ 1100 class hours
    Albanian, Amharic, Armenian, Azerbaijani, Bengali, Bosnian, Bulgarian,
    Burmese, Croatian, Czech, Estonian, Finish, Georgian, Greek, Hebrew,
    Hindi, Hungarian, Icelandic, Khmer, Lao, Latvian, Lithuanian,
    Macedonian, Mongolian, Nepali, Pashto, Persian (Dari, Farsi, Tajik),
    Polish, Russian, Serbian, Sinhalese, Slovak, Slovenian, Tagalog, Thai,
    Turkish, Ukranian, Urdu, Uzbek, Vietnamese, Xhosa, Zulu
    Exceptionally difficult for
    native English speakers
    ‭‭‭‭‭‭ 2200 class hours
    Arabic, Cantonese, Mandarin, Japanese, Korean
    

    View Slide

14. DIFFICULTY OF LEARNING LANGUAGES
    FOREIGN SERVICE INSTITUTE, US DEPARTMENT OF STATE
    Closely related to English
    ‭‱575-600 class hours
    Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese,
    Romanian, Spanish, Swedish
    Significant linguistic and/or
    cultural differences
    ‭‭‭ 1100 class hours
    Albanian, Amharic, Armenian, Azerbaijani, Bengali, Bosnian, Bulgarian,
    Burmese, Croatian, Czech, Estonian, Finish, Georgian, Greek, Hebrew,
    Hindi, Hungarian, Icelandic, Khmer, Lao, Latvian, Lithuanian,
    Macedonian, Mongolian, Nepali, Pashto, Persian (Dari, Farsi, Tajik),
    Polish, Russian, Serbian, Sinhalese, Slovak, Slovenian, Tagalog, Thai,
    Turkish, Ukranian, Urdu, Uzbek, Vietnamese, Xhosa, Zulu
    Exceptionally difficult for
    native English speakers
    ‭‭‭‭‭‭ 2200 class hours
    Arabic, Cantonese, Mandarin, Japanese, Korean
    

    View Slide

15. 持
    

    View Slide

16. 持
    /mo(tsu)/ "to carry"
    

    View Slide

17. 持 挂
    拝
    

    View Slide

18. distance(持, 挂) = ???
    

    View Slide

19. The space of kanji
    

    View Slide

20. View Slide

21. dog
    dough
    log
    

    View Slide

22. 持
    挂
    拝
    土
    

    View Slide

23. Approaches
    

    View Slide

24. Compare images
    

    View Slide

25. 持挂
    

    View Slide

26. View Slide

27. Compare components
    

    View Slide

28. 
    
    㣡 ౔ ੇ
    ኱ ౔ ੇ
    

    View Slide

29. Compare strokes
    

    View Slide

30. P R O S P E R I T Y
    P R O P E R T I E S
    

    View Slide

31. P R O S P E R I T Y
    P R O P E R T I E S
    distance: 6
    

    View Slide

32. 
    
    3, 11a, 2a, 2a
    3, 11a, 2a, 2a, 2a
    distance: 1
    

    View Slide

33. Compare trees
    

    View Slide

34. 
    
    
    
    
    
    
    
    
    
    
    
    
    

    View Slide

35. 
    
    
    
    
    
    
    
    
    
    
    
    
    tree edit distance
    

    View Slide

36. So what works?
    

    View Slide

37. View Slide

38. View Slide

39. Thanks!
    

    View Slide