Suffix Trees and Suffix Arrays

Transcript

1. Suffix Trees and Suffix Arrays M3 Tech Talk Hirohito Sasakawa,

   Data engineer, AI and ML team @M3, inc.

2. Suffix Tree ͱ Suffix Array • จࣈྻʹର͢ΔࡧҾσʔλߏ଄ • ࡧҾ (index):

   
 ݩσʔλʹର͢ΔΞΫηεΛޮ཰Α͘ఏڙ͢Δิॿσʔλߏ଄ • Suffix (Tree | Array) ͸ɼจࣈྻ ( or จࣈྻू߹) ʹରͯ͠෦෼จࣈྻ ͷݕࡧɼස౓ɼ࠷௕ڞ௨෦෼จࣈྻͳͲΛߴ଎ʹܭࢉ͢Δ • ࣮༻తͳԠ༻͕ɼΊͪΌͪ͘Ό͋Δ (ѹॖͱ͔ɼόΠΦܥͱ͔)

3. ͪͳΈʹ… • Goͷඪ४ϥΠϒϥϦʹSuffix Arrayؚ͕·Ε͍ͯΔ • ݕࡧ෦෼͸400ߦऑͰγϯϓϧ • GoͱΞϧΰϦζϜͷษڧʹ
 ྑͦ͞͏ ߏங෦෼͸1000ߦఔ౓Ͱɼߴ଎ͳ


   ΞϧΰϦζϜ (SA-IS) ͕࣮૷͞Ε͍ͯΔ
 ࿦จยखʹΏͬ͘ΓಡΉͷ͕ྑͦ͞͏

4. ༻ޠͷ४උ • Suffix (઀ඌࣙ): จࣈྻTʹରͯ͠ɼઌ಄0จࣈҎ্Λ࡟ͬͨจࣈྻ
 ྫ: T = ababb
 ઀ඌࣙ:

   ababb, babb, abb, bb, b, ‘’ (ۭจࣈ)

5. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦

6. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa

7. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

   ocoa coa oa a ઀ඌࣙͨͪ

8. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

   ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ

9. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

   ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ

10. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ

11. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a

12. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co

13. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a

14. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa

15. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o

16. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a

17. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a coa

18. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a coa Suffix Tree

19. Suffix Tree • ߏஙʹ͔͔Δ࣌ؒ 
 ී௨ʹ΍ΔͱO(n^2) ࣌ؒ
 Ukkonen’s Algorithm ͩͱ


    Online࡞੒ՄೳͰ O(n) ࣌ؒ a co a coa o a coa • ϝϞϦ: O(n)
 O(n^2) ʹͳΒͳ͍͜ͱʹ஫ҙ
 ݩͷจࣈྻʹର͢Δ࢝఺ɼऴ఺ΛϙΠϯλͰอ࣋͢Ε͹Α͍

20. Suffix TreeΛ༻͍ͨύλʔϯͷݕࡧ • rootϊʔυ͔ΒࢬΛબΜͰਐΉ͚ͩ • ౸ୡͨ͠ϊʔυͷ࣍਺ؚ͕·ΕΔස౓ʹͳΔ • ݕࡧͷܭࢉྔ: O(m)
 ͜͜Ͱm͸ύλʔϯ௕


    ˠ ͭ·ΓݩσʔλΛશ෦ᢞΊͳ͍ (΍ͬͨͶ) a co a coa o a coa

21. Ԡ༻: Longest Common Substrings (LCS) • ೖྗ: 2ͭҎ্ͷจࣈྻू߹S 
 ग़ྗ:

    Sʹڞ௨ͯ͠ݱΕΔ෦෼จࣈྻͷ͏ͪ࠷௕ͷ΋ͷ • Sʹରͯ͠Suffix TreeΛߏங͠ɼ
 root͔ΒḷΕΔϊʔυͷ͏ͪɼ྆ऀΛؚΉ
 ࠷௕ͷ෦෼Λฦͤ͹ྑ͍ $ana a $ana na banana$ana na S = { s1 = banana, s2 = ana } $ana na$ana $ana na$ana s2 s2 s2 s1, s2 s1, s2 s2 s2 s2

22. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦

23. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa

24. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ

25. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ

26. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ

27. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ

28. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a

29. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co

30. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a

31. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa

32. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o

33. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a

34. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a coa

35. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa

    ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a coa Suffix Tree

36. Suffix Array • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛιʔτͨ࣌͠ͷݩͷจࣈྻͷindexͷ഑ྻ

37. Suffix Array • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛιʔτͨ࣌͠ͷݩͷจࣈྻͷindexͷ഑ྻ T = cocoa

38. Suffix Array • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛιʔτͨ࣌͠ͷݩͷจࣈྻͷindexͷ഑ྻ T = cocoa ઀ඌࣙs

    
 with index cocoa ocoa coa oa a 
    
    
    
    

39. Suffix Array • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛιʔτͨ࣌͠ͷݩͷจࣈྻͷindexͷ഑ྻ T = cocoa ઀ඌࣙs

    
 with index cocoa ocoa coa oa a 
    
    
    
     
    
    
    
     a coa cocoa oa ocoa ιʔτ

40. Suffix Array • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛιʔτͨ࣌͠ͷݩͷจࣈྻͷindexͷ഑ྻ T = cocoa ઀ඌࣙs

    
 with index cocoa ocoa coa oa a 
    
    
    
     
    
    
    
     a coa cocoa oa ocoa ιʔτ 
    
    
    
     Suffix Array

41. Suffix Array • ߏஙʹ͔͔Δ࣌ؒ
 ී௨ʹ΍Δͱ O(n^2 logn) ࣌ؒ
 SA-ISΞϧΰϦζϜʹΑΓ O(n)

    ࣌ؒͰߏஙͰ͖Δ 
 (ιʔτ͢Δͷʹlogn͕͔ͭͳ͍ෆࢥٞͳํ๏) • ϝϞϦ͸ࣗ໌ʹO(n) 
    
    
    
     a coa cocoa oa ocoa

42. Suffix ArrayΛ༻͍ͨύλʔϯͷݕࡧ • Suffix Array্Λύλʔϯ௕͚ͩ܁Γฦ͠ೋ෼୳ࡧ͢Δ
 O(m logn) ࣌ؒ (஗͍) •

    LCP Array (࠷௕ڞ௨઀಄ࣙ഑ྻ) Λ
 ิॿσʔλߏ଄ͱͯ͠࢖ͬͯ
 O(m + log n) ࣌ؒΛୡ੒Ͱ͖Δ • LCP Array͸Suffix Array͔ΒO(n)࣌ؒͰߏஙՄೳ 
    
    
    
     a coa cocoa oa ocoa

43. Suffix Tree ͱ Suffix Array ͷؔ܎ • ྺ࢙తʹ͸ Suffix Tree

    ͷํ͕20೥Ҏ্ૣ͘ൃݟ͞Εͨ • όΠΦͷจ຺ͰϝϞϦফඅΛམͱ͍ͨ͠Ϟνϕʔγϣϯ͔Β
 Suffix Array ͕։ൃ͞Εͨ • ͷͪʹ LCP Array ͕։ൃ͞ΕɼSuffix Tree্ͷΞϧΰϦζϜͷଟ͘ ΛSuffix Array্Ͱ࣮ߦͰ͖ΔΑ͏ʹͳͬͨ • ST, SA྆ํͱ΋ɼݸผʹൃలͨ͠ΞϧΰϦζϜ͕ଟ਺։ൃ͞Ε͍ͯΔ

44. ·ͱΊ • Suffix Tree ͱ Suffix Array ʹ͍ͭͯ঺հͨ͠ • ߏங࣌ؒ

    O(n)ɼϝϞϦO(n) • ݕࡧ΍ɼ࠷௕ڞ௨෦෼จࣈྻͳͲΛߴ଎ʹղ͘͜ͱ͕Ͱ͖Δ • จࣈྻָ͍͠Αʂ