Suffix Trees and Suffix Arrays

M3 Tech Talk #150

Quick introduction of suffix trees and arrays data structures

Transcript

1. Suffix Trees and Suffix Arrays M3 Tech Talk Hirohito Sasakawa, Data engineer, AI and ML team @M3, inc.
2. Suffix Tree ͱ Suffix Array • จࣈྻʹର͢ΔࡧҾσʔλߏ଄ • ࡧҾ (index): 
 ݩσʔλʹର͢ΔΞΫηεΛޮ཰Α͘ఏڙ͢Δิॿσʔλߏ଄ • Suffix (Tree | Array) ͸ɼจࣈྻ ( or จࣈྻू߹) ʹରͯ͠෦෼จࣈྻ ͷݕࡧɼස౓ɼ࠷௕ڞ௨෦෼จࣈྻͳͲΛߴ଎ʹܭࢉ͢Δ • ࣮༻తͳԠ༻͕ɼΊͪΌͪ͘Ό͋Δ (ѹॖͱ͔ɼόΠΦܥͱ͔)
3. ͪͳΈʹ… • Goͷඪ४ϥΠϒϥϦʹSuffix Arrayؚ͕·Ε͍ͯΔ • ݕࡧ෦෼͸400ߦऑͰγϯϓϧ • GoͱΞϧΰϦζϜͷษڧʹ
 ྑͦ͞͏ ߏங෦෼͸1000ߦఔ౓Ͱɼߴ଎ͳ
 ΞϧΰϦζϜ (SA-IS) ͕࣮૷͞Ε͍ͯΔ
 ࿦จยखʹΏͬ͘ΓಡΉͷ͕ྑͦ͞͏
4. ༻ޠͷ४උ • Suffix (઀ඌࣙ): จࣈྻTʹରͯ͠ɼઌ಄0จࣈҎ্Λ࡟ͬͨจࣈྻ
 ྫ: T = ababb
 ઀ඌࣙ: ababb, babb, abb, bb, b, ‘’ (ۭจࣈ)
5. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦
6. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa
7. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ
8. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ
9. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ
10. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ
11. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a
12. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co
13. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a
14. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa
15. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o
16. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a
17. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a coa
18. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a coa Suffix Tree
19. Suffix Tree • ߏஙʹ͔͔Δ࣌ؒ 
 ී௨ʹ΍ΔͱO(n^2) ࣌ؒ
 Ukkonen’s Algorithm ͩͱ
 Online࡞੒ՄೳͰ O(n) ࣌ؒ a co a coa o a coa • ϝϞϦ: O(n)
 O(n^2) ʹͳΒͳ͍͜ͱʹ஫ҙ
 ݩͷจࣈྻʹର͢Δ࢝఺ɼऴ఺ΛϙΠϯλͰอ࣋͢Ε͹Α͍
20. Suffix TreeΛ༻͍ͨύλʔϯͷݕࡧ • rootϊʔυ͔ΒࢬΛબΜͰਐΉ͚ͩ • ౸ୡͨ͠ϊʔυͷ࣍਺ؚ͕·ΕΔස౓ʹͳΔ • ݕࡧͷܭࢉྔ: O(m)
 ͜͜Ͱm͸ύλʔϯ௕
 ˠ ͭ·ΓݩσʔλΛશ෦ᢞΊͳ͍ (΍ͬͨͶ) a co a coa o a coa
21. Ԡ༻: Longest Common Substrings (LCS) • ೖྗ: 2ͭҎ্ͷจࣈྻू߹S 
 ग़ྗ: Sʹڞ௨ͯ͠ݱΕΔ෦෼จࣈྻͷ͏ͪ࠷௕ͷ΋ͷ • Sʹରͯ͠Suffix TreeΛߏங͠ɼ
 root͔ΒḷΕΔϊʔυͷ͏ͪɼ྆ऀΛؚΉ
 ࠷௕ͷ෦෼Λฦͤ͹ྑ͍ $ana a $ana na banana$ana na S = { s1 = banana, s2 = ana } $ana na$ana $ana na$ana s2 s2 s2 s1, s2 s1, s2 s2 s2 s2
22. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦
23. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa
24. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ
25. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ
26. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ
27. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ
28. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a
29. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co
30. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a
31. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa
32. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o
33. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a
34. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a coa
35. Suffix Tree • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛࣙॻॱʹฒ΂ͯɼ্͔ΒଋͶͨ໦ T = cocoa cocoa ocoa coa oa a ઀ඌࣙͨͪ a coa cocoa oa ocoa ιʔτ a co a coa o a coa Suffix Tree
36. Suffix Array • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛιʔτͨ࣌͠ͷݩͷจࣈྻͷindexͷ഑ྻ
37. Suffix Array • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛιʔτͨ࣌͠ͷݩͷจࣈྻͷindexͷ഑ྻ T = cocoa
38. Suffix Array • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛιʔτͨ࣌͠ͷݩͷจࣈྻͷindexͷ഑ྻ T = cocoa ઀ඌࣙs 
 with index cocoa ocoa coa oa a 
    
    
    
    
39. Suffix Array • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛιʔτͨ࣌͠ͷݩͷจࣈྻͷindexͷ഑ྻ T = cocoa ઀ඌࣙs 
 with index cocoa ocoa coa oa a 
    
    
    
     
    
    
    
     a coa cocoa oa ocoa ιʔτ
40. Suffix Array • tl;dr 
 จࣈྻTͷ͢΂ͯͷ઀ඌࣙΛιʔτͨ࣌͠ͷݩͷจࣈྻͷindexͷ഑ྻ T = cocoa ઀ඌࣙs 
 with index cocoa ocoa coa oa a 
    
    
    
     
    
    
    
     a coa cocoa oa ocoa ιʔτ 
    
    
    
     Suffix Array
41. Suffix Array • ߏஙʹ͔͔Δ࣌ؒ
 ී௨ʹ΍Δͱ O(n^2 logn) ࣌ؒ
 SA-ISΞϧΰϦζϜʹΑΓ O(n) ࣌ؒͰߏஙͰ͖Δ 
 (ιʔτ͢Δͷʹlogn͕͔ͭͳ͍ෆࢥٞͳํ๏) • ϝϞϦ͸ࣗ໌ʹO(n) 
    
    
    
     a coa cocoa oa ocoa
42. Suffix ArrayΛ༻͍ͨύλʔϯͷݕࡧ • Suffix Array্Λύλʔϯ௕͚ͩ܁Γฦ͠ೋ෼୳ࡧ͢Δ
 O(m logn) ࣌ؒ (஗͍) • LCP Array (࠷௕ڞ௨઀಄ࣙ഑ྻ) Λ
 ิॿσʔλߏ଄ͱͯ͠࢖ͬͯ
 O(m + log n) ࣌ؒΛୡ੒Ͱ͖Δ • LCP Array͸Suffix Array͔ΒO(n)࣌ؒͰߏஙՄೳ 
    
    
    
     a coa cocoa oa ocoa
43. Suffix Tree ͱ Suffix Array ͷؔ܎ • ྺ࢙తʹ͸ Suffix Tree ͷํ͕20೥Ҏ্ૣ͘ൃݟ͞Εͨ • όΠΦͷจ຺ͰϝϞϦফඅΛམͱ͍ͨ͠Ϟνϕʔγϣϯ͔Β
 Suffix Array ͕։ൃ͞Εͨ • ͷͪʹ LCP Array ͕։ൃ͞ΕɼSuffix Tree্ͷΞϧΰϦζϜͷଟ͘ ΛSuffix Array্Ͱ࣮ߦͰ͖ΔΑ͏ʹͳͬͨ • ST, SA྆ํͱ΋ɼݸผʹൃలͨ͠ΞϧΰϦζϜ͕ଟ਺։ൃ͞Ε͍ͯΔ
44. ·ͱΊ • Suffix Tree ͱ Suffix Array ʹ͍ͭͯ঺հͨ͠ • ߏங࣌ؒ O(n)ɼϝϞϦO(n) • ݕࡧ΍ɼ࠷௕ڞ௨෦෼จࣈྻͳͲΛߴ଎ʹղ͘͜ͱ͕Ͱ͖Δ • จࣈྻָ͍͠Αʂ