今日から始める全文検索

2019年1月22日に開催された【とらのあな主催】オタクが最新技術を追うライトニングトークイベント２回目のライトニングトーク資料です。

Transcript

1. 
   
   Copyright (C) 2019 Toranoana Inc. All Right Reserved.
   
   
   ΦλΫ͕࠷৽ٕज़Λ௥͏ϥΠτχϯάτʔΫΠϕϯτ̎ճ໨
   
   
   
   

   View Slide

2. 
   
   Copyright (C) 2019 Toranoana Inc. All Right Reserved.
   
   • 201810
   • Fantia
   
   Ruby, JavaScript
   Java
   2
   
   
   

   View Slide

3. 
   
   Copyright (C) 2019 Toranoana Inc. All Right Reserved.
   ●
   ●
   ●
   ●
   ●
   
   ●
   ●
   3
   
   

   View Slide

4. 
   
   Copyright (C) 2019 Toranoana Inc. All Right Reserved.
   <,+%'>
   $)-2<&10: #0:
   69
   
   <(;>
   ● $)-2753
   8
   0*
   ● $)-2"/
   .4
   ! *
   4
   
   

   View Slide

5. EB EB EB EB E
   B EB
   Copyright (C) 2019 Toranoana Inc. All Right Reserved.
   0<>C
   0<>CH###
   D: Full text searchI
   &,(*-'F;<=H)$%
   +I?8<72">C )$%+4
   >C3.)$%+1<72>CA
   F;<= <=5! 0<"9G
   >C:6/@ !
   HWikipediaI
   5
   F;<=()$%+)5!
   0<7"9G>C
   

   View Slide

6. 
   
   Copyright (C) 2019 Toranoana Inc. All Right Reserved.
   
   
   
   
   6
   

   View Slide

7. $" $" $" $" $
   " $"
   Copyright (C) 2019 Toranoana Inc. All Right Reserved.
   7
   !#
   !#
   !#
   Apache Lucene)Java*
   Whoosh)Python*
   
   Apache Solr
   Elasticsearch
   Groonga)C&'*
   
   Google!#
   Fess
   
   
   %
   
   !#(
   

   View Slide

8. $
   " $
   " $
   " $
   " $
   
   " $
   "
   Copyright (C) 2019 Toranoana Inc. All Right Reserved.
   #
   
   #% #&
   ≒ # !
   8
   

   View Slide

9. 
   
   Copyright (C) 2019 Toranoana Inc. All Right Reserved.
   
   
   
   • grep
   •
   9
   

   View Slide

10. 
    
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    
    grep
    ⇒
    10
    

    View Slide

11. 41 41 41 41 4
    1 41
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    7-,2.)
    <.)>
    • "' 6
    9 %8*( 7/&
    5+:
    <$>
    • Linuxgrep!
    • #30
    Ctrl+F,2
    11
    

    View Slide

12. 
    
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    
    
    ⇒
    
    
    12
    

    View Slide

13. )$ )$ )$ )$ )
    $ )$
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    '-
    ,!*!%!&#
    
    
    13
    Ruby(,Ruby +2"
    

    View Slide

14. %! %! %! %! %
    ! %!
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    ##
    < >
    #&#(*(
    #)
    <>
    • Google"#
    
    • RDB (B-Tree)
    • #
    ('$ )
    14
    

    View Slide

15. $! $! $! $! $
    ! $!
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    %#
    "
    "
    '&"
    (
    15
    

    View Slide

16. 3/ 3/ 3/ 3/ 3
    / 3/
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    72-%
    <-%>
    56(1#$)&)
    .2*04,9
    +18"<8=&)!>
    :&) =2,3>
    1#(';"!
    16
    

    View Slide

17. ($ ($ ($ ($ (
    $ ($
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    &
    !
    "
    "ID
    1
    2 -
    3
    17
    )+.&+/ "ID
    2, 3
    1, 2
    3
    1, 2, 3
    1, 3
    - 2
    2
    & (,')
    N-Gram
    %*#
    ."/
    

    View Slide

18. .
    * .
    * .
    * .
    * .
    
    * .
    *
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    &)+
    ,
    <> )+
    → '!
    &(%
    / 03+$04 &(ID
    2, 3
    1, 2
    3
    1, 2, 3
    " 1, 3
    2 2
    2
    &(ID #
    1 "
    2 2
    3 "
    18
    +$ (1-)
    &(
    

    View Slide

19. &# &# &# &# &
    # &#
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    ') "
    )+
    ()
    .%))) ,!$
    *-
    19
    ⇒)
    

    View Slide

20. ., ., ., ., .
    , .,
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    /2%$*
    25
    12
    9&(2 26 2
    74!38
    20
    ⇒
    ...
    $*+
    • "#-0)7'8
    • N-Gram
    

    View Slide

21. (& (& (& (& (
    & (&
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    ')$
    
    +"%
    !#*,
    '-
    ,"-
    21
    

    View Slide

22. 
    
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    
    <>
    
    ↓
    ...
    
    !6
    22
    

    View Slide

23. 50 50 50 50 5
    0 50
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    $(16*/%
    <.>
    +2#
    
    83:N-Gram"
    +2
    '+2!4
    <,.>
    9)&4 +2
    7
    ;+2-<
    23
    

    View Slide

24. 
    
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    N-Gram
    !"
    
    N
    
    24
    

    View Slide

25. 
    
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    N-Gram
    <> 2-Gram
    "
    
    ↓ 2 ...
    "
    "
    
    
    !#10
    25
    

    View Slide

26. :4 :4 :4 :4 :
    4 :4
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    N-Gram3(
    < 2>
    /61A?9
    %9 /67.
    < B
    <02>
    /6"
    >8@')5;-#
    N(,&+$*)!= /6
    26
    

    View Slide

27. 63 63 63 63 6
    3 63
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    04#!
    -*04 "$@-*81
    04:(,
    047 @%<'-*
    04 -*A
    >).=-*+29;&?
    27
    
    045/
    

    View Slide

28. 82 82 82 82 8
    2 82
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    15!
    . >
    • )+4:/A9$ <→. >
    . >&,315 >&
    ,-15 ?(#7";@
    • 2-GramA9$ <→. >
    >&,-1560$
    
    =%*'
    ⇨15
    28
    

    View Slide

29. 
    
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    N-
    Gram
    29
    
    
    
    N-Gram
    

    View Slide

30. 94 94 94 94 9
    4 94
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    &"
    • &,06BA6(#
    
    C?.@(D
    • &,066( >8!
    • >8! :'=#-)+5;
    /N-GramE
    – 0627< ⇒ N-Gram%
    – * ⇒ )+5;/%
    – 0621*06
    ⇒ N-Gram + )+5;/ $3 30
    

    View Slide

31. %! %! %! %! %
    ! %!
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    %!
    Fantia
    # $
    # '
    • "& ( N-Gram
    
    31
    

    View Slide

32. 
    
    
    
    
    
    
    
    Copyright (C) 2019 Toranoana Inc. All Right Reserved.
    
    
    32
    

    View Slide