Google BigQueryを用いた次世代の検索手法

Dev7++ Study Night 第3回 登壇資料
----
公開加筆版 : https://speakerdeck.com/hikawa/google-bigquerywoyong-itaci-shi-dai-falsejian-suo-shou-fa-gong-kai-jia-bi-ban

Transcript

1. Google BigQueryΛ༻͍ͨ
   ࣍ੈ୅ͷݕࡧख๏
   

   View Slide

2. ςʔϚ
   (PPHMF͕ఏڙ͢Δɺେن໛෼ࢄσʔλ΢ΣΞϋ΢ε
   

   View Slide

3. ςʔϚ
   ߴ଎
   ௿Ձ֨
   ϑϧϚωʔδϝϯτ
   αʔόʔϨε
   ϖλόΠτڃσʔλ
   42-ߏจ
   6%'
   

   View Slide

4. ςʔϚ
   ߴ଎
   ௿Ձ֨
   ϑϧϚωʔδϝϯτ
   αʔόʔϨε
   ϖλόΠτڃσʔλ
   42-ߏจ
   6%'
   

   View Slide

5. User Defined Functions
   42-ؔ਺Λಠࣗʹ࡞੒͠ɺӡ༻
   +BWBTDSJQUɾ42-
   

   View Slide

6. 'J[[#V[[ؔ਺
   

   View Slide

7. View Slide

8. ػցֶश
   લॲཧ ֶश ਪ࿦
   

   View Slide

9. ػցֶश
   લॲཧ ֶश ਪ࿦
   

   View Slide

10. ೔ຊޠ෼͔ͪॻ͖ؔ਺
    

    View Slide

11. View Slide

12. ʮେྔͷσʔλʯʮߴ଎Ͱ҆ՁͳΫϥ΢υ؀ڥʯ
    Կ͕ग़དྷΔΑ͏ʹͳΔͷ͔ʁ
    

    View Slide

13. ैདྷͷݕࡧ
    ςΩετݕࡧ ΧςΰϦɾλάݕࡧ
    

    View Slide

14. View Slide

15. View Slide

16. ίϯςϯπͷதͰԿ͕ॏཁ͔ɺߟྀͯ͠ͳ͍ݕࡧ
    هࣄA
    MySQL
    BigQuery
    هࣄB
    MySQL BigQuery
    هࣄC
    MySQL
    BigQuery
    ʹશͯಉ͡ѻ͍
    

    View Slide

17. ίϯςϯπͷதͰԿ͕ॏཁ͔ɺߟྀͯ͠ͳ͍ݕࡧ
    هࣄA
    MySQL
    BigQuery
    هࣄB
    MySQL BigQuery
    هࣄC
    MySQL
    BigQuery
    ʹશͯಉ͡ѻ͍
    

    View Slide

18. هࣄD
    MySQL
    BigQuery
    هࣄE
    MySQL
    BigQuery
    هࣄF
    MySQL
    BigQuery
    هࣄG
    MySQL
    BigQuery
    ॏΈΛอ࣋ͨ͠ɺۃΊ͍ͯۙίϯςϯπͷݕࡧ
    هࣄA
    MySQL
    BigQuery
    

    View Slide

19. ͲͷΑ͏ʹԋࢉ͢Δ͔ʁ
    ର৅ͱͳΔυΩϡϝϯτ"ͱυΩϡϝϯτ#$%&Λ
    ൺֱ͠ɺྨࣅ౓ॱʹฒ΂ସ͑Δ
    

    View Slide

20. ίαΠϯྨࣅ౓ʢίαΠϯڑ཭ʣ
    O࣍ݩͷϕΫτϧಉ࢜Λൺֱ͠ɺʙͷൣғͰྨࣅ౓ΛٻΊΔ
    

    View Slide

21. ϕΫτϧΛBigQueryʹ֨ೲ͢Δ
    Ұൠతʹ͸)BEPPQ΍"QBDIF4QBSLΛ༻͍ͯ

    ܭࢉͨ͠಺༰Λσʔλ΢ΣΞϋ΢εʹ֨ೲ
    ϕΫτϧͷܭࢉʹ͸ʮେྔͷσʔλʯʮߴ଎ͳ؀ڥʯ͕ඞཁ
    

    View Slide

22. ࣮༻Մೳͳ

    4ͭͷख๏
    

    View Slide

23. έʔε1. Word2Vec
    จষʹ͓͚Δ୯ޠؒͷҙຯ߹͍ΛϕΫτϧԽ
    

    View Slide

24. έʔε2. Doc2Vec
    8PSE7FDΛ֦ு
    จষؒͷҙຯ߹͍ΛϕΫτϧԽ
    

    View Slide

25. έʔε3. TF-IDF
    จষʹ͓͚Δॏཁ୯ޠͷநग़
    

    View Slide

26. BigQueryͰυΩϡϝϯτͷϕΫτϧΛٻΊΔ
    Ͱԋࢉͯ͠ϦΞϧλΠϜʹϕΫτϧԽɾྨࣅ౓൑அ΋Մೳ
    5'*%'ͳΒɺશͯͷܭࢉΛ
    

    View Slide

27. %&.0
    

    View Slide

28. έʔε4. ػցֶश
    ίϯςϯπʹ͓͚Δಛ௃ϕΫτϧͷநग़
    

    View Slide

29. ݶΒΕͨσʔλΛ

    ਓྗͰλά෇͚
    ςΩετσʔλ
    ը૾
    Ի੠
    ࣌ܥྻσʔλ
    

    View Slide

30. ݶΒΕͨσʔλΛ

    ਓྗͰλά෇͚
    ςΩετσʔλ
    ը૾
    Ի੠
    ࣌ܥྻσʔλ
    

    View Slide

31. ςΩετσʔλ
    ը૾
    Ի੠
    ࣌ܥྻσʔλ
    σʔλΛϕΫτϧԽ͠ɺ

    ػցʹΑΔεϚʔτͳϥϕϧ෇͚
    

    View Slide

32. ϕΫτϧσʔλʹΑͬͯɺ

    ػց͕ҙຯΛཧղͨ͠ɺ
    εϚʔτͰݡ͍σʔλ΢ΣΞϋ΢ε
    

    View Slide

33. Ϣʔβ͕ٻΊΔ΋ͷΛɺΑΓ൓ө͍ͤͯ͘͞
    ࣍ੈ୅ͷݕࡧख๏
    

    View Slide

34. Google BigQueryΛ༻͍ͨ
    ࣍ੈ୅ͷݕࡧख๏
    ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide

35. ຊ೔࣋ͪؼͬͯ௖͖͍ͨςʔϚ
    #JH2VFSZ
    6%'

    ϕΫτϧʹΑΔλά෇͚

    ίαΠϯྨࣅ౓

    NFEJCB
    

    View Slide