Elasticsearch 구조 및 검색기능 기획 시 고려 사항

Transcript

1. ੿ೠ࠼/23.12.6 Elasticsearch
2. Contents • elasticsearchۆ • Full Scan VS Pre fi x VS Full Text • Tokenizer / Normalizer • Ѩ࢝ ঌҊ્ܻ / Fuzziness • ੉Ѧ۽ ަ ঳ਸ ࣻ ੓חо
3. Elasticsearchۆ
4. Full Scan VS Prefix VS Full Text
5. Full Scan
6. Prefix CREATE INDEX `IDX_content` ON document (content);
 or CREATE INDEX `IDX_content` ON document (content(n)); content ஸۢਵ۽ ੋؙझ ࢤࢿ
7. Prefix
8. Full Text - ৉࢝ੋ
9. Full Text - ৉࢝ੋ • Tokenizer - ࢝ੋೡ ױয Ѿ੿ ߑߨ • ੿۳ - ௪ܻо ঴݃ա ޙࢲ৬ ੸೤ೠо?
10. Full Text - ৉࢝ੋ • a, an, theܳ Ѩ࢝೧ঠೡө? • ‘׏’, ‘׏૓’, ‘׏૓झ’۽ Ѩ࢝ೞݶ
 ‘׏૓झ’о Ѩ࢝غযঠೡө? • “׏૓झо”੉ۄҊ Ѩ࢝ೞݶ
 “׏૓झо”݅ Ѩ࢝غযঠೡө? ইפݶ
 “׏૓झীѱ”ب Ѩ࢝غযঠೡө? • url਷ যڌѱ Ѩ࢝ೞ૑?
 googleਸ ଢ଼ਸٸ 
 “google.com", “https://google.com" ١੉ Ѩ࢝ؼ ࣻ ੓ਸө?
11. Full Text - ৉࢝ੋ ޙࢲܳ Ѩ࢝ೡ ࣻ ੓ח ױয(token)ਵ۽ աׇ઱ח ஹನք౟ Tokenizer
12. • ౠ੿ ష௾ਸ ׮ܲ ష௾ਵ۽ ߄Լ઱ח ஹನք౟ • Ex) ࠂࣻܳ ױࣻചदெષ, җѢഋਸ അ੤ഋਵ۽ ߄Լષ, ؀ޙ੗ܳ ࣗޙ੗۽ ߄Լષ • The foxes loved the little prince => [the, fox, love, little, prince] Normalizer
14. ੿۳ • elasticsearchীࢲח bm25ۄח ঌҊ્ܻਵ۽ ௪ܻی ޙࢲо ঴݃ա ҙ۲੓ח૑ܳ ੼ࣻ ۽ ն • ੌ߈੸ਵ۽ ੜ ٜযݏח ҕध • ցޖ ൔೠ ష௾਷ о઺஖ܳ ծ୸׮ • ష௾੉ ޙࢲীࢲ ৈ۞ߣ ঱әغݶ ੼ࣻܳ ֫ੋ׮ • ӡ੉о ૣ਷ ޙࢲо જ׮ ੌ߈੸ਵ۽ ੜٜযݏח ௪ܻ৬ ޙࢲ ҙ۲ب ੼ࣻ ҅࢑ध
15. BM25 ঌҊ્ܻ https://www.elastic.co/kr/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables • “׏૓झо જইೞח ֢ې” -> [“׏૓झ”, “જইೞ׮”, “֢ې”]
16. Ѩ࢝ ঌҊ્ܻ / Fuzziness Why ?????
17. Ѩ࢝ ঌҊ્ܻ / Fuzziness - Tokenizer / Normalizer - (ઑࢎо ੜ ୊ܻغחо?) - Fuzziness (auto)
18. Ѩ࢝ ӝמ ѐߊद ӝࣿ ৻੸ਵ۽ Ҋ۰೧ঠೞח ࢎ೦
19. ࠺ѐߊ ૒ҵ੉ ൔ൤ ೞח য়೧
20. Ѩ࢝ ӝמ ӝദ != Ѩ࢝ػ ചݶ
21. Ѩ࢝ ߧਤ ઁݾ? ղਊ? ੘ࢿ੗? ؆Ӗ? ӝఋ ݫఋؘ੉ఠ?
22. ஶబஎ ੿۳ ୭न ࢤࢿࣽ?
 Ѩ࢝ ੸೤ب ࣽ? જইਃ ࣽ?
23. Ӓ৻ Ѩ࢝ ҃೷җ ౟ۨ੉٘ য়೐ ਬ੷੄ য়ఋܳ ঴݃ա ࠊ઴ Ѫੋо? <-> ӝദ, ࢿמ ز੄য, ਬ੄যܳ ୊ܻ೧઴ Ѫੋо <-> ҳഅ դ੉ب, ١۾ ࢿמ, Ѩ࢝ ࢿמ Ѩ࢝ ௪ܻח যו੿ب ӡ੉ੋо <-> ҳഅ դ੉ب, Ѩ࢝ ࢿמ Ѩ࢝ ௪ܻח ױযੋо ޙ੢ੋо <-> ҳഅ դ੉ب, Ѩ࢝ ࢿמ