Full-Text Search Demystified with Java

Today’s applications are expected to provide powerful full-text search. But how does that work in general and how do I implement it on my site or in my application?

Actually, this is not as hard as it sounds at first. This talk covers:
* How full-text search works in general and what the differences to databases are.
* How to implement common requirements with Elasticsearch.

Attendees will learn how add common search patterns to their applications without breaking a sweat.

Transcript

1. Full-Text Search Demystified Philipp Krenn̴̴̴̴@xeraa
5. Infrastructure | Developer Advocate
6. ViennaDB Papers We Love Vienna
7. Who uses databases?
8. Who needs full-text search?
9. Who uses full-text search?
10. Who uses Elasticsearch?
11. Databases vs Full-text search
14. Elasticsearch
15. Node = Instance Cluster = Cluster Shard = ⁉
16. Index = Schema Type ~ Table Document = Row Field = Column
17. Apache Lucene
18. Indexing data
19. Indexing Remove formating Tokenize Stopwords Stem
20. Source The two <em>lazy</em> dogs, were slower than the less lazy <em>dog</em>, Rover.
21. html_strip char filter The two lazy dogs, were slower than the less lazy dog, Rover.
22. standard tokenizer The two lazy dogs were slower than the less lazy dog Rover
23. lowercase token filter the two lazy dogs were slower than the less lazy dog rover
24. stop token filter two lazy dogs were slower than less lazy dog rover
25. snowball token filter two lazi dog were slow than less lazi dog rover
26. Testing it in Elasticsearch GET _analyze { "char_filter": [ "html_strip" ], "tokenizer" : "standard", "token_filter" : ["lowercase", "stop", "snowball"], "text": "The two <em>lazy</em> dogs, were slower than the less lazy <em>dog</em>, Rover." }
27. { "tokens": [ { "token": "two", "start_offset": 4, "end_offset": 7, "type": "<ALPHANUM>", "position": 1 }, { "token": "lazi", "start_offset": 12, "end_offset": 21, "type": "<ALPHANUM>", "position": 2 },
28. Inverted index Metadata: Term frequency, position doc0 doc1 doc2 two 1 [1] 1 [5] 1 [3] lazi 2 [2,9] 0 1 [7] dog 1 [3] 0 3 [1,8,13] quick 0 1 [1] 0
29. Storing data PUT /quotes/quote/1 { "text": "The two <em>lazy</em> dogs, were slower than the less lazy <em>dog</em>, Rover." }
30. Storing data { "_index": "quotes", "_type": "quote", "_id": "1", "_version": 1, "_shards": { "total": 2, "successful": 1, "failed": 0 }, "created": false }
31. Searching POST /quotes/_search { "query": { "match": { "text": "dog" } } }
32. Searching ... "hits": [ { "_index": "quotes", "_type": "quote", "_id": "1", "_score": 0.25, "_source": { "char_filter": [ "html_strip" ], "tokenizer": "standard", "token_filter": [ "lowercase", "stop", "snowball" ], "text": "The two <em>lazy</em> dogs, were slower than the less lazy <em>dog</em>, Rover." } } ] } }
33. Searching POST /quotes/_search { "query": { "match": { "text": "lazy" } } } "_score": 0.35355338
34. Scoring
35. score(q,d) = queryNorm(q) * coord(q,d) * SUM ( tf(t in d), idf(t)², t.getBoost(), norm(t,d) ) (t in q)
36. PS: v5 changes scoring BM25 https://speakerdeck.com/elastic/ improved-text-scoring-with-bm25
37. tf Term frequency tf(t in d) = √frequency
39. idf Inverse document frequency idf(t) = 1 + log ( numDocs / (docFreq + 1))
41. Field length norm norm(d) = 1 / √numTerms
42. Vector space model Match multiple terms Calculate a score for each one Create a vector
43. Search happy hippopotamus I am happy in summer. After Christmas I’m a hippopotamus. The happy hippopotamus helped Harry.
45. Example
46. Who uses Java?
47. Who uses JPA?
48. @OneToMany(mappedBy = "destCustomerId") @ManyToMany @Fetch(FetchMode.SUBSELECT) @JoinTable(name = "customer_dealer_map", joinColumns = { @JoinColumn(name = "customer_id", referencedColumnName = "id")}, inverseJoinColumns = { @JoinColumn(name = "dealer_id", referencedColumnName = "id")}) private Collection<Client> dealers;
50. Relations vs Objects
51. Ted Neward: ORM is "The Vietnam of Computer Science" — http://blogs.tedneward.com/post/the-vietnam- of-computer-science/
52. Elasticsearch Java driver
53. import static org.elasticsearch.common.xcontent.XContentFactory.*; IndexResponse response = client.prepareIndex("movies", "movie", "1") .setSource(jsonBuilder() .startObject() .field("title", "The Godfather") .field("director", "Francis Ford Coppola") .field("year", 1972) .startArray("genres") .value("Crime").value("Drama") .endArray() .endObject() ).get();
55. Spring Data Elasticsearch
56. Object-Document Mapping POJO + Annotations
57. Dependency <dependency> <groupId>org.springframework.data</groupId> <artifactId>spring-data-elasticsearch</artifactId> <version>2.0.0.RELEASE</version> </dependency>
58. Annotations
59. @Document(indexName = "movies", type = "movie") public class Movie { @Id private String id; private String title; @Field(type = String, index = analyzed, store = true) private String director; private Integer year; @Field(type = FieldType.Nested) private List<String> genres; // Standard getters and setters }
60. Movie movie = new Movie("1"); movie.setTitle("The Godfather"); movie.setDirector("Francis Ford Coppola"); movie.setYear(1972); movie.setGenres(Arrays.asList("Crime", "Drama")); movieService.save(movie);
61. PS: Use Lombok
62. Repository
63. public interface MovieRepository extends ElasticsearchRepository<Movie, String> { Page<Movie> findByDirectorsName(String director, Pageable pageable); }
64. String directorToFind = "Francis Ford Coppola"; Page<Movie> movieByDirectorName = movieService.findByDirectorsName( directorToFind, new PageRequest(0, 10) );
65. Polymorphis m
66. public abstract class EmployeeEntity { protected String name; } public class ManagerEntity extends EmployeeEntity { private Boolean approveFunds; } public class WorkerEntity extends EmployeeEntity { private Integer yearsExperience; }
67. RDBMS 1. Union table with (many) NULL values
68. RDBMS 2. Concrete instances without common queries
69. RDBMS 3. Base table joined with all subtables
71. public abstract class EmployeeEntity { @Id protected String id; protected String name; } @Document(indexName = "employees", type = "manager") public class ManagerEntity extends EmployeeEntity { private Boolean approveFunds; } @Document(indexName = "employees", type = "worker") public class WorkerEntity extends EmployeeEntity { private Integer yearsExperience; }
72. Conclusion
73. Search must must_not should minimum_should_match
74. Filter
75. Aggregations
76. But is it fast?
77. Things you won't get Transactions JOINs
78. Things you will get Rapid development Replication and sharding Impedence match with rich documents & queries
79. To sum up Developers❤Elasticsearch
81. 18:10 Track 2 Going Elastic
82. Thanks! Questions? Philipp Krenn̴̴̴̴̴@xeraa PS: Stickers
83. Image Credit → Schnitzel https://flic.kr/p/9m27wm → Architecture https://flic.kr/p/6dwCAe → Conchita https://flic.kr/p/nBqSHT