better searching with elasticsearch - PHPConfPL

better
searching with
elasticsearch
Sunday, 3 November 2013

View Slide

richard miller
@mr_r_miller
richarddmiller.co.uk
sensiolabs uk

View Slide

why worry
about search?

View Slide

ok, so what
might
someone want
from search?

View Slide

Given I am a hungry person
When I search for somewhere to eat
Then I should see relevant results
And they should be ordered by relevance

View Slide

yep, no problem

View Slide

Then I should see results for relevant reviews

View Slide

sure

View Slide

Then I should see results for similar words

View Slide

yeah, we can do
that do that

View Slide

When I search for somewhere to eat near me
Then I should see results close to my location

View Slide

that’s
certainly
possible

View Slide

When I typo entering my search terms
Then I should see “did you mean?” suggestions

View Slide

well other
sites do that,
so why not?

View Slide

When I start entering my search terms
Then I should see suggestions straight away

View Slide

ok, ok

View Slide

When I start searching for somewhere to eat
Then I should be able to filter my results

View Slide

now this we
can do

View Slide

When I start view details of somewhere to eat
Then I should see suggestions of similar eateries

View Slide

yep, yep

View Slide

Given I am a restaurateur
When I upload my PDF menu
Then hungry people should be able to search it

View Slide

hmm...

View Slide

well let’s get
the easy stuff
out of the way

View Slide


View Slide

SELECT *
FROM
eateries
WHERE NAME
LIKE
“%searchterm%”

View Slide

multiple
searchterms?

View Slide

...WHERE (
name LIKE
“%multiple%”
OR
name LIKE
“%searchterm%”)

View Slide

but wait...

View Slide

but wait...
...multiple ﬁelds

View Slide

...WHERE (
name LIKE
“%multiple%”
OR
name LIKE
“%searchterm%”...

View Slide

...OR
desc LIKE
“%multiple%”
OR
desc LIKE
“%searchterm%”)

View Slide


View Slide

ORDER?

View Slide


View Slide

the

View Slide

the
with

View Slide

the
with
be

View Slide

the
but
with
be

View Slide

the
but
with
be
it

View Slide

it
the
but
with
be
food

View Slide

oh ok,

View Slide

oh ok,
full text search...

View Slide

SELECT *
FROM
eateries
MATCH (name, desc)
AGAINST(
“multiple
searchterms”)

View Slide

that should do!

View Slide

what was left?

View Slide


View Slide

...LEFT JOIN...

View Slide


View Slide

grilled

View Slide

grilling
grilled

View Slide

grilling
grilled
grills

View Slide

grill
grilling
grilled
grills

View Slide


View Slide

elasticsearch

View Slide

distributed,
schemaless,
document oriented,
Lucene based
search engine with a
REST API

View Slide

github

View Slide

stackoverﬂow

View Slide

gov.uk

View Slide

foursquare

View Slide

soundcloud

View Slide

wget https://download.elasticsearch.org/
elasticsearch/elasticsearch/
elasticsearch-0.90.5.tar.gz
tar -zxvf elasticsearch-0.90.5.tar.gz
cd elasticsearch-0.90.5

View Slide

bin/elasticsearch

View Slide

Lucene based

View Slide

request
curl -X GET http://localhost:9200/

View Slide

{
"ok" : true,
"status" : 200,
"name" : "Mr. Wu",
"version" : {
"number" : "0.90.5",
"build_hash" : "c8714e8...dedee",
"build_timestamp" : "2013-09-17T12:50:20Z",
"build_snapshot" : false,
"lucene_version" : "4.4"
},
"tagline" : "You Know, for Search"
}
response

View Slide

database index

View Slide

table type

View Slide

record document

View Slide

distributed

View Slide

0 1 2 3 4
Instance

View Slide

0
1
Instance 1
Instance 2
Instance 3
2
3
4

View Slide

0
1
2
3
4
Instance 1
Instance 2
Instance 3
0
1
2
3
4

View Slide

document oriented

View Slide

REST API

View Slide

curl -XPOST 'http://localhost:9200/eatly/eateries/' -d '{
"name" : "Jeff''s Burgers",
"desc" : "Blah Blah dirty burgers..."
}'

View Slide

{"ok":true,"_index":"eatly","_type":"eateries","_id":"EGHS
ABBxQkaPR_DJxPxXZA","_version":1}

View Slide

?pretty=true

View Slide

{
"ok" : true,
"_index" : "eatly",
"_type" : "eateries",
"_id" : "cmElnobYSy6386TOUVbGZQ",
"_version" : 1
}

View Slide

curl -XPUT 'http://localhost:9200/eatly/eateries/1' -d '{
}'

View Slide

{
"ok" : true,
"_index" : "eatly",
"_id" : "1",
"_version" : 1
}

View Slide

curl -XGET 'http://localhost:9200/eatly/eateries/1'

View Slide

{
"_index" : "eatly",
"_id" : "1",
"_version" : 1,
"exists" : true,
"_source" : {
"name" : "Jeff's Burgers",
}
}

View Slide

"name" : "Jeff''s Burger Joint"
}'

View Slide

{
"ok" : true,
"_index" : "eatly",
"_id" : "1",
"_version" : 2
}

View Slide


View Slide

{
"_index" : "eatly",
"_id" : "1",
"_version" : 2,
"exists" : true,
"_source" : {
"name" : "Jeff's Burger Joint"
}
}

View Slide

curl -XDELETE 'http://localhost:9200/eatly/eateries/1'

View Slide


View Slide

{
"_index" : "eatly",
"_id" : "1",
"_version" : 2,
"exists" : false
}

View Slide

search engine

View Slide

curl -XGET '.../eatly/eateries/_search?q=name:Burger'

View Slide

{
"took" : 7,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.15342641,
"hits" : [ {
"_index" : "eatly",
"_id" : "1",
"_score" : 0.15342641,
"_source" : {"name" : "Jeff's Burger Joint"}
} ]
}
}

View Slide

http://localhost:9200/eatly/cafes,pubs/_search?q=Burger
http://localhost:9200/eatly/_search?q=name:Burger
http://localhost:9200/eatly,cookr/_search?q=name:Burger
http://localhost:9200/_all/pubs/_search?q=name:Burger
http://localhost:9200/_search?q=name:Burger'

View Slide

schemaless?

View Slide

curl -XPUT 'http://localhost:9200/eatly/' -d '{
"settings" : {
"index" : {
"number_of_shards" : 4
"number_of_replicas" : 2
}
}
}'

View Slide


View Slide

name > desc

View Slide

curl -XPUT '.../eatly/eateries/_mapping' -d '{
"eateries" : {
"properties" : {
"name" : {"type" : "string", "boost" : "1.5"},
"desc" : {"type" : "string"}
}
}
}'

View Slide


View Slide

"eateries" : {
"properties" : {
"reviews" : {
"properties" : {
"review" : {"type" : "string"},
"reviewer" : {"type" : "string"}
}
}
}
}
}'

View Slide

"desc" : "Blah Blah dirty burgers...",
"reviews" : [
{
"review" : "Yadda, yadda, yadda",
"reviewer" : "John Smith"
},
{
"review" : "na na na",
"reviewer" : "Billy Badger"
}
]
}'

View Slide

curl -XGET '.../eateries/_search?q=reviews.review:Burger'

View Slide


View Slide

burgers
I really enjoyed Jeff's Burger Joint, the
burgers were excellent! can't fault it
standard tokenizer
I really enjoyed Jeff's Burger
Joint the were
excellent can't fault it

View Slide

excellent!
burgers
whitespace tokenizer
I really enjoyed Jeff's Burger
Joint, the were
can't fault it

View Slide

burgers
letter tokenizer
I really enjoyed Jeff Burger
Joint the were
excellent can fault it
s
t

View Slide

burgers
standard tokenizer + lowercase ﬁlter
i really enjoyed jeff's burger
joint the were
excellent can't fault it

View Slide

burgers
+ stop ﬁlter
i really enjoyed jeff's burger
joint were
excellent can't fault

View Slide

burger
+ stop ﬁlter + stemmer ﬁlter
i realli enjoi jeff burger
joint were
excel can't fault

View Slide

"settings" : {
"index" : {
"analysis" : {
"analyzer" : {
"snowball_analyzer" : {
"type" : "snowball"
}
}
}
}
}
}'

View Slide

"eateries" : {
"properties" : {
"desc" : {
"type" : "string",
"analyzer": "snowball_analyzer"
}
}
}
}'

View Slide

"settings" : {
"index" : {
"analysis" : {
"analyzer" : {
"custom_analyzer" : {
"type" : "custom",
"tokenizer" : "lowercase"
"filter" : ["stop", "extra_stop"]
}
},
"filter" : {
"extra_stop":{
"type" : "stop",
"stopwords" : ["food"]
}
}
}
}
}
}'

View Slide

"eateries" : {
"properties" : {
"desc" : {
"type" : "string",
"analyzer": "custom_analyzer"
}
}
}
}'

View Slide

query DSL

View Slide

curl -XGET '.../eatly/eateries/_search' -d '{
"query" : {
"match" : {
"name" : "burger"
}
}
}'

View Slide

"query" : {
"match" : {
"name" : "burger bar"
}
}
}'

View Slide

"query" : {
"match" : {
"name" : {
"query" : "burger bar",
"operator" : "and"
}
}
}
}'

View Slide

"query" : {
"multi_match" : {
"fields" : ["name","desc","reviews.review"]
}
}
}'

View Slide

"query" : {
"multi_match" : {
"fields" : ["name^2","desc","reviews.review"]
}
}
}'

View Slide

ﬁlters

View Slide


View Slide

"name" : "Jeff''s Burger Joint",
"tags" : ["greasy", "retro"]
}'

View Slide

"query" : {
"multi_match" : {
"fields" : ["name^2", "desc", "reviews.review"]
}
},
"filter" : {
"not" : {
"terms" : { "tags" : ["greasy", "meaty"]}
}
}
}'

View Slide

query
or ﬁlter?

View Slide


View Slide

"eateries" : {
"properties" : {
"reviews" : {
"properties" : {
"review" : {"type" : "string"},
"reviewer" : {"type" : "string"}
}
},
"location" : {
"type" : "geo_point"
}
}
}
}'

View Slide

"reviews" : [
{
"review" : "Yadda, yadda, yadda",
"reviewer" : "John Smith"
},
{
"review" : "na na na",
"reviewer" : "Billy Badger"
}
],
"location" : {
"lat" : 41.12,
"lon" : -71.34
}
}'

View Slide

"sort" : [
{
"_geo_distance" : {
"eatery.location" : [-40, 70],
"order" : "asc",
"unit" : "miles"
}
}
],
"query" : {
"multi_match" : {
"fields" : ["name^2","desc","reviews.review"]
}
}
}'

View Slide

sense (chrome extension)

View Slide

head (elasticsearch plugin)

View Slide

elastica

View Slide

sherlock
elasticsearch
elasticsearch-php (ofﬁcial)

View Slide

{
"require": {
"ruflin/elastica": "v0.90.2.0"
}
}

View Slide

$client = new \Elastica\Client();

View Slide

$index = $client->getIndex("eatly");
$type = $index->getType("eateries");

View Slide

$type->addDocument(
new \Elastica\Document(
null,
array(
"name" => "Jeff's Burger Joint",
"desc" => "Blah Blah..."
)
)
);

View Slide

$type->addDocument(
$id,
array(
"id" => $id,
)
)
);

View Slide

$type->addDocuments($documents);
$type->getDocument($id);
$type->deleteId($id);
$type->deleteDocument($document);
$type->delete();

View Slide

$type->search("burger");
$type->search("burger", 10);
$type->count("burger");

View Slide

$multiMatchQuery = new \Elastica\Query\MultiMatch();
$multiMatchQuery
->setQuery("burgers bar")
->setFields(
array("name^2", "desc", "reviews.review")
)
;
$query = new \Elastica\Query($multiMatchQuery);
$query->setFilter(
new \Elastica\Filter\BoolNot(
new \Elastica\Filter\Terms(
"tags",
array("greasy", "meaty")
)
)
);
$results = $type->search($query);

View Slide

$mapping = new \Elastica\Type\Mapping();
$mapping->setType($type);
$mapping->setProperties(
array(
"name" => array(
"type" => "string"
),
"desc" => array(
"type" => "string",
"boost" => 1.5
)
)
);
$mapping->send();

View Slide

✔

View Slide

?

View Slide

$facet = new \Elastica\Facet\Terms("tags");
$facet->setField("tags");
$facet->setSize(10);
$facet->setOrder("reverse_count");
$query->addFacet($facet);

View Slide

$facets = $resultSet->getFacets();
foreach ($facets["tags"]["terms"] as $facet)
{
printf(
"%s: %s\n",
$facet["term"],
$facet["count"]
);
}
thai: 4
italian: 8
greek:4

View Slide

When I start view details of somewhere to eat
Then I should see suggestions of similar eateries

View Slide

$type->moreLikeThis($document);

View Slide

$type->moreLikeThis(
new \Elastica\Document($id)
);

View Slide

Given I am a restaurateur
When I upload my PDF menu
Then hungry people should be able to search it

View Slide

bin/plugin -install \
elasticsearch/elasticsearch-mapper-attachments/1.9.0

View Slide

$propertyMapping = array(
//...
"menu" => array("type" => "attachment"),
)

View Slide

$document->addFile('menu', $filepath);

View Slide

When I typo entering my search terms
Then I should see “did you mean?” suggestions

View Slide

$query = new \Elastica\Query(
array(
"query" => array(...),
"suggest" => array(
"check1" => array(
"text" => "berger",
"term" => array(
"field" => "name"
)
)
)
)
);

View Slide

$data = $results->getResponse()->getData();
print_r($data["suggest"]);

View Slide

Array
(
[check1] => Array
(
[0] => Array
(
[text] => berger
[offset] => 0
[length] => 6
[options] => Array
(
[0] => Array
(
[text] => burger
[score] => 0.8333333
[freq] => 1
)
)
)
)
)

View Slide

When I start entering my search terms
Then I should see suggestions straight away

View Slide

$mapping->setProperties(
array(
"name" => array(
"type" => "string"
),
"desc" => array(
"type" => "string",
"boost" => 1.5
),
"name_suggest" => array(
"type" => "completion"
)
)
);

View Slide

$id,
array(
"id" => $id,
"name_suggest" => "Jeff's Burger Joint",
)
);

View Slide

$query = array(
"eateries" => array(
"text" => 'j',
"completion" => array(
"field" => "name_suggest",
)
)
);

View Slide

$response = $index->request(
"_suggest",
\Elastica\Request::GET,
$query
);
$data = $response->getData();
print_r($data["eateries"]);

View Slide

Array
(
[0] => Array
(
[text] => j
[offset] => 0
[length] => 1
[options] => Array
(
[0] => Array
(
[text] => Jeffs Burger Joint
[score] => 1
)
)
)
)

View Slide

better searching with elasticsearch - PHPConfPL

better searching with elasticsearch - PHPConfPL

Richard Miller

More Decks by Richard Miller

Other Decks in Technology

Featured

Transcript