Elasticsearch

An introductory talk about Elasticsearch I gave on MaceióDEVMeetup#5

Event: http://www.meetup.com/maceio-dev-meetup/events/220830067/

Transcript

1. Elasticsearch Tony Messias MaceioDevMeetup #5
2. who am I? ➔ Tony Messias ~ @tony0x01 ➔ Building web stuff since ~2010
4. search
5. what is “search”? “A search is the organized pursuit of information (...) that you want to find, but you have no idea where it is.” (NASA).
6. how do we add search to our apps? ➔ SQL queries (not just LIKE queries); ➔ Elasticsearch; ➔ Lucene; ➔ Solr; ➔ …
7. how do we add search to our apps? ➔ SQL queries (not just LIKE queries); ➔ Elasticsearch; ➔ Lucene; ➔ Solr; ➔ …
8. who uses it?
10. what is Elasticsearch? ➔ database server; ➔ document based; ➔ based on Apache Lucene; ➔ schema-free*;
11. why Elasticsearch? ➔ speaks HTTP/JSON; ➔ easy to setup; ➔ data replication; ➔ easily scalable;
12. what can I use it for?
14. but it can do more than that…
15. some analogy * Relational ES database index table type row document column field schema mapping index everything SQL QueryDSL
16. REST(ish) $ curl -XPOST http://es-server.com/index/type/id -d '{}'
17. REST(ish) $ curl -XPOST http://es-server.com/index/type/id -d '{}'
18. REST(ish) $ curl -XPOST http://es-server.com/index/type/id -d '{}'
19. REST(ish) $ curl -XPOST http://es-server.com/index/type/id -d '{}'
20. REST(ish) $ curl -XPOST http://es-server.com/index/type/id -d '{}'
21. enough with the talking… let’s do some searches!
22. but first, let’s talk about CRUD
24. indexing
25. { "email": "john@smith.com", "first_name": "John", "last_name": "Smith", "info": { "bio": "Eco-warrior and defender of the weak", "age": 25, "interests": [ "dolphins", "whales" ] }, "join_date": "2014/05/01" }
26. $ curl -XPOST localhost:9200/my-app/users -d ‘{ "email": "john@smith.com", "first_name": "John", "last_name": "Smith", "info": { "bio": "Eco-warrior and defender of the weak", "age": 25, "interests": [ "dolphins", "whales" ] }, "join_date": "2014/05/01" }’
27. { "_index": "my-app", "_type": "users", "_id": "AUxOBeJbue-jeR3jNcHj", "_version": 1, "created": true }
28. $ curl -XPUT localhost:9200/my-app/users/AUxOBeJbue-jeR3jNcHj -d ‘{ "email": "john@smith.com", "first_name": "John" }’
29. { "_index": "my-app", "_type": "users", "_id": "AUxOBeJbue-jeR3jNcHj", "_version": 2, "created": false }
30. $ curl -XGET localhost:9200/my-app/users/AUxOBeJbue-jeR3jNcHj { "_index": "my-app", "_type": "users", "_id": "AUxOJe-4ue-jeR3jNcHo", "_version": 2, "found": true, "_source": { "email": "john@smith.com", "first_name": "John" } }
31. $ curl -XDELETE localhost:9200/my-app/users/AUxOBeJbue-jeR3jNcHj { "found": true, "_index": "my-app", "_type": "users", "_id": "AUxOBeJbue-jeR3jNcHj", "_version": 3 }
32. searching
33. $ curl -XPOST localhost:9200/my-app/users/_search -d ‘{ "query": { "match": { "first_name": "John" } } }’
34. { ..., "hits": { "total": 4, "max_score": 0.30685282, "hits": [ { "_index": "my-app", "_type": "users", "_id": "AUxOJWlCue-jeR3jNcHn", "_score": 0.30685282, "_source": { "email": "john@smith.com", "first_name": "John", "last_name": "Doe", "info": { "bio": "Like rock music", "age": 35, "interests": [ "dolphins", "whales" ] }, "join_date": "2014/05/01" } },...
35. $ curl -XPOST localhost:9200/my-app/users/_search -d ‘{ "query": { "filtered": { "query": { "match": { "first_name": "John" } }, "filter": { "range": { "info.age": { "gt": 34 } } } } } }’
36. { "took": 6, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 1, "max_score": 0.30685282, "hits": [ { "_index": "my-app", "_type": "users", "_id": "AUxOJWlCue-jeR3jNcHn", "_score": 0.30685282, "_source": { "email": "john@smith.com", "first_name": "John", "last_name": "Doe", "info": { "bio": "Like rock music", "age": 35, "interests": ["dolphins","whales"] }, "join_date": "2014/05/01" } } ] } }
37. queries vs. filters
38. queries vs. filters
39. full-text search
40. $ curl -XPOST localhost:9200/my-app/users/_search -d ‘{ "query": { "match": { "info.bio": "rock climbing" } } }’
41. { "took": 9, "timed_out": false, "_shards": {...}, "hits": { "total": 2, "max_score": 0.2169777, "hits": [ { "_index": "my-app", "_type": "users", "_id": "AUxOJe-4ue-jeR3jNcHo", "_score": 0.2169777, "_source": { "email": "john@smith.com", "first_name": "John", "last_name": "Kennedy", "info": { "bio": "I love rock climbing!", "age": 29, "interests": ["dolphins","whales"] }, "join_date": "2014/05/01" } }, { "_index": "my-app", "_type": "users", "_id": "AUxOJWlCue-jeR3jNcHn", "_score": 0.02250402, "_source": { "email": "john@smith.com", "first_name": "John", "last_name": "Doe", "info": { "bio": "Like rock music", "age": 35, "interests": ["dolphins","whales"] }, "join_date": "2014/05/01" } } ] } }
42. $ curl -XPOST localhost:9200/my-app/users/_search -d ‘{ "query": { "match_phrase": { "info.bio": "rock climbing" } } }’
43. {…, "hits": { "total": 1, "max_score": 0.30685282, "hits": [ { "_index": "my-app", "_type": "users", "_id": "AUxOJe-4ue-jeR3jNcHo", "_score": 0.30685282, "_source": { "email": "john@smith.com", "first_name": "John", "last_name": "Kennedy", "info": { "bio": "I love rock climbing!", "age": 29, "interests": ["dolphins","whales"] }, "join_date": "2014/05/01" } } ] } }
44. $ curl -XPOST localhost:9200/my-app/users/_search -d ‘{ "query": { "match_phrase": { "info.bio": "rock climbing" } }, "highlight": { "fields": { "info.bio": {} } } }’
45. ... { "_index": "my-app", "_type": "users", "_id": "AUxOJe-4ue-jeR3jNcHo", "_score": 0.30685282, "_source": { "email": "john@smith.com", "first_name": "John", "last_name": "Kennedy", "info": { "bio": "I love rock climbing!", "age": 29, "interests": ["dolphins","whales"] }, "join_date": "2014/05/01" }, "highlight": { "info.bio": ["I love <em>rock</em> <em>climbing</em>!"] } } ] } }
46. is this it?
49. analytics
50. aggregations
51. buckets metrics
52. $ curl -XPOST localhost:9200/my-app/users/_search?search_type=count -d'{ "aggs": { "all_interests": { "terms": { "field": "info.interests" } } } }'
53. { ..., "hits": {"total": 4,"max_score": 0,"hits": []}, "aggregations": { "all_interests": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "whales", "doc_count": 4 }, { "key": "dolphins", "doc_count": 3 } ] } }}
54. $ curl -XPOST localhost:9200/my-app/users/_search?search_type=count -d'{ "aggs": { "all_interests": { "terms": { "field": "info.interests" } } } }'
55. $ curl -XPOST localhost:9200/my-app/users/_search?search_type=count -d'{ "aggs": { "all_interests": { "terms": { "field": "info.interests" }, "aggs": { "avg_age": { "avg": { "field": "info.age" } } } } } }'
56. { …, "aggregations": { "all_interests": { …, "buckets": [ { "key": "whales", "doc_count": 4, "avg_age": { "value": 31.5 } }, { "key": "dolphins", "doc_count": 3, "avg_age": { "value": 32.333333333333336 } } ] }}}
57. $ curl -XPOST localhost:9200/my-app/users/_search?search_type=count -d'{ "query": { "match_phrase": { "info.bio": "rock climbing" } }, "aggs": { "all_interests": { "terms": { "field": "info.interests" }, "aggs": { "avg_age": { "avg": { "field": "info.age" } } } } }}'
58. { …, "aggregations": { "all_interests": { …, "buckets": [ { "key": "whales", "doc_count": 1, "avg_age": { "value": 29 } } ] }}}
60. questions?
61. Resources ➔ http://www.elastic. co/guide/en/elasticsearch/guide/current/index. html ➔ https://leanpub.com/elasticsearch-quick-start ➔ https://www.elastic.co/use-cases ➔ https://speakerdeck.com/asm89/elasticsearch
62. Resources ➔ https://speakerdeck. com/johnbeynon/elasticsearch-101 ➔ http://blog.madewithlove.be/post/integrating- elasticsearch-with-your-laravel-app/ ➔ http://blog.madewithlove.be/post/elasticsearch- aggregations/