Using Elasticsearch with Ruby on Rails

Transcript

1. Copyright © 2014 Intridea Inc. All rights reserved. Elasticsearch with Rails July 10, 2014 Tom Zeng Director of Engineering tom@intridea.com @tomzeng www.linkedin.com/in/tomzeng
2. What is Elasticsearch ! Elasticsearch is a “flexible and powerful open source, distributed real-time search and analytics engine for the cloud” More than just full text search, it has powerful analytics capability, and can be used as a NoSQL data store Easy to setup, easy to use, easy to scale, easy to maintain Suitable for projects of any size (large and small, cloud or non-cloud) that need full text search and/or analytics, it’s our preferred search engine for Rails apps
3. Elasticsearch Quick Live Demo Use curl to add some data (local Elasticsearch instance at port 9200) ! curl -X DELETE "http://localhost:9200/todos" (clean up the index and start from scratch) ! curl -X POST "http://localhost:9200/todos/task/1" -d '{"title" : "Learn Elasticsearch", "due_date" : "20140710T00:00:00", "done" : true, "tags" : ["seach","backend"]}' ! curl -X POST "http://localhost:9200/todos/task/2" -d '{"title" : "Learn D3 and Backbone", "due_date" : "20140720T00:00:00", "done" : true, "tags" : ["frontend","javascript","visualization"]}' ! curl -X POST "http://localhost:9200/todos/task/3" -d '{"title" : "Learn Rails 4", "due_date" : "20140830T00:00:00", "done" : false, "tags" : ["backend","ruby","rails"]}' ! curl -X POST "http://localhost:9200/todos/task/4" -d '{"title" : "Learn Backbone Marionette", "due_date" : "20140715T00:00:00", "done" : true, "tags" : ["frontend","javascript"]}' !
4. Elasticsearch Quick Live Demo Use curl to query data curl http://localhost:9200/todos/task/1 (use as K/V store) curl http://localhost:9200/todos/_search?pretty&q=done:false curl http://localhost:9200/todos/_search?pretty&q=tags:backend curl http://localhost:9200/todos/_search?pretty&q=title:back* curl -X POST "http://localhost:9200/todos/task/_search?pretty" -d ' { query : { range : { due_date : { from : "20140701", to : "20140715" } } } }' !
5. Elasticsearch Quick Live Demo
6. What is Elasticsearch http://www.elasticsearch.org
7. What is Elasticsearch http://www.elasticsearch.org
8. What is Elasticsearch http://www.elasticsearch.org
9. Who uses Elasticsearch http://www.elasticsearch.org
10. Who uses Elasticsearch - Github Sort Highlight Facets Filters Fulltext Search Pagination
11. Elasticsearch Key Concepts Cluster – A cluster consists of one or more nodes which share the same cluster name. Each cluster has a single master node which is chosen automatically by the cluster and which can be replaced if the current master node fails. Node – A node is a running instance of elasticsearch which belongs to a cluster. Multiple nodes can be started on a single server for testing purposes, but usually you should have one node per server. At startup, a node will use unicast (or multicast, if specified) to discover an existing cluster with the same cluster name and will try to join that cluster. Index – An index is like a ‘database’ in a relational database. It has a mapping which defines multiple types. An index is a logical namespace which maps to one or more primary shards and can have zero or more replica shards. Type – A type is like a ‘table’ in a relational database. Each type has a list of fields that can be specified for documents of that type. The mapping defines how each field in the document is analyzed. http://www.elasticsearch.org/guide/reference/glossary
12. Elasticsearch Key Concepts Document – A document is a JSON document which is stored in elasticsearch. It is like a row in a table in a relational database. Each document is stored in an index and has a type and an id. A document is a JSON object (also known in other languages as a hash / hashmap / associative array) which contains zero or more fields, or key-value pairs. The original JSON document that is indexed will be stored in the _source field, which is returned by default when getting or searching for a document. Field – A document contains a list of fields, or key-value pairs. The value can be a simple (scalar) value (eg a string, integer, date), or a nested structure like an array or an object. A field is similar to a column in a table in a relational database. The mapping for each field has a field ‘type’ (not to be confused with document type) which indicates the type of data that can be stored in that field, eg integer, string, object. The mapping also allows you to define (amongst other things) how the value for a field should be analyzed. Mapping – A mapping is like a ‘schema definition’ in a relational database. Each index has a mapping, which defines each type within the index, plus a number of index-wide settings. A mapping can either be defined explicitly, or it will be generated automatically when a document is indexed http://www.elasticsearch.org/guide/reference/glossary
13. Elasticsearch Key Concepts Shard – A shard is a single Lucene instance. It is a low-level “worker” unit which is managed automatically by elasticsearch. An index is a logical namespace which points to primary and replica shards. Elasticsearch distributes shards amongst all nodes in the cluster, and can move shards automatically from one node to another in the case of node failure, or the addition of new nodes. Primary Shard – Each document is stored in a single primary shard. When you index a document, it is indexed first on the primary shard, then on all replicas of the primary shard. By default, an index has 5 primary shards. You can specify fewer or more primary shards to scale the number of documents that your index can handle. Replica Shard – Each primary shard can have zero or more replicas. A replica is a copy of the primary shard, and has two purposes: 1) increase failover: a replica shard can be promoted to a primary shard if the primary fails. 2) increase performance: get and search requests can be handled by primary or replica shards. ! ! http://www.elasticsearch.org/guide/reference/glossary
14. Elasticsearch Key Concepts ! Elasticsearch SQL ! Index => Database Type => Table Document => Row Field => Column Mapping => Schema Shard => Partition ! !
15. Elasticsearch Installation OS X brew install elasticsearch Ubuntu wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.deb sudo dpkg -i elasticsearch-1.1.1.deb Centos wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.noarch.rpm sudo yum install elasticsearch-1.1.1.noarch.rpm !
16. Elasticsearch Status Check
17. Elasticsearch Cluster Status
18. Elasticsearch Monitoring elasticsearch-head - https://github.com/mobz/elasticsearch-head ! ! ! ! Marvel - http://www.elasticsearch.org/guide/en/marvel/current/#_marvel_8217_s_dashboards Paramedic - https://github.com/karmi/elasticsearch-paramedic Bigdesk - https://github.com/lukas-vlcek/bigdesk/ !
19. Elasticsearch APIs
20. Elasticsearch API Examples Use curl to run the query and facet APIs curl -X POST "http://localhost:9200/todos/_search?pretty=true" -d ' { "query" : { "query_string" : {"query" : "Learn*"} }, "facets" : { "tags" : { "terms" : {"field" : "tags"} } } } ' Facets – todos tagged with keywords javascript: 2 frontend: 2 backend: 2 visualization: 1 ! !
21. Elasticsearch Query DSL http://www.elasticsearch.org
22. Elasticsearch Query DSL Examples http://www.elasticsearch.org
23. Elasticsearch Query DSL Examples
24. Elasticsearch Query DSL Examples http://www.elasticsearch.org
25. Elasticsearch Plugins and Rivers ! Use plugins to extend Elasticsearch functionality elasticsearch-head, paramedic, bigdesk are all plugins ! Rivers are pluggable services that pull and index data into Elasticsearch Rivers are available for mongodb, couchdb, rabitmq, twitter, wikipedia, mysql, and etc ! !
26. Elasticsearch and Hadoop Create an external Hive table using ES query q=china
27. Elasticsearch and Hadoop External Hive table data – wiki articles that reference the word 'china'
28. Elasticsearch and Rails Well supported with the following gems: ! elasticsearch-rails https://github.com/elasticsearch/elasticsearch-rails ! elasticsearch-ruby https://github.com/elasticsearch/elasticsearch-ruby ! searchkick https://github.com/ankane/searchkick ! tire (retire) https://github.com/karmi/retire ! !
29. Elasticsearch and Rails/Ruby
30. Elasticsearch vs Solr ! Feature Parity between Elasticsearch & Solr http://solr-vs-elasticsearch.com/ ! Elasticsearch is easier to use and maintain ! Built from ground up for scale (for all features) ! Solr - not all features are available in Solr Cloud ! ! ! !
31. Advanced Features of Elasticsearch ! Fuzzy and Proximity Search ! Autocomplete (term, phrase, completion, and context suggesters) ! Suggest API ! Geospatial Search (point, bounding box, polygon) ! Plugins to extend functionality ! Scripting in JavaScript, Python, Groovy, and Java ! !
32. Advanced Features of Elasticsearch ! Aggregation (more dimensions than Facets) ! Related Image Search using LIRE (search similar images based on criteria) ! Percolator (index queries & match on data - useful for event alert, i.e. back in stock) ! Re-scoring on query results ! Polymorphic Search ! ! !
33. Elasticsearch the ELK Stack ! Combining the massively popular Elasticsearch, Logstash and Kibana ! End-to-end stack that delivers actionable insights in real-time from almost any type of structured and unstructured data source ! ! !
34. Spree + Elasticsearch - you do e-commerce? ! Elasticsearch compliments or replaces Spree’s built-in search (AR with the ransack gem) ! Two existing gems spree_elasticsearch and spree_elastic ! spree_elasticsearch - replace built-in search - i.e. Product.search, etc spree_elastic - complement built-in search - i.e. Product.elasticsearch ! Both using model Decorators to add the ES search capabilities ! Can build on top of one of them, or use the elasticsearch_rails directly ! !
35. Elasticsearch and Spree - spree_elastic gem
36. Elasticsearch and Spree - spree_elasticsearch gem
37. Elasticsearch Resources ! http://www.elasticsearch.org/overview/ http://www.elasticsearch.org/guide/ https://github.com/elasticsearch/elasticsearch-hadoop https://github.com/mobz/elasticsearch-head http://railscasts.com/episodes/306-elasticsearch-part-1 http://railscasts.com/episodes/307-elasticsearch-part-2 ! ! !
38. ! # Copyright © 2014 Intridea Inc. All rights reserved. BY WORKING REMOTELY 9,816 Hours Saved ACROSS 4 COUNTRIES 31 Employees FOUNDED & STARTED IN 2007 Washington D.C. We Make. Designers, developers and project managers; this is who we are. Building, breaking and solving; this is what we do.
39. Gracias Merci ありがとう Danke 谢谢 Thank You Copyright © 2014 Intridea Inc. All rights reserved.