Elasticsearch - Quick Introduction

Hopper Elasticsearch Hackathon
Boston, MA - Sep 27, 2013

Transcript

1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Elasticsearch
   Quick Introduction
   Hopper Elasticsearch Hackathon
   
   Boston, MA - Sep 27, 2013
   

   View Slide

2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   About Me
   • Igor Motov
   
   • Developer at Elasticsearch Inc.
   
   • Github: imotov
   
   • Twitter: @imotov
   

   View Slide

3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   About Elasticsearch Inc.
   • Founded in 2012
   
   • By the people behind the Elasticsearch and Apache
   Lucene
   
   • http://www.elasticsearch.com
   
   • Headquarters: Amsterdam and Los Altos, CA
   
   • We provide
   
   • Training (public & onsite)
   
   • Development support
   
   • Production support subscription (SLA)
   

   View Slide

4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   About Elasticsearch
   • Real time search and analytics engine
   
   • JSON-oriented, Apache Lucene-based
   
   • Automatic Schema Detection
   
   • Enables control of it when needed
   
   • Distributed
   
   • Scales Up+Out, Highly Available
   
   • Multi-tenancy
   
   • Dynamically create/delete indices
   
   • API centric
   
   • Most functionality is exposed through an API
   

   View Slide

5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Basic Concepts
   • Cluster
   
   • a group of nodes sharing the same set of indices
   
   • Node
   
   • a running Elasticsearch instance (typically JVM process)
   
   • Index
   
   • a set of documents of possibly different types
   
   • stored in one or more shards
   
   • Type
   
   • a set of documents in an index that share the same schema
   
   • Shard
   
   • a Lucene index, allocated on one of the nodes
   

   View Slide

6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Basic Concepts - Document
   • JSON Object
   
   !
   !
   !
   !
   !
   !
   • Identified by index/type/id
   {
   "rank": 21,
   "city": "Boston",
   "state": "Massachusetts",
   "population2010": 617594,
   "land_area": 48.277,
   "density": 12793,
   "ansi": 619463,
   "location": {
   "lat": 42.332,
   "lon": 71.0202
   },
   "abbreviation": "MA"
   }
   

   View Slide

7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Downloading elasticsearch
   • http://www.elasticsearch.org/download/
   Windows Everything else
   

   View Slide

8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   What’s in a distribution?
   .
   ├── LICENSE.txt
   ├── NOTICE.txt
   ├── README.textile
   ├── bin
   │ ├── elasticsearch
   │ ├── elasticsearch.in.sh
   │ └── plugin
   ├── config
   │ ├── elasticsearch.yml
   │ └── logging.yml
   ├── data
   │ └── elasticsearch
   ├── lib
   │ ├── elasticsearch-x.y.z.jar
   │ ├── ...
   │ └──
   └── logs
   ├── elasticsearch.log
   └── elasticsearch_index_search_slowlog.log
   executable scripts
   node config files
   data storage
   libs
   log files
   

   View Slide

9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Configuration (multicast)
   • Configuration config/elasticsearch.yml
   cluster.name: "elasticsearch-imotov"
   unique
   
   name
   

   View Slide

10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Configuration (stand-alone)
    • Configuration config/elasticsearch.yml
    cluster.name: "elasticsearch-imotov"
    network.host: "127.0.0.1"
    discovery.zen.ping.multicast.enabled: false
    discovery.zen.ping.unicast.hosts: ["localhost"]
    unique
    
    name
    listen only
    
    on localhost
    disable
    
    multicast
    search for other
    
    nodes on localhost
    

    View Slide

11. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Starting elasticsearch
    • Foreground
    
    !
    !
    • Background
    $ bin/elasticsearch -f
    $ bin/elasticsearch
    

    View Slide

12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Is it running?
    {
    "ok" : true,
    "status" : 200,
    "name" : "Hensley Fargus",
    "version" : {
    "number" : "0.90.5",
    "build_hash" : "c8714e8e0620b62638f660f6144831792b9dedee",
    "build_timestamp" : "2013-09-17T12:50:20Z",
    "build_snapshot" : false,
    "lucene_version" : "4.4"
    },
    "tagline" : "You Know, for Search"
    }
    $ curl -XGET "http://localhost:9200/?pretty"
    

    View Slide

13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Communicating with Elasticsearch
    • REST API
    
    • Curl
    
    • Ruby
    
    • Python
    
    • PHP
    
    • Perl
    
    • JavaScript (community supported)
    
    • Binary Protocol
    
    • Java
    

    View Slide

14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Pick your client
    • Java
    
    • included in distribution
    
    • Ruby, PHP, Perl, Python
    
    • http://www.elasticsearch.org/blog/unleash-the-clients-
    ruby-python-php-perl/
    
    • Everything Else
    
    • http://www.elasticsearch.org/guide/clients/
    

    View Slide

15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Indexing a document
    $ curl -XPUT "http://localhost:9200/test-data/cities/21" -d '{
    "rank": 21,
    "city": "Boston",
    "state": "Massachusetts",
    "population2010": 617594,
    "land_area": 48.277,
    "density": 12793,
    "ansi": 619463,
    "location": {
    "lat": 42.332,
    "lon": 71.0202
    },
    "abbreviation": "MA"
    }'
    {"ok":true,"_index":"test-data","_type":"cities","_id":"21","_version":1}
    

    View Slide

16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Getting a document
    {
    "_index" : "test-data",
    "_type" : "cities",
    "_id" : "21",
    "_version" : 1,
    "exists" : true, "_source" : {
    "rank": 21,
    "city": "Boston",
    "state": "Massachusetts",
    "population2010": 617594,
    "land_area": 48.277,
    "density": 12793,
    "ansi": 619463,
    "location": {
    "lat": 42.332,
    "lon": 71.0202
    },
    "abbreviation": "MA"
    }
    }
    $ curl -XGET "http://localhost:9200/test-data/cities/21?pretty"
    

    View Slide

17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Updating a document
    $ curl -XPUT "http://localhost:9200/test-data/cities/21" -d '{
    "rank": 21,
    "city": "Boston",
    "state": "Massachusetts",
    "population2010": 617594,
    "population2012": 636479,
    "land_area": 48.277,
    "density": 12793,
    "ansi": 619463,
    "location": {
    "lat": 42.332,
    "lon": 71.0202
    },
    "abbreviation": "MA"
    }'
    {"ok":true,"_index":"test-data","_type":"cities","_id":"21","_version":2}
    

    View Slide

18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Searching
    $ curl -XGET 'http://localhost:9200/test-data/cities/_search?pretty' -d '{
    "query": {
    "match": {
    "city": "Boston"
    }
    }
    }'
    

    View Slide

19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Searching
    {
    "took" : 5,
    "timed_out" : false,
    "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
    },
    "hits" : {
    "total" : 1,
    "max_score" : 6.1357985,
    "hits" : [ {
    "_index" : "test-data",
    "_type" : "cities",
    "_id" : "21",
    "_score" : 6.1357985, "_source" : {"rank":"21","city":"Boston",...}
    } ]
    }
    }
    

    View Slide

20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Range Queries
    $ curl -XGET "http://localhost:9200/test-data/cities/_search?pretty" -d '{
    "query": {
    "range": {
    "population2012": {
    "from": 500000,
    "to": 1000000
    }
    }
    }
    }'
    

    View Slide

21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Boolean Queries
    $ curl -XGET "http://localhost:9200/test-data/cities/_search?pretty" -d '{
    "query": {
    "bool": {
    "should": [{
    "match": { "state": "Texas"}
    }, {
    "match": { "state": "California"}
    }],
    "must": {
    "range": {
    "population2012": {
    "from": 500000,
    "to": 1000000
    }
    }
    },
    "minimum_should_match": 1
    }
    }
    }'
    

    View Slide

22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    MatchAll Query
    $ curl -XGET "http://localhost:9200/test-data/cities/_search?pretty" -d '{
    "query": {
    "match_all": { }
    }
    }'
    

    View Slide

23. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sorting and Paging
    $ curl -XGET "http://localhost:9200/test-data/cities/_search?pretty" -d '{
    "query": {
    "match_all": { }
    },
    "sort": [
    {"state": {"order": "asc"}},
    {"population2010": {"order": "desc"}}
    ],
    "from": 0,
    "size": 20
    }'
    

    View Slide

24. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysis
    • By default string are
    
    • Divided into words (tokens)
    
    • All tokens are converted to lower-case
    
    • English stop words are removed
    
    • a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, no,
    not, of, on, or, such, that, the, their, then, there, these,
    they, this, to, was, will, with
    

    View Slide

25. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysis Example
    • “Elasticsearch is a powerful open source search
    and analytics engine.”
    
    1. elasticsearch
    
    2. powerful
    
    3. open
    
    4. source
    
    5. search
    
    6. analytics
    
    7. engine
    

    View Slide

26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Disabling stopwords in elasticsearch.yml
    index:
    analysis:
    analyzer:
    default:
    type: "custom"
    tokenizer: "standard"
    filters: ["lowercase"]
    

    View Slide

27. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Customizing the mapping
    curl -XPUT 'http://localhost:9200/my_index/' -d '{
    "settings": {
    "index": {
    "number_of_shards": 1,
    "number_of_replicas": 0
    }
    },
    "mappings": {
    "my_type": {
    "properties": {
    "description": { "type": "string" },
    "sku": { "type": "string", "index": "not_analyzed" },
    "count": { "type": "integer" },
    "price": { "type": "float" },
    "location": { "type": "geo_point" }
    }
    }
    }
    }'
    exact
    
    match
    analyzed
    
    text
    geo
    
    location
    

    View Slide

28. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch Reference
    • http://www.elasticsearch.org/guide/
    

    View Slide

29. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Ideas for hackathon
    • Explore data
    
    • wikipedia
    
    • twitter
    
    • enron emails
    
    • Play with Kibana
    
    • Build Elasticsearch plugins
    
    • Get elasticsearch T-shirt
    

    View Slide

30. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch Meetup
    http://www.meetup.com/Elasticsearch-Boston/
    

    View Slide

31. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    We are hiring
    http://www.elasticsearch.com/about/jobs/
    

    View Slide