You're not using Elasticsearch?

An introduction to Elasticsearch 1.0 and how Tolq uses it for its translation memory

Transcript

1. You’re not using ElasticSearch? Timon Vonk (@timonvonk)
2. About me • CTO @ Tolq, zero effort website translations • Freelance hacker and consultant
4. The battlefield • Ferret • Xapian • Solr • Postgresql’s TSVectors • Sphinx • Many more
5. What is ElasticSearch? • Search engine • Cloud in mind • JSON API • Scriptable • Nosql • Great for things different than search
6. Why use it? • Cloud setup out of the box • Fast indexing • Easy API • On the fly mappings • Very customisable
7. OMG WHAT IS LUCENE • Java library for search • Only handles the search bit • Terms based vector algorithm
8. Making a query • Just send JSON: GET localhost:9200/example/peanuts/_search
 { ‘query’: { text: { ‘my_field’: ’many search terms’ }}} { took: 5, timed_out: false,
 _shards: { total: 5, successful: 5, failed: 0 },
 hits: [
 { _index: “example”,
 _type: “peanuts”,
 _score: 0.9,
 _source: { …data }
 }
 ]
 }
 }
9. Other types of queries • Terms, full text, boolean, fuzzy, geolocation and lots more variants • Filters, aggregations, percolation, suggestions
10. Analysing • Pre-index and pre-search • This is when scoring happens • Remove stop words, stemming, other normalisations • You can create your own analysers
11. Aggregations { “query”: … } { “aggregations”: { “rubyist_stats”: { “stats”: { “field”: “meetup_visits” } } } } Aggregations are an upgrade over < 1.0 facets
12. Different kinds • min, max, avg, sum • stats, extended_stats (all of the above + stdev/ mean) • percentile • counts • … all scriptable!
13. Also buckets! • Create subsets based on conditions • Nest aggregations • … aaand scriptable
14. (Ruby) libraries • Good old Tire
 deprecated • Stretcher! • elasticsearch-ruby • Also libraries for Go, Node, Javascript, etc • … it’s just json
15. How Tolq uses ElasticSearch • Suggest better translations for translators • Fast access to text and translations for general search
16. Other fun stuff • Kibana • Hadoop • Hosted, autoscaling providers
17. Also, it works great for text search! Thanks! We’ve launched! http://www.tolq.com And hiring!