elasticsearch-lua Building Lua Applications on Top of Elasticsearch

https://fosdem.org/2016/schedule/event/building_lua_applications_on_top_of_elasticsearch/

Elasticsearch is a distributed and scalable data platform written in Java that, besides the transport protocol (Java to Java), offers a very complete REST API accessed through JSON. This talk will cover the details of the Elasticsearch client we built for Lua as a part of the GSoC program in the LabLua organization.

By using the elasticsearch-lua client a programmer can access most Elasticsearch functionalities and benefit from: proper load balancing across all nodes with pluggable and multiple selection strategies; a connection pool; and the reindex feature (not available in Elasticsearch).

We will also show how this client could be used to implement a search feature in a website that is based on a SQL database, and how can we use the data in Elasticsearch to also perform fast analytics.

Transcript

1. elasticsearch-lua
 Building Lua Applications on Top of Elasticsearch Pablo Musa @pablitomusa
2. www.elastic.co 2 an open source, document-oriented, RESTful full text search engine with soft real-time analytics capabilities Elasticsearch is...
3. www.elastic.co 3 an open source, document-oriented, RESTful, full text search engine with
 soft real-time analytics capabilities Elasticsearch is... Apache 2.0 License https://www.apache.org/licenses/LICENSE-2.0
4. www.elastic.co 4 an open source, document-oriented, RESTful, full text search engine with
 soft real-time analytics capabilities Elasticsearch is... { "name" : "Webinar" "geo" : { "city" : "Amsterdam", "lat" : 4.85, "lon" : 52.34 } } Source: http://json.org/
5. www.elastic.co 5 an open source, document-oriented, RESTful, full text search engine with
 soft real-time analytics capabilities Elasticsearch is... Source: https://httpwg.github.io/asset/http.svg
6. www.elastic.co 6 an open source, document-oriented, RESTful, full text search engine with
 soft real-time analytics capabilities Elasticsearch is...
7. www.elastic.co 7 an open source, document-oriented, RESTful, full text search engine with
 soft real-time analytics capabilities Elasticsearch is...
8. www.elastic.co 8 Getting up and running... is easy $ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-2.1.1.tar.gz $ tar -zxf elasticsearch-2.1.1.tar.gz $ cd elasticsearch-2.1.1 $ ./bin/elasticsearch http://localhost:9200
9. elasticsearch-lua
10. www.elastic.co 10 Lua vs JSON { title = "A Requirements Elicitation Approach Based in Templates and Patterns", author = "A. Durán Toro", add_authors = { "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" }, key_words = { "requirements engineering", "requirements elicitation" }, language = "en", conference = "WER99", } { "title" : "A Requirements Elicitation Approach Based in Templates and Patterns", "author" : "A. Durán Toro", "add_authors" : [ "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" ], "key_words" : [ "requirements engineering", "requirements elicitation" ], "language" : "en", "conference" : "WER99", }
11. www.elastic.co 11 CRUD local paper = { title = "A Requirements Elicitation Approach Based in Templates and Patterns", ... } local elasticsearch = require"elasticsearch" local client = elasticsearch.client() -- use default configs local r, e = client:index({ index = "wer", type = "papers", id = 1, body = paper }) local r, e = client:get( { index = "wer", type = "papers", id = 1 } ) local r, e = client:delete( { index = "wer", type = "papers", id = 1 } ) local r, e = client:update({ index = "wer", type = "papers", id = 1, body = { doc = { title = "Updated Title" } } })
12. www.elastic.co 12 Search local r, e = client:search{ index = "wer", type = "papers", body = { query = { filtered = { query = { multi_match = { query = "requirements", fields = { "key_words^10", "title^5", "abstract" } } }, filter = { range = { year = { gte = "now-3y" } } } } } } } local r, e = client:search( { index = "wer", type = "papers", q = "requirements" } ) local r, e = client:search( { index = "wer", type = "papers", q = "title:requirements"})
13. www.elastic.co 13 Search local r, e = client:search{ index = "papers", type = "wer", body = { query = { filtered = { query = { multi_match = { query = "requirements", fields = { "key_words^10", "title^5", "abstract" } } }, filter = { range = { year = { gte = "now-3y" } } } } } } } local r, e = client:search( { index = "papers", type = "wer", q = "requirements" } ) local r, e = client:search( { index = "papers", type = "wer", q = "title:requirements"}) { took = 1.0, timed_out = false, _shards = { total = 5.0, successful = 5.0, failed = 0.0 }, hits = { total = 1.0, max_score = 0.17673586, hits = [ { _index = "wer", "_type": "papers", "_id": "1", _score = 0.17673586, _source = { title = "A Requirements Elicitation Approach Based in Templates and Patterns", key_words = { "requirements engineering", "requirements elicitation" }, year = "2013", ... } } ] } }
14. Why Elasticsearch?
15. www.elastic.co 15
16. www.elastic.co 16
17. www.elastic.co 17
18. www.elastic.co 18 WERpapers Stack • Apache + CGILua + MySQL + Google Search • Apache + Sailor + Elasticsearch
19. www.elastic.co 19 VS.
20. www.elastic.co 20 • Elasticsearch is not a Database! • Do we really need a Database? VS.
21. www.elastic.co 21 VS. • MySQL is not a Search Engine! • Do we really need a Search Engine?
23. www.elastic.co • Implement your own search result page • Relevancy based on downloads, title, abstract, etc. • Auto-complete, Highlighting, Geo, etc. • Everything in Lua 23 Elasticsearch
24. Moving to Elasticsearch!
25. www.elastic.co 25 MySQL => Elasticsearch
26. www.elastic.co 26 • Denormalize your data • Nested objects • Parent/Child Data Modeling
27. www.elastic.co 27 Mappings ("Schema")
28. www.elastic.co 28 Minimal code changes
29. www.elastic.co 29 Modify search box
32. www.elastic.co 32 Search Is it that simple?
33. Elasticsearch is not Alone
34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 34 Elastic 2012 2013 2014 2015 Elasticsearch Store, Search & Analyze Logstash Collect, Enrich & Transport Kibana Explore & Visualize Found Elasticsearch as a Service Beats Collect, Parse & Ship 2016
35. www.elastic.co 35 WERpapers Statistics
36. www.elastic.co 36 Next Steps +Tests +Real applications +Use cases Automate Integration with Elasticsearch Make it an official client
37. www.elastic.co 37 Pablo Musa pablo@elastic.co Thanks for listening! Questions?
38. www.elastic.co 38 References https://github.com/DhavalKapil/elasticsearch-lua http://wer.inf.puc-rio.br/WERpapers/ http://wer.inf.puc-rio.br/ http://sailorproject.org/ www.elastic.co/