Elasticsearch for Lua Developers

This talk will cover the details of the Elasticsearch client we built for Lua as a part of the GSoC program in the LabLua organization.
By using the elasticsearch-lua client a programmer can access most Elasticsearch functionalities and benefit from: proper load balancing across all nodes with pluggable and multiple selection strategies; a connection pool; and the reindex feature (not available in Elasticsearch).
We also explain how this client could be used to implement a search feature and create a dashboard for analysis in a website that is based on the MySQL database.

Transcript

1. Elasticsearch for Lua Developers Pablo Musa pablo@elastic.co
2. + +
3. Me Pablo Musa Educational Engineer @ Elastic
4. www.elastic.co 4 • 5 interested students
 • 3 very good proposals
 • Key Points: - Background (Lua, Elasticsearch, Open Source) - Availability (how many hours per week) - Future (development likelihood after GSoC) Which student?
5. Dhaval Kapil 3rd year computer science student at IIT Roorkee Passionate about networking, security and development Active participant in CTFs. Also, a developer and challenge setter of Backdoor CTF. (http://backdoor.sdslabs.co/) Mostly self-taught himself by reading books and online articles. https://dhavalkapil.com/
6. www.elastic.co 6 • Elasticsearch • Elasticsearch & Lua • elasticsearch-lua • WERpapers • Elastic Agenda
7. Elasticsearch is ...
8. www.elastic.co 8 an open source, distributed, scalable, highly available, document-oriented, RESTful full text search engine with soft real-time search and analytics capabilities Elasticsearch is...
9. www.elastic.co 9 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is... Apache 2.0 License https://www.apache.org/licenses/LICENSE-2.0
10. www.elastic.co 10 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
11. www.elastic.co 11 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
12. www.elastic.co 12 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
13. www.elastic.co 13 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is... { "name" : "Webinar" "geo" : { "city" : "Amsterdam", "lat" : 4.85, "lon" : 52.34 } } Source:  http://json.org/
14. www.elastic.co 14 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is... Source:  https://httpwg.github.io/asset/http.svg
15. www.elastic.co 15 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
16. www.elastic.co 16 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
17. www.elastic.co 17 Getting up and running... is easy $ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.tar.gz $ tar -zxf elasticsearch-1.7.1.tar.gz $ cd elasticsearch-1.7.1 $ ./bin/elasticsearch http://localhost:9200
18. Elasticsearch & Lua
19. www.elastic.co 19 About the Project • 3-month project • A "complete" Lua client to access the Elasticsearch REST API • Influence of official clients • Respect the student
20. "Nobody should have a reason not to use the client" We have no opinions Respect the language Elasticsearch Clients Design Principles
21. www.elastic.co 21 Much more than just a "JSON and HTTP wrapper" • Distributed systems - sniffer - selector • Add Features - reindex - common syntax
22. www.elastic.co 22 High Level Architecture Client Sniffer Transport Connection  Pool Selector ... Random RoundRobin Connection Connection Connection other  http  lib luasocket Parser lua-­‐yajl cjson Helper
23. elasticsearch-lua
24. www.elastic.co 24 Lua vs JSON { title = "A Requirements Elicitation Approach Based in Templates and Patterns", author = "A. Durán Toro", add_authors = { "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" }, key_words = { "requirements engineering", "requirements elicitation" }, language = "en", conference = "WER99", } { "title" : "A Requirements Elicitation Approach Based in Templates and Patterns", "author" : "A. Durán Toro", "add_authors" : [ "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" ], "key_words" : [ "requirements engineering", "requirements elicitation" ], "language" : "en", "conference" : "WER99", }
25. www.elastic.co 25 CRUD local paper = { title = "A Requirements Elicitation Approach Based in Templates and Patterns", ... } local elasticsearch = require"elasticsearch" local client = elasticsearch.client() -- use default configs local r, e = client:index({ index = "papers", type = "wer", id = 1, body = paper }) local r, e = client:get( { index = "papers", type = "wer", id = 1 } ) local r, e = client:delete( { index = "papers", type = "wer", id = 1 } ) local r, e = client:update({ index = "papers", type = "wer", id = 1, body = { doc = { title = "Updated Title" } } })
26. www.elastic.co 26 Searching local r, e = client:search{ index = "papers", type = "wer", body = { query = { filtered = { query = { multi_match = { query = "requirements", fields = { "key_words^10", "title^5", "abstract" } } }, filter = { range = { year = { gte = "now-3y" } } } } } } } local r, e = client:search( { index = "papers", type = "wer", q = "requirements" } ) local r, e = client:search( { index = "papers", type = "wer", q = "title:requirements"}) { took = 1.0, timed_out = false, _shards = { total = 5.0, successful = 5.0, failed = 0.0 }, hits = { total = 1.0, max_score = 0.17673586, hits = [ { _index = "papers", "_type": "wer", "_id": "1", _score = 0.17673586, _source = { title = "A Requirements Elicitation Approach Based in Templates and Patterns", key_words = { "requirements engineering", "requirements elicitation" }, year = "2013" } } ] } }
27. Why Elasticsearch?
28. www.elastic.co 28 Workshop on Requirements Engineering Ibero-american community of researchers on Requirements Engineering
29. www.elastic.co 29 WERpapers • Dedicated site to the WER publications • Apache + CGILua + MySQL + Google Search
30. • Slow • No text analysis • No scoring • No auto-completion • ... Not for full text search
32. www.elastic.co • Implement your own search result page • Relevancy based on downloads, title, abstract, etc. • Auto-complete, Highlighting, Geo, etc. • Everything in Lua 32 Elasticsearch
33. Elasticsearch is not Alone
34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 34 Elastic 2012 2013 2014 2015 Found Elasticsearch as a Service Elasticsearch Store, search, analyze Logstash Collect, parse and enrich data Kibana Visualize and explore data
36. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 36 • Forums for all products • Issue tracker • User groups & Meetups Community, community, community
37. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 37 • Forums for all products • Issue tracker • User groups & Meetups Community, community, community
38. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 38 • Forums for all products • Issue tracker • User groups & Meetups Community, community, community
39. www.elastic.co 39 Next Steps +Tests +Real applications +Use cases Automate Integration with Elasticsearch Make it an official client
40. www.elastic.co 40 Pablo Musa pablo@elastic.co Thanks for listening! Questions?
41. www.elastic.co 41 References https://github.com/DhavalKapil/elasticsearch-lua https://dhavalkapil.com/blogs/Elasticsearch-Lua/ http://lua-users.org/wiki/JsonModules http://wer.inf.puc-rio.br/WERpapers/