Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch for Lua Developers

A8c1cd7556870cf906064041cc5db121?s=47 Pablo Musa
October 16, 2015

Elasticsearch for Lua Developers

This talk will cover the details of the Elasticsearch client we built for Lua as a part of the GSoC program in the LabLua organization.
By using the elasticsearch-lua client a programmer can access most Elasticsearch functionalities and benefit from: proper load balancing across all nodes with pluggable and multiple selection strategies; a connection pool; and the reindex feature (not available in Elasticsearch).
We also explain how this client could be used to implement a search feature and create a dashboard for analysis in a website that is based on the MySQL database.

A8c1cd7556870cf906064041cc5db121?s=128

Pablo Musa

October 16, 2015
Tweet

Transcript

  1. Elasticsearch for Lua Developers Pablo Musa pablo@elastic.co

  2. + +

  3. Me Pablo Musa Educational Engineer @ Elastic

  4. www.elastic.co 4 • 5 interested students
 • 3 very good

    proposals
 • Key Points: - Background (Lua, Elasticsearch, Open Source) - Availability (how many hours per week) - Future (development likelihood after GSoC) Which student?
  5. Dhaval Kapil 3rd year computer science student at IIT Roorkee

    Passionate about networking, security and development Active participant in CTFs. Also, a developer and challenge setter of Backdoor CTF. (http://backdoor.sdslabs.co/) Mostly self-taught himself by reading books and online articles. https://dhavalkapil.com/
  6. www.elastic.co 6 • Elasticsearch • Elasticsearch & Lua • elasticsearch-lua

    • WERpapers • Elastic Agenda
  7. Elasticsearch is ...

  8. www.elastic.co 8 an open source, distributed, scalable, highly available, document-oriented,

    RESTful full text search engine with soft real-time search and analytics capabilities Elasticsearch is...
  9. www.elastic.co 9 an open source, distributed, scalable, highly available, document-oriented,

    RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is... Apache 2.0 License https://www.apache.org/licenses/LICENSE-2.0
  10. www.elastic.co 10 an open source, distributed, scalable, highly available, document-oriented,

    RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
  11. www.elastic.co 11 an open source, distributed, scalable, highly available, document-oriented,

    RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
  12. www.elastic.co 12 an open source, distributed, scalable, highly available, document-oriented,

    RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
  13. www.elastic.co 13 an open source, distributed, scalable, highly available, document-oriented,

    RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is... { "name" : "Webinar" "geo" : { "city" : "Amsterdam", "lat" : 4.85, "lon" : 52.34 } } Source:  http://json.org/
  14. www.elastic.co 14 an open source, distributed, scalable, highly available, document-oriented,

    RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is... Source:  https://httpwg.github.io/asset/http.svg
  15. www.elastic.co 15 an open source, distributed, scalable, highly available, document-oriented,

    RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
  16. www.elastic.co 16 an open source, distributed, scalable, highly available, document-oriented,

    RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
  17. www.elastic.co 17 Getting up and running... is easy $ wget

    https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.tar.gz $ tar -zxf elasticsearch-1.7.1.tar.gz $ cd elasticsearch-1.7.1 $ ./bin/elasticsearch http://localhost:9200
  18. Elasticsearch & Lua

  19. www.elastic.co 19 About the Project • 3-month project • A

    "complete" Lua client to access the Elasticsearch REST API • Influence of official clients • Respect the student
  20. "Nobody should have a reason not to use the client"

    We have no opinions Respect the language Elasticsearch Clients Design Principles
  21. www.elastic.co 21 Much more than just a "JSON and HTTP

    wrapper" • Distributed systems - sniffer - selector • Add Features - reindex - common syntax
  22. www.elastic.co 22 High Level Architecture Client Sniffer Transport Connection  Pool

    Selector ... Random RoundRobin Connection Connection Connection other  http  lib luasocket Parser lua-­‐yajl cjson Helper
  23. elasticsearch-lua

  24. www.elastic.co 24 Lua vs JSON { title = "A Requirements

    Elicitation Approach Based in Templates and Patterns", author = "A. Durán Toro", add_authors = { "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" }, key_words = { "requirements engineering", "requirements elicitation" }, language = "en", conference = "WER99", } { "title" : "A Requirements Elicitation Approach Based in Templates and Patterns", "author" : "A. Durán Toro", "add_authors" : [ "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" ], "key_words" : [ "requirements engineering", "requirements elicitation" ], "language" : "en", "conference" : "WER99", }
  25. www.elastic.co 25 CRUD local paper = { title = "A

    Requirements Elicitation Approach Based in Templates and Patterns", ... } local elasticsearch = require"elasticsearch" local client = elasticsearch.client() -- use default configs local r, e = client:index({ index = "papers", type = "wer", id = 1, body = paper }) local r, e = client:get( { index = "papers", type = "wer", id = 1 } ) local r, e = client:delete( { index = "papers", type = "wer", id = 1 } ) local r, e = client:update({ index = "papers", type = "wer", id = 1, body = { doc = { title = "Updated Title" } } })
  26. www.elastic.co 26 Searching local r, e = client:search{ index =

    "papers", type = "wer", body = { query = { filtered = { query = { multi_match = { query = "requirements", fields = { "key_words^10", "title^5", "abstract" } } }, filter = { range = { year = { gte = "now-3y" } } } } } } } local r, e = client:search( { index = "papers", type = "wer", q = "requirements" } ) local r, e = client:search( { index = "papers", type = "wer", q = "title:requirements"}) { took = 1.0, timed_out = false, _shards = { total = 5.0, successful = 5.0, failed = 0.0 }, hits = { total = 1.0, max_score = 0.17673586, hits = [ { _index = "papers", "_type": "wer", "_id": "1", _score = 0.17673586, _source = { title = "A Requirements Elicitation Approach Based in Templates and Patterns", key_words = { "requirements engineering", "requirements elicitation" }, year = "2013" } } ] } }
  27. Why Elasticsearch?

  28. www.elastic.co 28 Workshop on Requirements Engineering Ibero-american community of researchers

    on Requirements Engineering
  29. www.elastic.co 29 WERpapers • Dedicated site to the WER publications

    • Apache + CGILua + MySQL + Google Search
  30. • Slow • No text analysis • No scoring •

    No auto-completion • ... Not for full text search
  31. None
  32. www.elastic.co • Implement your own search result page • Relevancy

    based on downloads, title, abstract, etc. • Auto-complete, Highlighting, Geo, etc. • Everything in Lua 32 Elasticsearch
  33. Elasticsearch is not Alone

  34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 34 Elastic 2012 2013 2014 2015 Found Elasticsearch as a Service Elasticsearch Store, search, analyze Logstash Collect, parse and enrich data Kibana Visualize and explore data
  35. None
  36. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 36 • Forums for all products • Issue tracker • User groups & Meetups Community, community, community
  37. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 37 • Forums for all products • Issue tracker • User groups & Meetups Community, community, community
  38. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 38 • Forums for all products • Issue tracker • User groups & Meetups Community, community, community
  39. www.elastic.co 39 Next Steps +Tests +Real applications +Use cases Automate

    Integration with Elasticsearch Make it an official client
  40. www.elastic.co 40 Pablo Musa pablo@elastic.co Thanks for listening! Questions?

  41. www.elastic.co 41 References https://github.com/DhavalKapil/elasticsearch-lua https://dhavalkapil.com/blogs/Elasticsearch-Lua/ http://lua-users.org/wiki/JsonModules http://wer.inf.puc-rio.br/WERpapers/