Elasticsearch for Lua Developers

This talk will cover the details of the Elasticsearch client we built for Lua as a part of the GSoC program in the LabLua organization.
By using the elasticsearch-lua client a programmer can access most Elasticsearch functionalities and benefit from: proper load balancing across all nodes with pluggable and multiple selection strategies; a connection pool; and the reindex feature (not available in Elasticsearch).
We also explain how this client could be used to implement a search feature and create a dashboard for analysis in a website that is based on the MySQL database.

Transcript

1. Elasticsearch for Lua Developers
   Pablo Musa
   pablo@elastic.co
   

   View Slide

2. + +
   

   View Slide

3. Me
   Pablo Musa
   Educational Engineer @ Elastic
   

   View Slide

4. www.elastic.co
   4
   • 5 interested students

   • 3 very good proposals

   • Key Points:
   - Background (Lua, Elasticsearch, Open Source)
   - Availability (how many hours per week)
   - Future (development likelihood after GSoC)
   Which student?
   

   View Slide

5. Dhaval Kapil
   3rd year computer
   science student at
   IIT Roorkee
   Passionate
   about
   networking,
   security and
   development
   Active participant in CTFs. Also, a
   developer and challenge setter of
   Backdoor CTF.
   (http://backdoor.sdslabs.co/)
   Mostly self-taught
   himself by reading
   books and online
   articles.
   https://dhavalkapil.com/
   

   View Slide

6. www.elastic.co
   6
   • Elasticsearch
   • Elasticsearch & Lua
   • elasticsearch-lua
   • WERpapers
   • Elastic
   Agenda
   

   View Slide

7. Elasticsearch is ...
   

   View Slide

8. www.elastic.co
   8
   an open source, distributed, scalable,
   highly available, document-oriented, RESTful
   full text search engine
   with soft real-time search and analytics capabilities
   Elasticsearch is...
   

   View Slide

9. www.elastic.co
   9
   an open source, distributed, scalable, highly available, document-oriented,
   RESTful, full text search engine with real-time search and analytics capabilities
   Elasticsearch is...
   Apache 2.0 License
   https://www.apache.org/licenses/LICENSE-2.0
   

   View Slide

10. www.elastic.co
    10
    an open source, distributed, scalable, highly available, document-oriented,
    RESTful, full text search engine with real-time search and analytics capabilities
    Elasticsearch is...
    

    View Slide

11. www.elastic.co
    11
    an open source, distributed, scalable, highly available, document-oriented,
    RESTful, full text search engine with real-time search and analytics capabilities
    Elasticsearch is...
    

    View Slide

12. www.elastic.co
    12
    an open source, distributed, scalable, highly available, document-oriented,
    RESTful, full text search engine with real-time search and analytics capabilities
    Elasticsearch is...
    

    View Slide

13. www.elastic.co
    13
    an open source, distributed, scalable, highly available, document-oriented,
    RESTful, full text search engine with real-time search and analytics capabilities
    Elasticsearch is...
    {
    "name" : "Webinar"
    "geo" : {
    "city" : "Amsterdam",
    "lat" : 4.85, "lon" : 52.34
    }
    }
    Source:  http://json.org/
    

    View Slide

14. www.elastic.co
    14
    an open source, distributed, scalable, highly available, document-oriented,
    RESTful, full text search engine with real-time search and analytics capabilities
    Elasticsearch is...
    Source:  https://httpwg.github.io/asset/http.svg
    

    View Slide

15. www.elastic.co
    15
    an open source, distributed, scalable, highly available, document-oriented,
    RESTful, full text search engine with real-time search and analytics capabilities
    Elasticsearch is...
    

    View Slide

16. www.elastic.co
    16
    an open source, distributed, scalable, highly available, document-oriented,
    RESTful, full text search engine with real-time search and analytics capabilities
    Elasticsearch is...
    

    View Slide

17. www.elastic.co
    17
    Getting up and running... is easy
    $ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.tar.gz
    $ tar -zxf elasticsearch-1.7.1.tar.gz
    $ cd elasticsearch-1.7.1
    $ ./bin/elasticsearch
    http://localhost:9200
    

    View Slide

18. Elasticsearch & Lua
    

    View Slide

19. www.elastic.co
    19
    About the Project
    • 3-month project
    • A "complete" Lua client to access the Elasticsearch REST API
    • Influence of official clients
    • Respect the student
    

    View Slide

20. "Nobody should have a reason not to use the client"
    We have no opinions
    Respect the language
    Elasticsearch Clients Design Principles
    

    View Slide

21. www.elastic.co
    21
    Much more than just a "JSON and HTTP wrapper"
    • Distributed systems
    - sniffer
    - selector
    • Add Features
    - reindex
    - common syntax
    

    View Slide

22. www.elastic.co
    22
    High Level Architecture
    Client
    Sniffer
    Transport
    Connection  Pool Selector
    ...
    Random
    RoundRobin
    Connection
    Connection
    Connection
    other  http  lib
    luasocket
    Parser
    lua-­‐yajl
    cjson
    Helper
    

    View Slide

23. elasticsearch-lua
    

    View Slide

24. www.elastic.co
    24
    Lua vs JSON
    {
    title = "A Requirements Elicitation Approach Based in Templates and Patterns",
    author = "A. Durán Toro",
    add_authors = { "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" },
    key_words = { "requirements engineering", "requirements elicitation" },
    language = "en",
    conference = "WER99",
    }
    {
    "title" : "A Requirements Elicitation Approach Based in Templates and Patterns",
    "author" : "A. Durán Toro",
    "add_authors" : [ "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" ],
    "key_words" : [ "requirements engineering", "requirements elicitation" ],
    "language" : "en",
    "conference" : "WER99",
    }
    

    View Slide

25. www.elastic.co
    25
    CRUD
    local paper = {
    title = "A Requirements Elicitation Approach Based in Templates and Patterns",
    ...
    }
    local elasticsearch = require"elasticsearch"
    local client = elasticsearch.client() -- use default configs
    local r, e = client:index({
    index = "papers", type = "wer", id = 1, body = paper
    })
    local r, e = client:get( { index = "papers", type = "wer", id = 1 } )
    local r, e = client:delete( { index = "papers", type = "wer", id = 1 } )
    local r, e = client:update({
    index = "papers", type = "wer", id = 1, body = { doc = { title = "Updated Title" } }
    })
    

    View Slide

26. www.elastic.co
    26
    Searching
    local r, e = client:search{
    index = "papers",
    type = "wer",
    body = {
    query = {
    filtered = {
    query = {
    multi_match = {
    query = "requirements", fields = { "key_words^10", "title^5", "abstract" }
    } },
    filter = { range = { year = { gte = "now-3y" } } }
    } } }
    }
    local r, e = client:search( { index = "papers", type = "wer", q = "requirements" } )
    local r, e = client:search( { index = "papers", type = "wer", q = "title:requirements"})
    {
    took = 1.0, timed_out = false,
    _shards = { total = 5.0, successful = 5.0, failed = 0.0 },
    hits = {
    total = 1.0, max_score = 0.17673586,
    hits = [ {
    _index = "papers", "_type": "wer", "_id": "1",
    _score = 0.17673586,
    _source = {
    title = "A Requirements Elicitation Approach Based in Templates and Patterns",
    key_words = { "requirements engineering", "requirements elicitation" },
    year = "2013"
    }
    }
    ]
    }
    }
    

    View Slide

27. Why Elasticsearch?
    

    View Slide

28. www.elastic.co
    28
    Workshop on Requirements Engineering
    Ibero-american community of researchers on Requirements Engineering
    

    View Slide

29. www.elastic.co
    29
    WERpapers
    • Dedicated site to the WER publications
    • Apache + CGILua + MySQL + Google Search
    

    View Slide

30. • Slow
    • No text analysis
    • No scoring
    • No auto-completion
    • ...
    Not for full text search
    

    View Slide

31. View Slide

32. www.elastic.co
    • Implement your own search result page
    • Relevancy based on downloads, title, abstract, etc.
    • Auto-complete, Highlighting, Geo, etc.
    • Everything in Lua
    32
    Elasticsearch
    

    View Slide

33. Elasticsearch is not Alone
    

    View Slide

34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    34
    Elastic
    2012 2013 2014 2015
    Found
    Elasticsearch as a Service
    Elasticsearch
    Store, search, analyze
    Logstash
    Collect, parse and enrich data
    Kibana
    Visualize and explore data
    

    View Slide

35. View Slide

36. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    36
    • Forums for all products
    • Issue tracker
    • User groups & Meetups
    Community, community, community
    

    View Slide

37. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    37
    • Forums for all products
    • Issue tracker
    • User groups & Meetups
    Community, community, community
    

    View Slide

38. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    38
    • Forums for all products
    • Issue tracker
    • User groups & Meetups
    Community, community, community
    

    View Slide

39. www.elastic.co
    39
    Next Steps
    +Tests
    +Real applications
    +Use cases
    Automate Integration with Elasticsearch
    Make it an official client
    

    View Slide

40. www.elastic.co
    40
    Pablo Musa
    pablo@elastic.co
    Thanks for listening! Questions?
    

    View Slide

41. www.elastic.co
    41
    References
    https://github.com/DhavalKapil/elasticsearch-lua
    https://dhavalkapil.com/blogs/Elasticsearch-Lua/
    http://lua-users.org/wiki/JsonModules
    http://wer.inf.puc-rio.br/WERpapers/
    

    View Slide