Search Engines using Python and Elasticsearch

Transcript

1. Search Engines using Python and Elasticsearch
2. ABOUT ME https://spekerdeck.com/pattyvader/pybr12
3. Athena: my project https://github.com/pattyvader/athena
4. Roadmap 1. Documents search 2. Indexing 3. Crawling the web
5. Documents search
6. Documents search Store result in Elasticsearch Search term
7. Documents search Django server GET Browser (HTML)
8. Django server Documents search GET Browser (HTML)
9. Documents search . Django server
10. Documents search views.py Call “search” function Django server
11. Documents search views.py Call “search” function Invoke “search_term” function Django server
12. Indexing https://www.elastic.co/downloads/elasticsearch Relevancy score Restful protocol Json messages
13. Indexing Relevance score
14. Indexing Restful/Json PUT GET
15. Indexing https://www.elastic.co/use-cases
16. Indexing In the indexing stage, we use a library called Elasticsearch-py to integrate Elasticsearch and Python. indexer.py https://pypi.python.org/pypi/elasticsearch https://elasticsearch-py.readthedocs.io/en/master/ https://github.com/elastic/elasticsearch-py Creates the index Adds a new document to the index
17. indexing scraper.py The scraper extracts data from HTML files using the library BeautifulSoup. https://www.crummy.com/software/BeautifulSoup/
18. scraper.py indexing HTML metatags
19. Crawling the web crawler.py
20. Crawling the web 1 3 2 4 5 Read robot.txt Download HTML Extract new links Extract data Insert data in elasticsearch indexer.py scraper.py crawler.py
21. Crawling the web Before crawling a site, always check for permission in the robot.txt file. It is a good practice. crawler.py
22. Crawling the web The library “urllib2” have functions that helps to extract HTML tags for a given URL. crawler.py
23. Crawling the web crawler.py New links are extracted only in the url seed domain.
24. Crawling the web Extracting data scraper.py crawler.py
25. Crawling the web Indexing documents in Elasticsearch. indexer.py crawler.py
26. Summing up... Browser (HTML) GET Django server scraper.py crawler.py indexer.py GET internet
27. https://github.com/pattyvader https://br.linkedin.com/in/patricia-regina-18790040 Contact *Designed by Freepik from www.flaticon.com* pattyvaderbr@gmail.com