Apache Solr

Introduction about Apache Solr

Transcript

1. Apache Solr Abdulaziz AlShetwi @ecleel
2. Apache Solr Abdulaziz AlShetwi @ecleel with BUILD YOUR OWN SEARCH ENGINE
3. What Features you want in your search server?
4. Scalable Flexible Easy of Use Fast Accept any data Open Source Code-less Extendable Configurable
6. open source enterprise search server built on top of Apache Lucene Apache Solr
7. http://wiki.apache.org/solr/PublicServers
8. Lightweight, No Dependence Java Library for indexing and searching Inverted Index 53% from companies use it. [IDC] Apache Lucene
9. IndexWriter , IndexSearcher , Directory , Analyzer , ... Using Java Objects to deal with index
10. Index Structure Index > Document > Field > Term
11. Solr is Lucene and more ...
13. Indexing
14. Full Index Delta Index
15. Inverted Index
16. Index Every Thing Database, text, PDF, MS WORD, MPEG RSS Feeds, Email, XML, HTML, CSV
17. DIH: Data Input Handler Database CSV XML Files RSS Feeds Email
18. <dataConfig> <dataSource driver="org.hsqldb.jdbcDriver" url="jdbc:hsqldb:/temp/ example/ex" user="sa" /> <document> <entity name="item" query="select * from item"> <entity name="feature" query="select description as features from feature where item_id='${item.ID}'"/> <entity name="item_category" query="select CATEGORY_ID from item_category where item_id='${item.ID}'"> <entity name="category" query="select description as cat from category where id = '${item_category.CATEGORY_ID}'"/> </entity> </entity> </document> </dataConfig> Database
19. <dataConfig> <dataSource type="FileDataSource" encoding="UTF-8" /> <document> <entity name="page" processor="XPathEntityProcessor" stream="true" forEach="/mediawiki/page/" url="/data/enwiki-20080724-pages-articles.xml" transformer="RegexTransformer,DateFormatTransformer"> <field column="id" xpath="/mediawiki/page/id" /> <field column="title" xpath="/mediawiki/page/title" /> <field column="revision" xpath="/mediawiki/page/revision/id" /> <field column="user" xpath="/mediawiki/page/revision/contributor/username" /> <field column="userId" xpath="/mediawiki/page/revision/contributor/id" /> <field column="text" xpath="/mediawiki/page/revision/text" /> <field column="timestamp" xpath="/mediawiki/page/revision/timestamp" dateTimeFormat="yyyy-MM-dd'T'hh:mm <field column="$skipDoc" regex="^#REDIRECT .*" replaceWith="true" sourceColName="text"/> </entity> </document> </dataConfig> XML
20. HTTP Interface
21. Restful-API XML, JSON, Ruby, Python
22. HTTP POST to /update <add> <doc boost=“2”> <field name=“article”>05991</field> <field name=“title”>Apache Solr</field> <field name=“subject”>An intro...</field> <field name=“category”>search</field> <field name=“category”>lucene</field> <field name=“body”>Solr is a full...</field> </doc> </add>
23. Delete by Query (multiple documents) HTTP POST to /update <delete> <id>05591</id> </delete> Delete by Id <delete> <query>manufacturer:microsoft</query> </delete>
24. makes changes visible <commit/> <optimize/> Optimize index segments HTTP POST to /update
25. Querying
26. ttp://localhost:8983/solr/select?q=video&start=0&rows=2&fl=name,price
27. <response> <responseHeader> <status>0</status> <QTime>1</QTime> </responseHeader> <result numFound="16173" start="0"> <doc> <str name="name">Apple 60 GB iPod with Video</str> <float name="price">399.0</float> </doc> <doc> <str name="name">ASUS Extreme N7800GTX/2DHTV</str> <float name="price">479.95</float> </doc> </result> </response>
28. Sorting
29. ... q=video&sort=created_at desc
30. ...q=video&sort=div(popularity,add(price,1)) desc
31. Spatial Search geographical , bioinformatics, fingerprints search, facial search
32. &q=*:*&fq={!geofilt pt=45.15,-93.85 sfield=store d=5}
33. Faceted Search
36. ...&q=*:*&facet=true&facet.field=cat
37. ...&q=ipod&facet=true& facet.query=price:[0 TO 100]&facet.query=price:[100 TO *]
38. ...&q=*:*&facet=true&facet.date=manufacturedate_dt& facet.date.start=2004-01-01T00:00:00Z& facet.date.end=2010-01-01T00:00:00Z& facet.date.gap=+1YEAR
39. Clustering Using Carrot2 framework
40. cluster both search results and documents
42. Highlighting
44. ...&q=video card&fl=name,id&hl=true&hl.fl=name,features
45. Auto Suggestions
46. Spell Checking
47. Boosting text:NeXT ==> (text:NeXT^10 OR text:next)
48. Scaling
49. Highly scalable Millions of documents
50. Caching filterCache, resultCache, documentCache
51. Pagination
52. Large indices Sharding
53. High Query Volume Replication
54. Admin Interface
56. No Code Requirement
57. scheme.xml solr.xml
58. <field name="id" type="string" indexed="true" stored="true" required="true" /> <field name="dedupe" type="string" indexed="true" stored="true" /> <field name="filetype" type="string" indexed="true" stored="true" /> <field name="hostname" type="string" indexed="true" stored="true" /> <field name="filename" type="string" indexed="true" stored="true" /> <field name="category" type="string" indexed="true" stored="false" multiValued="true" /> <field name="popularity" type="double" indexed="true" stored="true" /> <field name="title" type="text" indexed="true" stored="true" termVectors="true" termPositions="true" termOffsets="true" default="No Title Found"/> <field name="text" type="text" indexed="true" stored="true" termVectors="true" termPositions="true" termOffsets="true"/> <field name="timestamp" type="date" indexed="true" stored="true" default="NOW/SECOND" multiValued="false"/> <field name="content_type" type="string" indexed="true" stored="true" multiValued="false"/> <field name="last_modified_date" type="date" indexed="true" stored="true" multiValued="false"/>
59. <analyzer type="query"> <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.SnowballPorterFilterFactory" language="English"/> </analyzer>
60. Easy to Integrate with other Apache’s Projects Hadoop, Nutch, Cassandra ...
61. Active Community
62. Q Q Q Q Q Q Q QQ
63. !ank Y"