Nutch2 Scoring Model

Nutch2 Solr Lucene OPIC

Transcript

1. Nutch2 Scoring Model 杨学平 <xueping.yang@gmail.com> 2013/10/18
2. Outline 1.Introduction 2.Webpage Crawl 3.Webpage Score
3. 1.1 What's Nutch • Apache Nutch is a highly extensible and scalable open source web crawler software project. // Official • Two codebases • 1.x // data on HDFS, matured • 2.x // storage abstracted by Gora • Plugins • Modular • Crawl base hadoop, store on hbase/mysql/nosql etc
4. 1.2 Nutch/Solr/Lucene • Nutch: crawl webpage • Solr: search server • Lucene: index and search • Solr base on Lucene • Nutch + Solr = Open Source Online Search Engine
5. 1.3 Lunece Scoring • Sort order field • Boost control score • Depend boost field • Document boost • Field boost • Influencing search • Index­time boost • Query­time boost • http://lucene.apache.org/core/4_5_0/core/
6. 1.4 Nutch Scoring • Show web page importance • Prepare for search • Base on PageRank • Implement by ScoringFilter plugin
7. 1.5 Scoring Relation • Nutch scoring generate lucene document boost • initial ­> 1.0f • calculate by OPIC indexerScore method: (float)Math.pow(row.getScore(), scorePower) * initScore; // scorePower = 0.5f
8. 2.1 Crawl workflow • Standard Crawling • Inject // first time • Generate • Fetch • Parse • DbUpdate • Index • Custom Crawling • Generate • BatchFile // generate batch webpage key file • Fetch • Parse • DbUpdate // only process batch relations • Index • Implement by hadoop job
9. 2.2 Generate(1) • Creates a new batch // mark row • Mapper • Select Crawlable row – Marker – Normalise – UrlFilter • DomainURLFilter • RegexURLFilter – Fetch Schedule // shouldFetch – Generate sort value • ScoringFilter • Depend on row's scoring // value = score * initScore • Wrire <SelectorEntry, Webpage> for intermediate
10. 2.2 Generate(2) • Partitioner • generate.topN • generate.max.count • SelectorEntryPartitioner – Domain – Host // default – IP • SelectorEntry – sorted by score – best first (reversed order, how to control order? Plugin?)
11. 2.2 Generate(3) • Reducer • counting – generate.topN – generate.max.count • Output to webpage
12. 2.3 Fetch • Fetches all urls which have been marked by the generator with a given batch ID
13. 2.4 Parse • Parses all webpages from a given batch id
14. 2.5 DbUpdate(1) • Updates all rows: • inlinks (backlinks) • fetchtime • correct score
15. 2.5 DbUpdate(2) • Mapper • fetch all outlink ­­> ScoreDatum list – ScoreDatum:outlink+score+anchor • distribute score to outlink // opic • Write row to intermediate – Key:URLWritable:URL+Float.MAX_VALUE – Value:NutchWritable:PageWritable • Write outlink to intermediate – Key:URLWritable:Outlink+Score – Value:NutchWritable:ScoreDatum
16. 2.5 DbUpdate(3) • Job Control • Partitioner – URLOnly – job.setPartitionerClass() • SortComparator – URLScore // base url and score reversed – job.setSortComparatorClass() • GroupComparator – URLOnly – job.setGroupingComparatorClass() • Why do this? – output inlinks by hadoop framework
17. 2.5 DbUpdate(4­1) • Reducer • find origin page and prepare inlink list • determine new row or old row – new row: • initial schedule • initial score // score:0.0f, cash:0.0f • Set status // unfetched – Old row: • set modified status • set fetch schedule • force refetch // maxInterval < fetchInterval • determine gone page/set gone schedule
18. 2.5 DbUpdate(4­2) • Reducer • set inlinks to page • update score • output to webpage
19. 3.1 PageRank vs OPIC • PageRank • Offline • Iteration link matrix • http://ilpubs.stanford.edu:8090/422/ • OPIC(online page imptance computation) • Online • No iteration • http://www2003.org/cdrom/papers/refereed/
20. 3.2 ScoringFilter • compute page score • Plugin • Interface • injectedScore // 1.0f • initialScore // 0.0f why? • generatorSortValue • distributeScoreToOutlinks • updateScore • indexerScore
21. 3.3 OPICScoringFilter(1) • OPICScoringFilter • injectedScore // 1.0f • initialScore // 0.0f, change by updateScore • GeneratorSortValue // same as score • indexerScore • distributeScoreToOutlinks • updateScore
22. 3.3 OPICScoringFilter(2) • OPICScoringFilter • distributeScoreToOutlinks – Get cash on hand, divide it by the number of outlinks and apply – internalScore // internalScoreFactor – externalScore // externalScoreFactor – reset cash to zero
23. 3.3 OPICScoringFilter(3) • OPICScoringFilter • updateScore – Increase the score by a sum of inlinked scores – Score = old score + adjustScore – Cash = adjustScore
24. Problem • Cash • Score