Project Demonstration for CS273 (Fall 2012)

A simple project on Data Integration, between 2 bibliographical databases (DBLP and C-DBLP).

Transcript

1. Data Integration on DBLP and C-DBLP Susen Zhao Tuesday, March 5, 13
2. Introduction Tuesday, March 5, 13
3. Introduction • Data Integration: combining data from different sources and providing users with a unified view of these data Tuesday, March 5, 13
4. Introduction • Data Integration: combining data from different sources and providing users with a unified view of these data • Useful and Interesting Tuesday, March 5, 13
5. Introduction • Data Integration: combining data from different sources and providing users with a unified view of these data • Useful and Interesting • More accurate and effective data Tuesday, March 5, 13
6. Introduction • Data Integration: combining data from different sources and providing users with a unified view of these data • Useful and Interesting • More accurate and effective data • Integrating DBLP and C-DBLP Tuesday, March 5, 13
7. Introduction Tuesday, March 5, 13
8. Introduction • DBLP: a computer science bibliography website; since 1980s; 2.1+ million articles by November 2012 Tuesday, March 5, 13
9. Introduction • DBLP: a computer science bibliography website; since 1980s; 2.1+ million articles by November 2012 • C-DBLP: a computer science (partly) bibliography website in Chinese; tracks 14 computer science journals in China; since 2001; 94,000+ articles by now Tuesday, March 5, 13
10. Project: Overview Tuesday, March 5, 13
11. Project: Overview • Data Integration - 2 approaches Tuesday, March 5, 13
12. Project: Overview • Data Integration - 2 approaches • This project: handle queries in real-time Tuesday, March 5, 13
13. Project: Overview • Data Integration - 2 approaches • This project: handle queries in real-time • Publications data updates everyday; more interesting Tuesday, March 5, 13
14. Project: Overview • Data Integration - 2 approaches • This project: handle queries in real-time • Publications data updates everyday; more interesting • DBLP itself does’t use database either (a large XML file) Tuesday, March 5, 13
15. Project: Overview Tuesday, March 5, 13
16. Project: Overview User query: find author “
    ” Query “
    ” from C-DBLP Map to “Xiaofeng Meng” or “Xiao-feng Meng” Query “Xiaofeng Meng” or “Xiao-feng Meng” from DBLP C-DBLP record 1; C-DBLP record 2; ... DBLP record 1; DBLP record 2; ... Parse Parse Author matching (disambiguation) Final result Tuesday, March 5, 13
17. Project: Overview User query: find author “
    ” Query “
    ” from C-DBLP Map to “Xiaofeng Meng” or “Xiao-feng Meng” Query “Xiaofeng Meng” or “Xiao-feng Meng” from DBLP C-DBLP record 1; C-DBLP record 2; ... DBLP record 1; DBLP record 2; ... Parse Parse Author matching (disambiguation) Final result • A Sample Query: Find author “孟⼩小 峰” (Xiaofeng Meng in English) Tuesday, March 5, 13
18. Project: Overview User query: find author “
    ” Query “
    ” from C-DBLP Map to “Xiaofeng Meng” or “Xiao-feng Meng” Query “Xiaofeng Meng” or “Xiao-feng Meng” from DBLP C-DBLP record 1; C-DBLP record 2; ... DBLP record 1; DBLP record 2; ... Parse Parse Author matching (disambiguation) Final result • A Sample Query: Find author “孟⼩小 峰” (Xiaofeng Meng in English) • Query Tuesday, March 5, 13
19. Project: Overview User query: find author “
    ” Query “
    ” from C-DBLP Map to “Xiaofeng Meng” or “Xiao-feng Meng” Query “Xiaofeng Meng” or “Xiao-feng Meng” from DBLP C-DBLP record 1; C-DBLP record 2; ... DBLP record 1; DBLP record 2; ... Parse Parse Author matching (disambiguation) Final result • A Sample Query: Find author “孟⼩小 峰” (Xiaofeng Meng in English) • Query • Parse/Re-structure Tuesday, March 5, 13
20. Project: Overview User query: find author “
    ” Query “
    ” from C-DBLP Map to “Xiaofeng Meng” or “Xiao-feng Meng” Query “Xiaofeng Meng” or “Xiao-feng Meng” from DBLP C-DBLP record 1; C-DBLP record 2; ... DBLP record 1; DBLP record 2; ... Parse Parse Author matching (disambiguation) Final result • A Sample Query: Find author “孟⼩小 峰” (Xiaofeng Meng in English) • Query • Parse/Re-structure • Author Matching Tuesday, March 5, 13
21. Project: Overview User query: find author “
    ” Query “
    ” from C-DBLP Map to “Xiaofeng Meng” or “Xiao-feng Meng” Query “Xiaofeng Meng” or “Xiao-feng Meng” from DBLP C-DBLP record 1; C-DBLP record 2; ... DBLP record 1; DBLP record 2; ... Parse Parse Author matching (disambiguation) Final result • A Sample Query: Find author “孟⼩小 峰” (Xiaofeng Meng in English) • Query • Parse/Re-structure • Author Matching • Final Result Tuesday, March 5, 13
22. Project: Overview User query: find author “
    ” Query “
    ” from C-DBLP Map to “Xiaofeng Meng” or “Xiao-feng Meng” Query “Xiaofeng Meng” or “Xiao-feng Meng” from DBLP C-DBLP record 1; C-DBLP record 2; ... DBLP record 1; DBLP record 2; ... Parse Parse Author matching (disambiguation) Final result • A Sample Query: Find author “孟⼩小 峰” (Xiaofeng Meng in English) • Query • Parse/Re-structure • Author Matching • Final Result • Author is the Bridge! Tuesday, March 5, 13
23. Project: Data Retrieving Tuesday, March 5, 13
24. Project: Data Retrieving • DBLP: XML APIs, Web pages Tuesday, March 5, 13
25. Project: Data Retrieving • DBLP: XML APIs, Web pages • Request by API and URL Tuesday, March 5, 13
26. Project: Data Retrieving • DBLP: XML APIs, Web pages • Request by API and URL • Get by author, journal/year/issue, journal/ volume, conference/year Tuesday, March 5, 13
27. Project: Data Retrieving • DBLP: XML APIs, Web pages • Request by API and URL • Get by author, journal/year/issue, journal/ volume, conference/year • C-DBLP: Web pages only (and not well- formatted) Tuesday, March 5, 13
28. Project: Data Retrieving • DBLP: XML APIs, Web pages • Request by API and URL • Get by author, journal/year/issue, journal/ volume, conference/year • C-DBLP: Web pages only (and not well- formatted) • Request by URL Tuesday, March 5, 13
29. Project: Data Retrieving • DBLP: XML APIs, Web pages • Request by API and URL • Get by author, journal/year/issue, journal/ volume, conference/year • C-DBLP: Web pages only (and not well- formatted) • Request by URL • Get by author, journal/year/issue Tuesday, March 5, 13
30. Project: Data Parsing Tuesday, March 5, 13
31. Project: Data Parsing • Using XML/HTML parser + RegExp • Re-format Tuesday, March 5, 13
32. Project: Data Parsing • Using XML/HTML parser + RegExp • Re-format publication = { 'title': 'Ranking the Difficulty Level of the Knowledge Units Based on Learning Dependency', 'authors': ['Jun Liu', 'Sha Sha', 'Qinghua Zheng', 'Wei Zhang'], 'venue-type': 'journal', 'venue': 'IJDET', 'year': '2012', 'dblp-key': 'journals/ijdet/LiuSZZ12' } Tuesday, March 5, 13
33. Project: Data Parsing • Using XML/HTML parser + RegExp • Re-format publication = { 'title': 'Ranking the Difficulty Level of the Knowledge Units Based on Learning Dependency', 'authors': ['Jun Liu', 'Sha Sha', 'Qinghua Zheng', 'Wei Zhang'], 'venue-type': 'journal', 'venue': 'IJDET', 'year': '2012', 'dblp-key': 'journals/ijdet/LiuSZZ12' } author = { 'author-name': 'Author Name', 'key': urlpt | URL, 'coauthors': [coauthor1, coauthor2, ...], 'publications': [publication1, publication2, ...] } Tuesday, March 5, 13
34. Project: Author Matching • Naive Approach: by coauthors Tuesday, March 5, 13
35. Project: Author Matching • Naive Approach: by coauthors .i, *."GSmi?Q` UrBi? +Qmi?Q` b2i C1 V- ."GSmi?Q` UrBi? +Qmi?Q` b2i C2 V _2bmHi, h?2 irQ mi?Q`bǶ Ki+? `MF `MF ← yc *QmMi."GS ← yc *QmM/*."GS ← yc C ← C1 ∩ C2 c 7Q` c ∈ C1 /Q B7 c ∈ C i?2M *QmMi*."GS ← *QmMi*."GS Y *Qmi?Q`2/hBK2bU+- *."GSmi?Q`Vc 2M/ 2M/ 7Q` c ∈ C2 /Q B7 c ∈ C i?2M *QmMi."GS ← *QmMi."GS Y *Qmi?Q`2/hBK2bU+- ."GSmi?Q`Vc 2M/ 2M/ `MF ← KtU*QmMi*."GS f bBx2UC1 V Y *QmMi."GS f bBx2UC2 VVc `2im`M `MF H;Q`Bi?K R, *H+mHiBM; irQ mi?Q`bǶ Ki+? `MF h?Bb H;Q`Bi?K +H+mHi2  `MF 7Q` i?2 Ki+? `i2 Q7 irQ mi?Q`b UQM2 BM *@."GS M/ i?2 Qi?2` BM ."GSV- }MHHv B7 i?2 `MF #2ir22M irQ mi?Q`b Bb H`;2 i?M  i?`2b?QH/- A rQmH/ bv i?i i?2b2 irQ mi?Q`b `2 Ki+?2/X 7i2` b2p2`H i2bib- A 7QmM/ i?Bb H;Q`Bi?K 7bi M/ 2{+B2MiX h?Bb H;Q`Bi?K rQ`Fb }M2 #2+mb2 ."GS T`QpB/2b  ;QQ/ /BbK#B;miBQM 7Q` mi?Q`b- #v T2`7Q`KBM; i?Bb H;Q`Bi?K r2 `2 +imHHv 2M?M+BM; i?2 BM7Q`KiBQM 7Q` *@."GS #v mbBM; Tuesday, March 5, 13
36. Project: Sample Queries Tuesday, March 5, 13
37. Project: Sample Queries • Find all publications of an author (in both DBLP and C-DBLP). Tuesday, March 5, 13
38. Project: Sample Queries • Find all publications of an author (in both DBLP and C-DBLP). • Find all coauthors of an author (in both DBLP and C-DBLP) with combined coauthorship. Tuesday, March 5, 13
39. Project: Sample Queries • Find all publications of an author (in both DBLP and C-DBLP). • Find all coauthors of an author (in both DBLP and C-DBLP) with combined coauthorship. • Find all venues (conferences/journals/...) that an author is involved. Tuesday, March 5, 13
40. Project: Sample Queries • Find all publications of an author (in both DBLP and C-DBLP). • Find all coauthors of an author (in both DBLP and C-DBLP) with combined coauthorship. • Find all venues (conferences/journals/...) that an author is involved. • Find all publications coauthored by 2 authors. Tuesday, March 5, 13
41. Project: Sample Queries • Find all publications of an author (in both DBLP and C-DBLP). • Find all coauthors of an author (in both DBLP and C-DBLP) with combined coauthorship. • Find all venues (conferences/journals/...) that an author is involved. • Find all publications coauthored by 2 authors. • Find authors that published in a journal in C-DBLP who have also published in a journal/conference in DBLP. Tuesday, March 5, 13
42. Query Process: Query #5 Tuesday, March 5, 13
43. Query Process: Query #5 Journal query: “
    ” and “SIGMOD” Query “
    ” from C-DBLP Query “SIGMOD” from DBLP C-DBLP author 1; C-DBLP author 2; ... DBLP author 1; DBLP author 2; ... Parse Parse Simple matching (only match Pinyin with English name) Final result C-DBLP author x -> DBLP author u C-DBLP author y -> DBLP author v C-DBLP author z -> DBLP author w ... Author matching (disambiguation) C-DBLP author x[0] -> DBLP author u[0] C-DBLP author x[1] -> DBLP author u[1] ... Tuesday, March 5, 13
44. Implementation Miscellanies • Python 3 • Multi-thread Tuesday, March 5, 13
45. Future Work Tuesday, March 5, 13
46. Future Work • Optimize the most basic operations: for example, reduce HTTP request and I/O to make it faster. Tuesday, March 5, 13
47. Future Work • Optimize the most basic operations: for example, reduce HTTP request and I/O to make it faster. • Enhance exception handling: sometimes there might be problems with the encoding or output of DBLP/C-DBLP’s web page, and error will occur if these exceptions cannot be properly handled. Tuesday, March 5, 13
48. Future Work • Optimize the most basic operations: for example, reduce HTTP request and I/O to make it faster. • Enhance exception handling: sometimes there might be problems with the encoding or output of DBLP/C-DBLP’s web page, and error will occur if these exceptions cannot be properly handled. • Caching: cache some frequently requested entries or relationships in advance to optimize the performance. Tuesday, March 5, 13
49. Future Work • Optimize the most basic operations: for example, reduce HTTP request and I/O to make it faster. • Enhance exception handling: sometimes there might be problems with the encoding or output of DBLP/C-DBLP’s web page, and error will occur if these exceptions cannot be properly handled. • Caching: cache some frequently requested entries or relationships in advance to optimize the performance. • More queries: provide more query patterns (e.g., more joint queries) for users to satisfy different needs. Tuesday, March 5, 13
50. Demo Tuesday, March 5, 13
51. Thank you all! Tuesday, March 5, 13