Search and Query: An {Over, Re}view.

Invited talk to the National Academy of Science Committee on Internet Navigation and the Domain Name System: Technical Alternatives and Policy Implications, July 2001. As a reductio ad absurdum, it shows how much one could put into DNS with query processing technology a la Telegraph. Also makes the point that "query" means much more than "search".

Transcript

1. Search and Query: An {Over, Re}view Joseph M. Hellerstein Computer Science Division UC Berkeley jmh@cs.berkeley.edu
2. Background My interests: Database & Information Systems n Tech: Indexing, Query Processing, Dataflow, Parallel/Distributed/Federated Systems n Apps: Interactive Data Analysis, eMarkets, eCatalogs, Structured Querying on Internet, Querying Sensors My (lack of) agenda for today n Examples to defamiliarize today’s modes of operation n Technical breakdown/review of system components n Core/common vs. Value-Added n Natural policy discussion n Core = “free” and “fair”? n Value-Added = “expensive” and “market-driven”? n Conflicting needs of core scaling and policy?
3. A Note of Caution Contemplating a redesign? Fine. But… Familiarity often trumps other desiderata n Memories are short, cross-fertilization is poor n Even in the technical community n Trap: common culture/techniques: n “What’s hot”: everybody knows web searching n “What’s packaged”: Database technology = Oracle n Rather than “What’s core technology” In Expanding DNS, be sure to review known technology n What’s core under the packaging of related technologies? n Reuse/generalize common ideas n Leave specialized ideas above the core (minimize core constraints) n Good way to evaluate “brand new” ideas, too n How do they challenge the core? Is the core “right”?
4. Beware of Today’s Common Culture Rise of the web => Rise of Full Text Search n Used to be a library app (yawn) n Suddenly the dominant paradigm (??) Beware: Internet ¹ Web! n DNS is not document-centric n Neither are Peer-to-Peer filesharing or “Deep Web” Facts and Figures n Keyword search may not be core functionality If DNS is to serve many needs… n Must separate core search services from varying apps! n Try to accommodate what’s hot tomorrow n What’s a core service? What’s an app?
5. Road Map Defamiliarization n Some new Internet “search” apps n Provided by Telegraph An {over, re}view n Some basics of querying n Some separation of the issues Discussion…
6. Peer-to-Peer Filesharing Napster, Gnutella, Aimster, etc. n And a pile of ill-defined startups n This is not the web, but it is part of the Internet n Lessons already! Distributed files and search by substring n Isn’t that heaven?! n Would like to handle spelling, language translation (pig latin), etc. n A layer of canonicalization (“nameprep”) Well, what about these translations… n I post my name mapping scheme on my website n Download the Billboard Top Ten n Download tunes I’ve paid for No problem…
7. TeleNap: Amazon Meets Napster OpenNap Servers Album Information
8. Telenap: Amazon Meets Napster
9. “Search” vs. Query “Search” can return only what’s been stored E.g., best match at iWon, Google, AskJeeves top ten:
10. “Search” vs. Query “Search” can return only what’s been stored E.g., best match at iWon, Google, AskJeeves top ten:
11. But the Basic Facts are There!
12. Query >> Search: http://fff.cs.berkeley.edu “Federated Facts and Figures” n Yahoo join FECInfo Imagine DNS analogies: n Find IP addresses of hosts registered by companies that donated to GW Bush n FECInfo join WHOIS Note: errors! n Can do fuzzy matching and ranking, too
13. Query >> Search: http://fff.cs.berkeley.edu “Federated Facts and Figures” n APBNews join FECInfo Imagine DNS analogies: n Show # of live nodes, Roll Up By Domain n Show Avg Ping time, Roll Up By Domain n Show Best Ping Time per Group, Roll Up By Domain
14. Slippery Slope of “Mapping” Step 1: Exact Match Lookups Step 2: Simple canonicalization n E.g. toupper() Step 3: Complex canonicalization n E.g. “nameprep” Step 4: Join w/Arbitrary Mapping Tables n Centralized “keywords” (RealNames) n Personalized mapping, “bookmarks” n Geo-located mapping tables for Localization n Load-balancing mapping via net status queries n Ad-hoc mapping: ad hoc database join queries! n E.g. “The lightest-loaded machine containing >5 of the Billboard Top Ten Songs” Q: How much of this belongs in DNS??
15. Slippery Slope of “Mapping” Input Value Output Value Cs.Berkeley.EDU 128.32.35.1 cHeese CHEESE Nueva_York_Yanquis NYYankees Cheese www.kraft.com Dear old Dad hellerst@math.wisc.edu Thai Food www.plearn.com White Album Back in the USSR White Album Dear Prudence www.yahoo.com server47.yahoo.com
16. Slippery Slope of Querying Step 1: Single collection, exact lookup Step 2: Single collection, fancy lookup n “Search” Step 2: Combine multiple collections n E.g. Mapping (join), “Metasearch” (union) Step 3: Roll-up/drill-down Step 4: Statistical analyses Step 5: “Turing-complete” queries n Prolog/Datalog How much of this belongs in DNS?? n Note: Step 5 was even rejected in databases!
17. Where Does it End?? All this functionality seems plausibly useful n Some of it even necessary! But what should be in network infrastructure? n Vs. centralized, replicated services n Who pays for these services?
18. Querying 101 Access Methods e.g. B-trees, Hash Indexes Used in Directories, Databases, Search Engines. Can be distributed. e.g. Joins, Intersect, Union, Rank/Sort, Group/Aggregate (Stat. Summaries) Used in Databases, Search Engines Parallelized on Well-Maintained Clusters Bulk Data Processing Query Optimization e.g. Rewriting in canonical form, Choosing AMs and Processing Options. Very different techniques in Databases, Search Engines, though not mutually exclusive. This is Core and Common to All Query systems.
19. Access Methods Local search: n Main-memory data structures n binary trees, hashtables, skip lists, etc. n Disk-based data structures n B-trees, linear hash indexes, etc. n Typically equality and/or range lookups Distributed search n Flat partitioning (hash, range), w/replication n Hierarchical partitioning n More recent multi-hop search & replication n E.g. CAN, Chord, PAST, Tapestry, Pastry n Equality lookups only (so far), no need for hierarchies n Proposed as a DNS replacement by networking researchers
20. Data Processing Flow the data through processing code Absent in Directory Services Used in database systems (in the box) n Ad hoc combinations of operators possible Constrained use in text search engines (in the box) n 1 collection: (word, docID, position, score, …) n Indexed by stemmed word n OR, AND, NOT: Union/Intersect/Subtract n Map docIDs to URLs, snippets, etc. n Sort by a (magic) function of position, score, etc. n Text search is one (highly tuned!) database query Fun research: genericize this dataflow technology n Goal of Telegraph project: adaptive dataflow (out of the box) n Cluster-based implementation n Distributed (P2P) implementation
21. Query Optimization Text Search n Query rewrite: stemming, stop words, thesaurus n Scheduling: which machine(s) on cluster n Based on data partitioning, load & data statistics Database Systems n Query rewrite: authorization, “views” n Choices among (redundant) access methods n Choices among data processing algorithms (joins) n Choices of reorderings for these algorithms n Scheduling: which machines(s) on cluster n Based on data partitioning, load & data statistics Lots of fancy tricks here!
22. And what about storage semantics?? So far we only looked at the query side n Query results only as good as the data! n Again, varying solutions here…
23. Replication & Data Consistency Databases do Transactions n Atomic, durable updates across multiple records n Data consistency guaranteed n Distributed transactions possible, but slow n Two-Phase Commit n Most people do “warm” replication n Log-shipping w/xactional networking -- MQ n Heavyweight technology! n Brewer’s CAP “theorem” Directories tend to use “leases” (TTL) n Tend to be per record or collection n Cross-object consistency not guaranteed n A little drift is often OK n In a scenario with mapping, may need to think about atomicity across records/tables
24. One View: Core vs. Apps Ensure that core services: n Scale to large # of machines (~size of Internet) n Work in presence of failures n Well-defined standard results: no surprises/ambiguity! n E.g. SQL subsets, Boolean text search n Need not be a user in the loop! n Unanticipated scenarios in future n Support ad hoc queries -- think cross-paradigm Apps: n Scale to clusters, need not scale to size of Internet n Allow for mapping/customization n Results can be preference-dependent n What about time-/geo-dependent?? n Allow result browsing, analysis n Fuzzy results, roll-ups, summaries all OK: user in the loop! n Impose a query paradigm
25. More? jmh@cs.berkeley.edu http://www.cs.berkeley.edu/~jmh Telegraph: http://telegraph.cs.berkeley.edu Federated Facts and Figures: http://fff.cs.berkeley.edu