Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Task-Based Support in Search Engines

Task-Based Support in Search Engines

Date: December 4, 2019
Venue: UiS, Stavanger, Norway. My PhD Viva presentation

Please cite, link to or credit this presentation when using it or part of it in your work.

#InformationRetrieval #IR #TaskBasedSearch #TaskCompletionEngines

Darío Garigliotti

December 04, 2019
Tweet

More Decks by Darío Garigliotti

Other Decks in Research

Transcript

  1. Outline • Motivation • Type-Aware Entity Retrieval • Target Entity

    Type Identification • Entity-Oriented Search Intents • Task-Based Query Suggestions • Task Recommendations • Conclusions and Future Directions 2
  2. Motivation • Today's web search experience aims to understand the

    user query • A way to organize information is via structured knowledge centered around entities • Large knowledge repositories and knowledge bases have become available 7
  3. • Underlying search goal is often a complex and knowledge-intensive

    task • For example, to plan a travel - How to get there? - Where to stay? - What to do? • Task completion would provide support for the user when accomplishing complex search tasks Motivation 15
  4. An example • Searching for wedding cakes cake shops Konditoriet

    i Sandnes Olja's Kake Boutique Baker Corner Lura 18
  5. An example • Searching for wedding cakes wedding cake shops

    Stavanger Conditori Olja's Kake Boutique Gjestalveien Conditori 19
  6. • Entity retrieval is the task of obtaining a ranked

    list of entities relevant to a search query • We investigate the utilization of entity type information for entity retrieval Entity Retrieval 20
  7. … Shop … … … Company Organization … Stavanger Conditori

    Bank Law Firm … Entity types • A characteristic property of entities is that they are typed • Types are organized in hierarchies (or taxonomies) 21
  8. … Shop … … … Company Organization … cake shops

    cake shops Bank Law Firm … • Target types: types of entities sought by the query Target entity types 22
  9. Type-aware Entity Retrieval • Type information is known to improve

    entity retrieval • Yet it is a multifaceted problem query entity wedding cake shops target types Stavanger Conditori term-based similarity type-based similarity … … entity types 23
  10. • We assume oracle-given type information Type-aware Entity Retrieval …

    Shop … … … Company Organization … cake shops cake shops Bank Law Firm … 25
  11. • We assume oracle-given type information • We identify dimensions

    in utilizing entity type information - Type taxonomy - Type representation - Retrieval model Type-aware Entity Retrieval 26
  12. • Which type taxonomy to use? - DBpedia Ontology (7

    levels, 600 types) - Freebase Types (2 levels, 2K types) - Wikipedia Categories (34 levels, 600K types) - YAGO Taxonomy (19 levels, 500K types) • These vary a lot in terms of hierarchical structure and in how entity-type assignments are recorded Type-aware Entity Retrieval 27
  13. • How to use type information into entity retrieval? •

    Retrieval task is defined in a generative probabilistic framework P(q | e) • Both query and entity are considered in the term space as well as in the type space Type-aware Entity Retrieval 29
  14. • We assume oracle-given type information • We conduct an

    evaluation of dimensions in utilizing entity type information - Type taxonomy - Type representation - Retrieval model • We use a strong text-based baseline • We test with the DBpedia Entity collection v2 Type-aware Entity Retrieval 33
  15. • Wikipedia, in combination with the most specific type representation,

    performs best • Hierarchical relationships from ancestor types improve retrieval effectiveness, but most specific types provide the best performance • Results regarding most effective type-aware retrieval model vary across configurations Type-aware Entity Retrieval 34
  16. Target Type Identification • "We assume oracle-given type information" -

    How to identify target entity types? - How do these target types automatically identified perform for type-aware entity retrieval? 39
  17. Target Type Identification • We revisit the task of hierarchical

    target type identification • Task: to find the main target types of a query, from a type taxonomy, such that these are the most specific category of entities that are relevant to the query. If no matching type can be found in the taxonomy then the query is assigned a special NIL-type 41
  18. Target Type Identification • We develop a Learning-to-Rank approach •

    We evaluate it using a purpose-built test collection 42
  19. Target Type Identification • We also conduct an evaluation utilizing,

    rather than a target types oracle, target entity types automatically identified 44
  20. We identify and evaluate dimensions in utilizing target entity type

    information for ad-hoc entity retrieval. We build a test collection for target entity type identification, and develop and evaluate a Learning-to-Rank approach for this problem. SUMMARY 46
  21. Entity-Oriented Search Intents • Intent: the underlying user need in

    a entity- oriented search query - For example, the intent of booking a hotel room • Refiner: a way to express an intent in an entity-oriented query - For example, for booking a hotel room: "booking", "book", "reservation", "rooms" 49
  22. <fashion designer> instagram Understanding entity- oriented search intents • We

    obtain a collection of type-level query patterns stella mccartney instagram vivienne westwood instagram 62
  23. Understanding entity- oriented search intents • We obtain a collection

    of type-level query patterns • Pick a Freebase type if it covers 100+ prominent entities • Get query suggestions for top 1000- entities per type • For each query, replace entity by type • Aggregate all frequencies for each (type, refiner) pair • Filter out all type-level refiners with frequency of 4- • Select 50 representative types by stratified sampling 63
  24. Understanding entity- oriented search intents • We define a scheme

    of intent categories Property Website Service 67
  25. Understanding entity- oriented search intents • We define a scheme

    of intent categories Property Website Service Other 68
  26. Understanding entity- oriented search intents • We define a scheme

    of intent categories - Website, Property, Service, Other => Website => Property => Service vivienne westwood age vivienne westwood instagram vivienne westwood customer care 69
  27. Understanding entity- oriented search intents • We annotate 2.3K+ unique

    type-level refiners with intent category via crowdsourcing • We observe the proportions of refiners in each category Property: 28.6% Service: 54.06% Website: 5.34% Other: 12.08% 70
  28. Understanding entity- oriented search intents 71 organization business operation chemical

    compound film location event food hotel disease restaurant travel destination 0 50 100 150 200 250 university house person newspaper airport basketball player album professional sports team game artwork 0 50 100 150 200 railway human language tv station political party amusement park exhibition venue chef programming language academic institution netflix genre 0 20 40 60 80 100 120 war currency blogger hobby football match sports championship star muscle olympic sport company 0 10 20 30 40 50 WroSicaO cycOone kingdoP PedicaO sSeciaOWy coPic book SubOisher oiO fieOd Wower beer counWry region eOecWion asWeroid beOief 0 10 20 30 40 50 3roSerWy WebsiWe Service 2Wher
  29. We propose a scheme of entity-oriented search intent categories. We

    annotate a collection of query refiners using the scheme, and observe that there is a large proportion of service-oriented intents. SUMMARY 72
  30. Roadmap Type-Aware Entity Retrieval Entity-Oriented Search Intents Utilizing Entity Type

    Information Identifying Target Entity Type Information Understanding Entity- Oriented Search Intents 73
  31. 1. Intents searched for a type of entities paris map,

    sydney map => [city] map 2. Categories assigned to refiners vivienne westwood instagram => Website vivienne westwood age => Property vivienne westwood customer care => Service 3. Multiple refiners expressing an intent "booking", "book", "make a reservation", "rooms" 75 A knowledge base of entity- oriented search intents
  32. 1. Intents searched for a type of entities paris map,

    sydney map => [city] map • (intent ID, searchedForType, entity type, confidence) 2. Categories assigned to refiners vivienne westwood instagram => Website vivienne westwood age => Property vivienne westwood customer care => Service 3. Multiple refiners expressing an intent "booking", "book", "make a reservation", "rooms" 76 A knowledge base of entity- oriented search intents
  33. 1. Intents searched for a type of entities paris map,

    sydney map => [city] map • (intent ID, searchedForType, entity type, confidence) 2. Categories assigned to refiners vivienne westwood instagram => Website vivienne westwood age => Property vivienne westwood customer care => Service • (intent ID, ofCategory, intent category, confidence) 3. Multiple refiners expressing an intent "booking", "book", "make a reservation", "rooms" 77 A knowledge base of entity- oriented search intents
  34. 1. Intents searched for a type of entities paris map,

    sydney map => [city] map • (intent ID, searchedForType, entity type, confidence) 2. Categories assigned to refiners vivienne westwood instagram => Website vivienne westwood age => Property vivienne westwood customer care => Service • (intent ID, ofCategory, intent category, confidence) 3. Multiple refiners expressing an intent "booking", "book", "make a reservation", "rooms" • (intent ID, expressedBy, refiner, confidence) A knowledge base of entity- oriented search intents 78
  35. Approach Refiners acquisition Refiners categorization Intents discovery [hotel] airport [hotel]

    spa [hotel] booking ... [hotel] airport: Service [hotel] address: Property [hotel] expedia: Website ... taxi arrive Hotel_Arriving booking make a reservation Hotel_Booking address Hotel_Address KB construction Intent ID Predicate Object Confidence Hotel_Booking searchedForType [hotel] c1 Hotel_Booking ofCategory Service c2 Hotel_Booking expressedBy "booking" c3 Hotel_Booking expressedBy "make a reservation" c4 Hotel_Booking expressedBy "rooms" c5 79
  36. Approach Refiners acquisition Refiners categorization Intents discovery [hotel] airport [hotel]

    spa [hotel] booking ... [hotel] airport: Service [hotel] address: Property [hotel] expedia: Website ... taxi arrive Hotel_Arriving booking make a reservation Hotel_Booking address Hotel_Address Intent profile { KB construction Intent ID Predicate Object Confidence Hotel_Booking searchedForType [hotel] c1 Hotel_Booking ofCategory Service c2 Hotel_Booking expressedBy "booking" c3 Hotel_Booking expressedBy "make a reservation" c4 Hotel_Booking expressedBy "rooms" c5 80
  37. Knowledge base construction • Application of the pipeline to extract

    all quadruples from 581 unseen types • 155K quadruples, 31K intent profiles - Excerpt of the KB, for intent ID <aviation.airline-65-customer_service> 81
  38. Experimental evaluation • Experts judge correctness, ignoring confidence, of around

    1.29% of IntentsKB 82 [0, 0.87) [0.87, 0.88) [0.88, 0.9) [0.9, 0.93) [0.93, 1] Confidence intervals according to the splitting percentiles 0% 20% 40% 60% 80% 100% Proportion of triples 6,337 6,370 6,335 6,368 6,314 Correct Incorrect, OFCATEGORY Incorrect, EXPRESSEDBY
  39. We design and build a knowledge base of entity- oriented

    search intents. We evaluate each component in our approach, as well as the correctness of the obtained knowledge base. SUMMARY 83
  40. Roadmap Type-Aware Entity Retrieval Entity-Oriented Search Intents Utilizing Entity Type

    Information Identifying Target Entity Type Information Understanding Entity- Oriented Search Intents Modeling Entity- Oriented Search Intents 84
  41. An example Cheap wedding cake Make your own invitations Buy

    a used wedding gown low wedding budget 91
  42. Cheap wedding cake Make your own invitations Buy a used

    wedding gown low wedding budget An example 92
  43. Cheap wedding cake Make your own invitations Buy a used

    wedding gown Excerpt from TREC Tasks test dataset low wedding budget 1 low budget wedding dresses 0 low wedding budget cars 1 find a gown ... 0 wedding flowers 1 cup cake wedding 1 wedding cakes ... 2 wedding invitation 1 find wedding invitation templates 0 designer dresses wedding ... An example 93
  44. Cheap wedding cake Make your own invitations Buy a used

    wedding gown Excerpt from TREC Tasks test dataset } low wedding budget 1 low budget wedding dresses 0 low wedding budget cars 1 find a gown ... 0 wedding flowers 1 cup cake wedding 1 wedding cakes ... 2 wedding invitation 1 find wedding invitation templates 0 designer dresses wedding ... An example 94
  45. Cheap wedding cake Make your own invitations Buy a used

    wedding gown Excerpt from TREC Tasks test dataset } } low wedding budget 1 low budget wedding dresses 0 low wedding budget cars 1 find a gown ... 0 wedding flowers 1 cup cake wedding 1 wedding cakes ... 2 wedding invitation 1 find wedding invitation templates 0 designer dresses wedding ... An example 95
  46. An example Cheap wedding cake Make your own invitations Buy

    a used wedding gown Excerpt from TREC Tasks test dataset } } } low wedding budget 1 low budget wedding dresses 0 low wedding budget cars 1 find a gown ... 0 wedding flowers 1 cup cake wedding 1 wedding cakes ... 2 wedding invitation 1 find wedding invitation templates 0 designer dresses wedding ... 96
  47. • Given an initial query, Suggesting queries to support task-based

    search wedding cake wedding cake gallery wedding cake recipes wedding cake flavors 98
  48. • Given an initial query, to get a ranked list

    of query suggestions that cover all the possible subtasks related to the task that the user is trying to achieve. Suggesting queries to support task-based search wedding cake wedding cake gallery wedding cake recipes wedding cake flavors 99
  49. • Given an initial query, to get a ranked list

    of query suggestions that cover all the possible subtasks related to the task that the user is trying to achieve. Suggesting queries to support task-based search wedding cake wedding cake gallery wedding cake recipes wedding cake flavors • This is the task understanding problem 100
  50. Suggesting queries to support task-based search • We propose an

    end-to-end generative probabilistic model • We exploit different information sources 101
  51. Suggesting queries to support task-based search • We propose an

    end-to-end generative probabilistic model 104
  52. • Components: q0 Suggesting queries to support task-based search •

    We propose an end-to-end generative probabilistic model 105
  53. • Components: • Source importance q0 API SUGGS. WEB SNIPPETS

    WEB DOCS. WH Suggesting queries to support task-based search • We propose an end-to-end generative probabilistic model 106
  54. • Components: • Source importance • Document importance q0 API

    SUGGS. WEB SNIPPETS WEB DOCS. WH Suggesting queries to support task-based search • We propose an end-to-end generative probabilistic model 107
  55. • Components: • Source importance • Document importance • Keyphrase

    relevance q0 Keyphrases API SUGGS. WEB SNIPPETS WEB DOCS. WH Suggesting queries to support task-based search • We propose an end-to-end generative probabilistic model 108
  56. • Components: • Source importance • Document importance • Keyphrase

    relevance • Query suggestion • We propose an end-to-end generative probabilistic model Query suggestions q0 Keyphrases API SUGGS. WEB SNIPPETS WEB DOCS. WH Suggesting queries to support task-based search 109
  57. • We make use of the 2015 and 2016 TREC

    Tasks track datasets for the task understanding problem • We conduct a principled estimation of the components, and analyze the best performing estimators per component Suggesting queries to support task-based search 110
  58. Suggesting queries to support task-based search 111 • We observe

    a heavy reliance on query suggestions from suggestion APIs
  59. Generating suggestion candidates Query completion Query refinement wedding cake wedding

    cake gallery wedding cake recipes wedding cake flavors wedding cake beautiful wedding cakes unique wedding cake designs simple wedding cake • Two query suggestion modes 112
  60. • How to jointly generate query suggestions in query completion

    and refinement modes? - Can we do it without relying on log data / API? • We consider a two-step pipeline: - Candidate generation - Candidate ranking • And focus on the first component Generating suggestion candidates 113
  61. • We study alternative generation methods and information sources -

    Methods: popular suffix, neural language, sequence- to-sequence - Sources: AOL query log, KnowHow, WikiAnswers • We build a test collection of query suggestion candidates Generating suggestion candidates 114
  62. • End-to-end is still the best method overall, but limited

    as it depends on API suggestions • Log data is the most useful information source, but the other sources provide valuable suggestions too • Different method-source configurations contribute unique suggestions in both modes Generating suggestion candidates 115
  63. We propose and evaluate a generative probabilistic model for task-based

    query suggestions. We further study alternative methods and information sources for suggestion candidate generation, and build a test collection. SUMMARY 116
  64. Roadmap Type-Aware Entity Retrieval Task-Based Search Entity-Oriented Search Intents Utilizing

    Entity Type Information Identifying Target Entity Type Information Understanding Entity- Oriented Search Intents Modeling Entity- Oriented Search Intents Suggesting Queries 117
  65. Task Recommendation • The underlying search goal is often a

    complex and knowledge-intensive task • We propose to recommend specific tasks to users, based on their search queries 119
  66. An example • Planning a wedding reception wedding reception Plan

    a wedding reception Recommended Tasks: Plan your wedding reception exit Announce the bridal party at a reception Throw a Hawaiian wedding reception Choose wedding reception activities 120
  67. Task Recommendation • Some terminology: - Task repository: a catalog

    of task descriptions - Task description: a semi-structured document that explains the steps involved in how to complete a given task - Search mission: a set of queries that all share the same underlying task 122
  68. Task Recommendation • We introduce two problems: 1. Query-based task

    recommendation Given a query, to return a ranked list of tasks that correspond to the task behind the query 2. Mission-based task recommendation Given a search mission, to return a ranked list of recommended tasks, corresponding to the queries in the mission 123
  69. Task Recommendation • We use a collection of WikiHow articles

    as our task repository 124 How to Make a Wedding Cake Co-authored by wikiHow Staff ✔ You can make a wedding cake for a customer if you bake for a living, or you might make a cake for loved one’s wedding to help them save money. If you love to bake, then you might even want to make your own wedding cake! Steps 1 Decide on the number and shape of the cake’s layers. Consider how many layers and what shape you want the cake to have. 2 Preheat the oven to the temperature indicated by your recipe. Many recipes call for the oven to be pre-heated to 350 °F (177 °C). 3 Prepare the cake batter according to your recipe’s instructions. Choose a recipe to create the cake batter for your cake. 4 Pour the batter into a greased, parchment-lined cake pan. Spray your cake pan with non-stick cooking spray. Explanation Main Act Detailed Act Title
  70. Task Recommendation • We focus on a subset of tasks,

    the procedural tasks • Procedural task: a search task that can be accomplished by following a sequence of specific actions or subtasks 125
  71. Task Recommendation • From a corpus of search queries and

    missions, we obtain a set of procedural search missions 126
  72. Query-based task recommendation • We propose a Learning-to-Rank method for

    query-based task recommendation, that combines a text-based ranking technique with continuous semantic representations • We experiment with different word embeddings and word function sets according to POS-tag 128
  73. • To address mission-based task recommendation, we propose methods that

    aggregate the individual query-based recommendations for each query into mission- level recommended tasks 131 Mission-based task recommendation
  74. We introduce the problems of query-based and mission-based task recommendation.

    We develop a test collection for task recommendation, and propose and evaluate approaches for these problems. SUMMARY 133
  75. Roadmap Type-Aware Entity Retrieval Task-Based Search Entity-Oriented Search Intents Utilizing

    Entity Type Information Identifying Target Entity Type Information Understanding Entity- Oriented Search Intents Modeling Entity- Oriented Search Intents Suggesting Queries Recommending Tasks 134
  76. Conclusions 136 wedding cake Stavanger Conditori Olja's Kake Boutique Gjestalveien

    Conditori Cake shops > Wedding cake shops Recommended tasks Make a Chocolate Cake Basic Chocolate Cake Moist & Fluffy Chocolate Cake Bake an Easy Applesauce Cake See ingredients See steps Address: Gjesdalveien 27, 4306 Sandnes Hours Today: 9AM-5PM Address: Godesetdalen 10, 4034 Stavanger CALL CALL CALL Decorate a Cake Working with Fondant Adding Quick Decorations Queries suggested for wedding cake wedding cake recipes for beginners best wedding cake recipes wedding cake recipes video chocolate wedding cake recipes homemade wedding cake recipes from scratch
  77. Conclusions 137 wedding cake Stavanger Conditori Olja's Kake Boutique Gjestalveien

    Conditori Cake shops > Wedding cake shops Recommended tasks Make a Chocolate Cake Basic Chocolate Cake Moist & Fluffy Chocolate Cake Bake an Easy Applesauce Cake See ingredients See steps Address: Gjesdalveien 27, 4306 Sandnes Hours Today: 9AM-5PM Address: Godesetdalen 10, 4034 Stavanger CALL CALL CALL Decorate a Cake Working with Fondant Adding Quick Decorations Queries suggested for wedding cake wedding cake recipes for beginners best wedding cake recipes wedding cake recipes video chocolate wedding cake recipes homemade wedding cake recipes from scratch Type-Aware Entity Retrieval Entity-Oriented Search Intents Task-Based Query Suggestions Task Recommendations