Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Amy: The Email-based Virtual Assistant...

Building Amy: The Email-based Virtual Assistant by x.ai

Hakka Labs

March 17, 2015
Tweet

More Decks by Hakka Labs

Other Decks in Technology

Transcript

  1. Outline of talk • Product Intro • Conceptual overview •

    Current level of human involvement • Time expressions in email text • Performance • Email classification • Future Work
  2. Product Intro ai trainer ai • What is Amy or

    twin brother Andrew Amy is an automated assistant who schedules meetings for you • What is meeting scheduling • What involves scheduling a meeting • Meeting negotiations happen over email • What is not Amy • Using Amy
  3. Amy is a conversation model Host CC Amy Amy Guests

    New Meeting request Propose time to participant Request location from host Reject time / propose time Accept Time Location sent Meeting state needs to be determined - so that it can be resolved
  4. Intermediate States Meeting Invite Accept Time Accept Location New Meeting

    preferences Initial State Changing meeting states trigger actions Meeting invite Conceptual Overview
  5. Information Extraction Natural Language Processing Architectural design Current human involvement

    Natural Language Processing Preference analysis → The role of the AI Trainer Email Classification
  6. Actual example of the simplest kind Sure thing. I’ve Cc’d

    Amy who can help us find a time with Matt on Monday Hi Matt, Happy to get something on Dennis’ calendar. Does Monday, Oct 13 at 11:00 AM EDT work ? Alternatively, Dennis is available on Monday, Oct 13 at 1:00 PM or 2:00 PM. Dennis’ offce is at 48 Wall Street, New York, NY 10005 (5th Floor). Amy 1-2 works Classification (request meeting). Information extraction Calendar preferences. Availability Classification (accept time). Information extraction
  7. Temporal expression challenges : 1. Detection 2. Type 3. Coreference

    4. Resolution Time expressions in Email text lets do Tuesday at around 4 Hour of Day, Day of Week, etc ... Merge Day of Week and Time February 25th, 2015 at 13:00 EST
  8. Our Dataset Large dataset of fully annotated emails ⇒ We

    have undertaken a massive human annotation campaign where we fully annotate all emails going through our system to enable machine learning / training - Times, People, Location, Intents, etc … ← various frequencies for different cases (showing arbitrary slices)
  9. Temporal expression challenges : 1. Detection 2. Type 3. Coreference

    4. Resolution Data-driven solutions Tokens Regex-based model combined with POS taggers (Conditional Random Fields) SUTime library We built our own set of cases based on top of Timex3 Defined a set of closed operators on time ‘cases’ to check whether they should be “merged” or not Next slide … :-) Use context ! Example of type merge operation TimeConstraint (13:00) + DayOfWeek(2) = WeekDayTime(2,13:00)
  10. Our current approach ... → break complex logic into a

    set of simple binary questions Yes Yes Yes No Yes No No Yes No No Detected time entities Resolved times with email intent
  11. This is where singular focus helps ... Accept or decline

    time ? Yes Yes Matches to any ? No No Previous proposed times ? • Context • Fuzzy matching • Machine Learning
  12. How does this compare to state-of-the-art ? Recall 85% Precision

    97% x.ai on x.ai dataset using context → Ref: Context-dependent Semantic Parsing for Time Expressions, Kenton Lee , Yoav Artzi , Jesse Dodge∗, and Luke Zettlemoyer
  13. Email Classification • “One vs all” support vector machine for

    each intent • Feature reduction through “mutual information” filter • Optimise kernel params through automatic param survey Features used: • N-grams (origin of Amy Ingram’s name… ) • POS tagging • Syntax rules • Time-entities, people, locations • Context !
  14. Future work Enhance Amy’s calendar analysis : • Multiple people

    preferences • Automatically detect patterns Meeting location model • Suggest locations based on preferences • Enhance travel time understanding Meeting social network • Relative negotiator importance of meeting participants • Relative importance of meetings
  15. marcos @ x.ai chief data scientist and co-founder 48 Wall

    Street, 5th Floor New York, NY 10005 E: [email protected] T: @xdotai