a personal assistant who schedules meetings for you MACHINE LEARNING NYC FEBRUARY 2015 NEW YORK CITY x.ai 

Outline of talk ● Product Intro ● Conceptual overview ● Current level of human involvement ● Time expressions in email text ● Performance ● Email classification ● Future Work 

Product Intro ai trainer ai ● What is Amy or twin brother Andrew Amy is an automated assistant who schedules meetings for you ● What is meeting scheduling ● What involves scheduling a meeting ● Meeting negotiations happen over email ● What is not Amy ● Using Amy 

Amy is a conversation model Host CC Amy Amy Guests New Meeting request Propose time to participant Request location from host Reject time / propose time Accept Time Location sent Meeting state needs to be determined - so that it can be resolved 

Intermediate States Meeting Invite Accept Time Accept Location New Meeting preferences Initial State Changing meeting states trigger actions Meeting invite Conceptual Overview 

Information Extraction Natural Language Processing Architectural design Current human involvement Natural Language Processing Preference analysis → The role of the AI Trainer Email Classification 

Actual example of the simplest kind Sure thing. I’ve Cc’d Amy who can help us find a time with Matt on Monday Hi Matt, Happy to get something on Dennis’ calendar. Does Monday, Oct 13 at 11:00 AM EDT work ? Alternatively, Dennis is available on Monday, Oct 13 at 1:00 PM or 2:00 PM. Dennis’ offce is at 48 Wall Street, New York, NY 10005 (5th Floor). Amy 1-2 works Classification (request meeting). Information extraction Calendar preferences. Availability Classification (accept time). Information extraction 

Temporal expression challenges : 1. Detection 2. Type 3. Coreference 4. Resolution Time expressions in Email text lets do Tuesday at around 4 Hour of Day, Day of Week, etc ... Merge Day of Week and Time February 25th, 2015 at 13:00 EST 

Our Dataset Large dataset of fully annotated emails ⇒ We have undertaken a massive human annotation campaign where we fully annotate all emails going through our system to enable machine learning / training - Times, People, Location, Intents, etc … ← various frequencies for different cases (showing arbitrary slices) 

Our Dataset 

Temporal expression challenges : 1. Detection 2. Type 3. Coreference 4. Resolution Data-driven solutions Tokens Regex-based model combined with POS taggers (Conditional Random Fields) SUTime library We built our own set of cases based on top of Timex3 Defined a set of closed operators on time ‘cases’ to check whether they should be “merged” or not Next slide … :-) Use context ! Example of type merge operation TimeConstraint (13:00) + DayOfWeek(2) = WeekDayTime(2,13:00) 

Our current approach ... → break complex logic into a set of simple binary questions Yes Yes Yes No Yes No No Yes No No Detected time entities Resolved times with email intent 

This is where singular focus helps ... Accept or decline time ? Yes Yes Matches to any ? No No Previous proposed times ? ● Context ● Fuzzy matching ● Machine Learning 

Performance Recall 85% Precision 97% 

How does this compare to state-of-the-art ? Recall 85% Precision 97% x.ai on x.ai dataset using context → Ref: Context-dependent Semantic Parsing for Time Expressions, Kenton Lee , Yoav Artzi , Jesse Dodge∗, and Luke Zettlemoyer 

Email Classification ● “One vs all” support vector machine for each intent ● Feature reduction through “mutual information” filter ● Optimise kernel params through automatic param survey Features used: ● N-grams (origin of Amy Ingram’s name… ) ● POS tagging ● Syntax rules ● Time-entities, people, locations ● Context ! 

Email Classification We are actively working on improving the performance of our classifiers … more data ! 

Future work Enhance Amy’s calendar analysis : ● Multiple people preferences ● Automatically detect patterns Meeting location model ● Suggest locations based on preferences ● Enhance travel time understanding Meeting social network ● Relative negotiator importance of meeting participants ● Relative importance of meetings 

marcos @ x.ai chief data scientist and co-founder 48 Wall Street, 5th Floor New York, NY 10005 E: [email protected] T: @xdotai