Evaluation of Unsupervised Entity and Event Salience Estimation

https://journals.flvc.org/FLAIRS/article/view/128482/130078

Transcript

1. May 18, 2021 Evaluation of Unsupervised Entity and Event Salience Estimation Jiaying Lu, Jinho D. Choi Emory University
2. 2 Entity and Event Salience Detection search engine Question Answering news recommendation E&E detector Kobe Bryant, helicopter crash DarkSide, attack, Colonial Pipeline Mars 2020, touch down, launch
3. 3 Task Definition Entity and Event Salience Task • Identify entities and events that contains most salient information of documents Prerequisites • How to detect entity and event mentions in text • How to measure terms salience
4. 4 Entity Definition References: ACE (Doddington et al. 2004), ECB+ (Cybulska and Vossen 2014), Rich ERE (Song et al. 2015), RED (O’Gorman et al. 2016)
5. 5 Event Definition
6. 6 Entity and Event (Pseudo) Annotation Dependency parsing tree derived by He and Choi (2020) Entities are base noun phrases. Events are verbal predicates or nominal predicates.
7. 7 Entity and Event (Pseudo) Annotation Examples adapted from Emory Deep Dependency Guidelines (https://emorynlp.github.io/ddr/doc/pages/overview.html)
8. 8 Pseudo Salient Ground-Truth Annotation Assumption (Dunietz and Gillick 2014; Xiong et al. 2018; Liu et al. 2018) - a term is considered salient if a summary written by humans includes it. The pseudo annotation is available at https://github.com/lujiaying/pseudo-ee-salience-estimation
9. 9 Proposed Approach RGCN Graph Construction Node Classification Reference: RGCN, Schlichtkrull et al. 2018.
10. 10 Graph Construction Module
11. 11 Experiment Setting References: New York Times (Dunietz and Gillick 2014), Semantic Scholar (Xiong et al. 2017) Evaluation Metrics • (Macro-averaged) Precision, Recall, F1 @Top-1/3/5/10 • neglecting punctuations and articles; using exact match
12. 12 Experiment Results Entity Salience Estimation Scores Event Salience Estimation Scores References: TextRank (Mihalcea and Tarau 2004); KCE (Liu et al. 2018)
13. 13 Conclusion • A light yet practical pseudo annotation is proposed. • Following the proposed annotation guideline, two datasets are released. • The empirical result shows the salience assumption is somehow biased to frequent occurred terms, which can be further improved in the future.
14. 14 Q & A?