Modeling Fine-Grained Entity Types with Box Embeddings

Neural entity typing models typically represent entity types as vectors in a high-dimensional space, but such spaces are not well-suited to modeling these types' complex interdependencies. We study the ability of box embeddings, which represent entity types as d-dimensional hyperrectangles, to represent hierarchies of fine-grained entity type labels even when these relationships are not defined explicitly in the ontology. Our model represents both types and entity mentions as boxes. Each mention and its context are fed into a BERT-based model to embed that mention in our box space; essentially, this model leverages typological clues present in the surface text to hypothesize a type representation for the mention. Soft box containment can then be used to derive probabilities, both the posterior probability of a mention exhibiting a given type and the conditional probability relations between types themselves. We compare our approach with a strong vector-based typing model, and observe state-of-the-art performance on several entity typing benchmarks. In addition to competitive typing performance, our box-based model shows better performance in prediction consistency (predicting a supertype and a subtype together) and confidence (i.e., calibration), implying that the box-based model captures the latent type hierarchies better than the vector-based model does.

Transcript

1. /QFGNKPI)KPG*TCKPGF(PVKV[6[RGUYKVJ %QZ(ODGFFKPIU ;CUWOCUC1PQG/KEJCGN%QTCVMQ$PFTGY/E&CNNWO*TGI'WTTGVV 76$WUVKP7/CUU$OJGTUV*QQING4GUGCTEJ
2. 7KH+XQJHU*DPHVWKHILUVWRIEHVWVHOOLQJERRNVE\6X]DQQH&ROOLQV )KPGITCKPGF(PVKV[6[RKPI -"/0,+ /1&01 21%,/  (PVKV[6[RGU (PVKV[/GPVKQP
3. 7KH+XQJHU*DPHVWKHILUVWRIEHVWVHOOLQJERRNVE\6X]DQQH&ROOLQV 6X]DQQH&ROOLQV )KPGITCKPGF(PVKV[6[RKPI  (PEQFGT 8GEVQT5RCEG
4. 7KH+XQJHU*DPHVWKHILUVWRIEHVWVHOOLQJERRNVE\6X]DQQH&ROOLQV 6X]DQQH&ROOLQV )KPGITCKPGF(PVKV[6[RKPI  -"/0,+ /1&01 21%,/ -"/0,+ -,)&1& &+ /1&01 (PEQFGT 䐂 6[RG(ODGFFKPIU 21%,/ 8GEVQT5RCEG
5. 7KH+XQJHU*DPHVWKHILUVWRIEHVWVHOOLQJERRNVE\6X]DQQH&ROOLQV )KPGITCKPGF(PVKV[6[RKPI -"/0,+ /1&01 21%,/  䐂 6[RG(ODGFFKPIU (PEQFGT 6X]DQQH&ROOLQV -"/0,+ -,)&1& &+ /1&01 21%,/ 8GEVQT5RCEG
6. +KGTCTEJKECN5VTWEVWTGKP(PVKV[6[RGU  /1&01 -"/0,+ 21%,/ -,)&1& &+ *20& &+ EQCTUG ȤPG Ɣ 6[RGUGVECPDGNCTIG(I!M 7)(6&JQK 
7. +KGTCTEJKECN5VTWEVWTGKP(PVKV[6[RGU  /1&01 -"/0,+ 21%,/ -,)&1& &+ *20& &+ EQCTUG ȤPG Ɣ 6[RGUGVECPDGNCTIG(I!M 7)(6&JQK  Ɣ 0QVCNYC[NKMGCVTGGȀ9JCVKH -"/0,+KU21%,/CPF-,)&1& &+"
8. +CTFVQECRVWTGJKGTCTEJKGUKPXGEVQTURCEG  6X]DQQH&ROOLQV -"/0,+ -,)&1& &+ /1&01 21%,/ 8GEVQT5RCEG Ɣ +QYFQYGMPQYKH/1&01KU VJGUWDV[RGQH-"/0,+"
9. +CTFVQECRVWTGJKGTCTEJKGUKPXGEVQTURCEG  Ɣ +QYFQYGMPQYKH/1&01KU VJGUWDV[RGQH-"/0,+" Ɣ +QYFQYGMPQYKH-"/0,+ CPF), 1&,+CTGOWVWCNN[ GZENWUKXG" ), 1&,+ 6X]DQQH&ROOLQV -"/0,+ -,)&1& &+ /1&01 21%,/ 8GEVQT5RCEG
10. %QZ(ODGFFKPIU  -"/0,+ -,)&1& &+ /1&01 21%,/ 8GEVQT5RCEG %QZ5RCEG "/0,+ 21%,/ ,)&1& &+ 27++" ,))&+0 6X]DQQH&ROOLQV
11. 7KH+XQJHU*DPHVWKHILUVWRIEHVWVHOOLQJERRNVE\6X]DQQH&ROOLQV (ODGFFGFKP%QZ5RCEG 6JKU9QTM   (PEQFGT %QZ5RCEG 27++" ,))&+0
12. 7KH+XQJHU*DPHVWKHILUVWRIEHVWVHOOLQJERRNVE\6X]DQQH&ROOLQV (ODGFFGFKP%QZ5RCEG 6JKU9QTM  -"/0,+ /1&01 21%,/  䐅 6[RG(ODGFFKPIU (PEQFGT %QZ5RCEG "/0,+ 21%,/ ,)&1& &+ 27++" ,))&+0
13. (ODGFFGFKP%QZ5RCEG 6JKU9QTM   %QZ5RCEG "/0,+ 21%,/ ,)&1& &+ 27++" ,))&+0 5WKVCDNGIGQOGVT[ Ɣ %QZ$EQPVCKPUCPQVJGTDQZ% ĺ$KUUWRGTV[RGQH% Ɣ %QZGUQXGTNCRĺECPDGDQVJ Ɣ %QZGUFKULQKPVĺOWVWCNN[ GZENWUKXG
14. $RRNKECVKQPUQH%QZ(ODGFFKPIU  Ɣ 9QTF0GVJ[RGTP[ORTGFKEVKQP 8KNPKU.K'CUIWRVC2CVGN   ż /QFGNVJGVTCPUKVKXGENQUWTGDGVYGGPYQTFUWUKPIDQZGU Ɣ 7PEGTVCKPMPQYNGFIGITCRJU &JGP ż (PVKVKGUCUDQZGUCPFTGNCVKQPUCUDQZVQDQZVTCPUHQTOCVKQPU Ɣ 4GEQOOGPFCVKQPU[UVGOU <JCPI ż 7UGTUCUDQZGUCPFKVGOUCURQKPVU
15. $RRNKECVKQPUQH%QZ(ODGFFKPIU  Ɣ 9QTF0GVJ[RGTP[ORTGFKEVKQP 8KNPKU.K'CUIWRVC2CVGN  ż /QFGNVJGVTCPUKVKXGENQUWTGDGVYGGPYQTFUWUKPIDQZGU Ɣ 7PEGTVCKPMPQYNGFIGITCRJU &JGP ż (PVKVKGUCUDQZGUCPFTGNCVKQPUCUDQZVQDQZVTCPUHQTOCVKQPU Ɣ 4GEQOOGPFCVKQPU[UVGOU <JCPI ż 7UGTUCUDQZGUCPFKVGOUCURQKPVU Ɣ =6JKUYQTM?)KPGITCKPGFGPVKV[V[RKPI ż 0CVWTCNNCPIWCIGKPRWV ż (XGT[VJKPIKUCDQZ GPVKV[OGPVKQPUCPFV[RGU
16. /QFGN1XGTXKGY  Ɣ (ODGFCPGPVKV[OGPVKQPCPF GPVKV[V[RGUKPVQVJGUCOGDQZ URCEG Ɣ 7UGDQZQRGTCVKQPU GIXQNWOG KPVGTUGEVKQPGVE VQEQORWVG 3 W\SH_PHQWLRQDQGFRQWH[W Ɣ 6TGCVGCEJV[RGRTGFKEVKQPCUCP KPFGRGPFGPVDKPCT[ENCUUKȤECVKQP %QZ5RCEG "/0,+ 21%,/ ,)&1& &+ 27++" ,))&+0
17.  + ,!"/ 7KH+XQJHU*DPHVWKH ILUVWRIEHVWVHOOLQJERRNV E\6X]DQQH&ROOLQV "+1&,++!,+1"51 /+0#,/* >&/6@ Ɣ (PEQFGOGPVKQPCPFEQPVGZVYKVJ%(46  %QZDCUGF(PVKV[6[RKPI/QFGN "+1&,++!,+1"51 " 1,/
18. Ɣ (PEQFGOGPVKQPCPFEQPVGZVYKVJ%(46 Ɣ 6TCPUHQTOVJGGPEQFKPIKPVQCDQZ "+1"/ ##0"1  %QZDCUGF(PVKV[6[RKPI/QFGN 5,&+1p &+,&+1p  + ,!"/ 7KH+XQJHU*DPHVWKH ILUVWRIEHVWVHOOLQJERRNV E\6X]DQQH&ROOLQV "+1&,++!,+1"51 /+0#,/* >&/6@ "+1&,++!,+1"51 " 1,/
19. Ɣ (ODGFGPVKV[V[RGUWUKPIDQZGU "+1"/ ##0"1  %QZDCUGF(PVKV[6[RKPI/QFGN 6-"*"!!&+$0 5,&+1p &+,&+1p Nv1%6-"" 1,/ -"/0,+ /1&01 21%,/ -,)&1& &+ 䐂 䐂 䐂 䐂
20.  %QZDCUGF/WNVKNCDGN6[RG&NCUUKȤGT Ɣ 2 V[RG^OGPVKQP  8QN V[RGDQZŀOGPVKQPDQZ 8QN OGPVKQPDQZ
21. Ɣ 2 V[RG^OGPVKQP  8QN V[RGDQZŀOGPVKQPDQZ 8QN OGPVKQPDQZ Ɣ &QORWVGUQHVXQNWOGQHKPVGTUGEVKQP  %QZDCUGF/WNVKNCDGN6[RG&NCUUKȤGT 21%,/ -,)&1& &+ 6X]DQQH &ROOLQV 6X]DQQH &ROOLQV ͳPVGTUGEVKQP%QZ
22.  %QZDCUGF/WNVKNCDGN6[RG&NCUUKȤGT 21%,/ -,)&1& &+ 6X]DQQH &ROOLQV 6X]DQQH &ROOLQV ͳPVGTUGEVKQP%QZ 2*"),5"0ƒ0$2-1n:8:8„ Ɣ 2 V[RG^OGPVKQP  8QN V[RGDQZŀOGPVKQPDQZ 8QN OGPVKQPDQZ Ɣ &QORWVGUQHVXQNWOGQHKPVGTUGEVKQP
23.  %QZDCUGF/WNVKNCDGN6[RG&NCUUKȤGT 21%,/ -,)&1& &+ 6X]DQQH &ROOLQV 6X]DQQH &ROOLQV &QORWVGͳPVGTUGEVKQP8QNWOG /GPVKQPCPF&QPVGZV%QZ8QNWOG ͳPVGTUGEVKQP%QZ 2*"),5"0ƒ0$2-1n:8:8„ Ɣ 2 V[RG^OGPVKQP  8QN V[RGDQZŀOGPVKQPDQZ 8QN OGPVKQPDQZ Ɣ &QORWVGUQHVXQNWOGQHKPVGTUGEVKQP
24. Ɣ &QORWVGEQPFKVKQPCNRTQDCDKNKV[D[FKXKFKPIXQNWOGU  %QZDCUGF/WNVKNCDGN6[RG&NCUUKȤGT 6X]DQQH&ROOLQV ŀ21%,/ 6X]DQQH&ROOLQV  6X]DQQH&ROOLQV ŀ-,)&1& &+ 6X]DQQH&ROOLQV  2TQDCDKNKV[QHNvVJV[RGƒWN„IKXGPOGPVKQPƒP„CPFEQPVGZVƒV„p
25. Ɣ /KPKOK\GVJGUWOQHDKPCT[ETQUUGPVTQR[NQUUGUQXGTCNNV[RGUQXGTCNN GZCORNGU  .QUU)WPEVKQP 2TQDCDKNKV[QHNvVJV[RGƒWN„IKXGPOGPVKQPƒP„CPFEQPVGZVƒV„p 1DLGEVKXG
26. (ZRGTKOGPVU Ɣ (PVKV[6[RKPI ż 'QGUCDQZOQFGNKORTQXGV[RKPIRGTHQTOCPEG" Ɣ &QPUKUVGPE[ ż 9JGPCOQFGNRTGFKEVUCȤPGV[RGFQGUKVCNUQRTGFKEVVJGEQCTUGV[RG" Ɣ 4QDWUVPGUU ż +QYOWEJFQGURGTHQTOCPEGFGITCFGKHNCDGNUCTGPQKU[" Ɣ %QZ(FIGU Ɣ &CNKDTCVKQP Ɣ 'QYPUVTGCO6CUM 
27. (PVKV[6[RKPI 7)(6 &JQK QH6[RGUM %CUGNKPGU Ɣ 8GEVQTOQFGN %(46 Ɣ 8GEVQTOQFGN (./Q   1PQG'WTTGVV Ɣ +[RGTDQNKEOQFGN   .°RG\5VTWDG 'GX4GUWNVU  7UGFFKͯGTGPVCWIOGPVGFFCVC %QZ 8GEVQT %(46 8GEVQT (./Q +[RGTDQNKE /CETQ)
28. (PVKV[6[RKPI 1PVQ0QVGU *KNNKEM QH6[RGUV[RGU %CUGNKPGU Ɣ 8GEVQTOQFGN %(46 Ɣ 'QEWOGPVNGXGN <JCPI Ɣ +KGTCTEJKECNTCPMKPI &JGP  6GUV4GUWNVU  0QVG9GTGRQTV%%0CPF)ͳ*(4TGUWNVUKPQWTRCRGT %QZ 8GEVQT %(46 +KGT TCPMKPI 'QENGXGN /CETQ)
29. (PVKV[6[RKPI 6GUV4GUWNVU  0QVG9GTGRQTV%%0CPF)ͳ*(4TGUWNVUKPQWTRCRGT %QZ 8GEVQT %(46 +KGT TCPMKPI 'QENGXGN /GPVKQP $VVP /CETQ) 1PVQ0QVGU *KNNKEM QH6[RGUV[RGU %CUGNKPGU Ɣ 8GEVQTOQFGN %(46 Ɣ 'QEWOGPVNGXGN <JCPI Ɣ +KGTCTEJKECNTCPMKPI &JGP  Ɣ /GPVKQPURCPCVVGPVKQP .KP,K ż 7UGFNCTIGUECNGCWIOGPVGFFCVC
30. (PVKV[6[RKPI 6GUV4GUWNVU  0QVG9GTGRQTV%%0CPF)ͳ*(4TGUWNVUKPQWTRCRGT %QZ 8GEVQT %(46 +KGT TCPMKPI 'QENGXGN /GPVKQP $VVP /CETQ) 1PVQ0QVGU *KNNKEM QH6[RGUV[RGU %CUGNKPGU Ɣ 8GEVQTOQFGN %(46 Ɣ 'QEWOGPVNGXGN <JCPI Ɣ +KGTCTEJKECNTCPMKPI &JGP  Ɣ /GPVKQPURCPCVVGPVKQP .KP,K ż 7UGFNCTIGUECNGCWIOGPVGFFCVC Ɣ %QZKUDGVVGTVJCPXGEVQTKPIGPGTCN
31. &QPUKUVGPE[  9JGPCOQFGNRTGFKEVUCȤPGV[RG CTVKUV FQGUKVCNUQRTGFKEVVJGEQCTUG V[RG RGTUQP " Ɣ EQCTUGV[RGUȤPGV[RGU ż RGTUQP ż NQECVKQP ż RNCEG ż QTICPK\CVKQP 5WRGTV[RG RGTUQP NQECVKQP RNCEG QTICPK\CVKQP $EEWTCE[
32. &QPUKUVGPE[  9JGPCOQFGNRTGFKEVUCȤPGV[RG CTVKUV FQGUKVCNUQRTGFKEVVJGEQCTUG V[RG RGTUQP " Ɣ EQCTUGV[RGUȤPGV[RGU ż RGTUQP ż NQECVKQP ż RNCEG ż QTICPK\CVKQP Ɣ %QZKUEQPUKUVGPVN[DGVVGT VJCPXGEVQTHQTCNNEQCTUG V[RGU 5WRGTV[RG RGTUQP NQECVKQP RNCEG QTICPK\CVKQP $EEWTCE[
33. 4QDWUVPGUU +QYOWEJFQGURGTHQTOCPEGFGITCFGKHNCDGNUCTGPQKU["   Ɣ 1TKIKPCN7)(6VTCKPKPIFCVC Ɣ 0QKUGF&QCTUGFTQRVJGIQNF EQCTUGV[RGUYKVJR  Ɣ 0QKUGF)KPG 7NVTCȤPGFTQRVJG IQNFȤPGCPFWNVTCȤPGV[RGUYKVJR  /CETQ) %QZ 8GEVQT
34. 4QDWUVPGUU +QYOWEJFQGURGTHQTOCPEGFGITCFGKHNCDGNUCTGPQKU["   Ɣ 1TKIKPCN7)(6VTCKPKPIFCVC Ɣ 0QKUGF&QCTUGFTQRVJGIQNF EQCTUGV[RGUYKVJR  Ɣ 0QKUGF)KPG 7NVTCȤPGFTQRVJG IQNFȤPGCPFWNVTCȤPGV[RGUYKVJR  Ɣ %QZKUOQTGTQDWUVCICKPUVOKUUKPI NCDGNUVJCPXGEVQT /CETQ) %QZ 8GEVQT
35. %QZ(FIGU  (FIGUQH-"/0,+CPF 1,/DQZGU
36.   1,/ G G G -"/0,+ 7KHLQWHUVHFWLRQRIWKH 1,/ER[$1'PHQWLRQ ER[HVRIDOOWKHGHYH[DPSOHVZKHUHWKHPRGHO SUHGLFWVWKH 1,/W\SH LHWKHHIIHFWLYH UHJLRQRIWKH 1,/ER[   1,/ER[ 0HQWLRQER[RIWKH GHYH[DPSOH G  7KHHIIHFWLYHUHJLRQRI WKH 1,/ER[ (ͯGEVKXG4GIKQP
37. %QZ(FIGU  (FIGUQH-"/0,+CPFVJGGͯGEVKXGDQZQHVJG 1,/V[RG (FIGUQH-"/0,+CPF 1,/DQZGU
38. 6CMGCYC[U Ɣ %QZGODGFFKPIUECRVWTGFGRGPFGPEKGUDGVYGGP GPVKV[V[RGU YKVJQWVGZRNKEKVUWRGTXKUKQP Ɣ (CU[VQWUGDQZGODGFFKPIUQPN[TGSWKTG OQFKȤECVKQPUQPVJGQWVRWVNC[GT JVVRUIKVJWDEQO[CUWOCUCQPQG%QZ6[RGU Ɣ 5VTQPITGUWNVUQPGPVKV[V[RKPIEQPUKUVGPE[CPF TQDWUVPGUU