「知識」のDeep Learning @PFIセミナー

Transcript

1. Deep Learning 1 Preferred Infrastructure (@unnonouno) 2015/06/04 PFI
2. !  1 !  !  NLP 12014-
3. NLP 1YANS !  YANS 19 !  140 !  !  :l !  YANS 13 !  !  "
4. Knowledge Representation
5. 1. 3. 4. 2.
6. 1. 3. 4. 2.
7. 1. 3. 4. 2. RNN
8. Recurrent Neural Network Language Model (RNNLM) [Mikolov+10] !  RNN !  !  !  LSTMT   
   
9. Long Short-Term Memory (LSTM) !  !  1 ( 05$ ! 
10. RNN1LSTM !  ! 
11. !  V;f !  bK1+3 argmaxy P(y|x) = argmaxy P(x|y) P(y)
12. 1. 3. 4. 2.
13. [ 15] !  :A@A[1 !  -‐‑‒ -5.4&,
14. -‐‑‒ [ 15] !  Pydata Tokyo pSQgD]
15. 1. 3. 4. 2.
16. -‐‑‒ !  !  ! 
17. Knowledge Representation
18. !  EMNLP2014 Bordes Weston 1 part 2 [Bordes&Weston14]
19. !  [ 15]
20. Q: Deep Learning Representation Learning -‐‑‒ A: Deep
21. ਪ࿦Λಋ͚ΔΑ͏ͳ஌ࣝͷදݱɺ͓Αͼͦͷํ๏ Λ։ൃ͢Δਓ޻஌ೳݚڀͷྖҬͰ͋Δɻ
    
22. !  !  !  1
23. 1 !  2 t t 3 !  t t s 2/3X u !  RDF (x, r, y) x y r
24. 1Knowledge Base
25. !  New York NY
26. !  EL1+3 /#,3qO !  t t !  t t !  t t
27. -‐‑‒ !  -‐‑‒ NLP "-4)%Y <
28. !  {(x i , r i , y i )}: !  x, y: t t !  r: t t !  x, y !  r
29. !  !  1 argmax ∑ i f(xi , ri , yi )
30. Distance model (Structured Embedding) [Bordes +11] !  e !  R left , R right !  !  f f(x, r, y) = || R left (r) e(x) – R right (r) e(y) || 1
31. TransE model [Brodes+13] !  r /#,3 r ] !  f(x, r, y) = || e(x) + r – e(y) || 2 2
32. TransE model x y r
33. TransE model 1 TransE BrN/#,3?8 ia !  TransM: !  TransH:
34. TransM model [Fan+14] !  r !  w r r x, y f(x, r, y) = w r || e(x) + r – e(y) || 2 2
35. TransH model [Wang+14] !  dJR6mc TransE
36. Bilinear model !  r !  f(x, r, y) = e(x)T W r e(y)
37. Neural Tensor Network (NTN) [Socher+13] f(x, r, y) = u r tanh(e(x)W r e(y) + V1 r e(x) + V2 r e(y) + b r ) !  r *5'3 ! 
38. [Yang+15] !  2 !  Bilinear
39. Q: -‐‑‒ A: 9n<P [Nickel+11]
40. Link prediction !  t t 1 !  QA !  -‐‑‒ !  t t (e1 , r, e2 ) (e1 , r, ? )
41. TransH TransE [Bordes&Weston14]
42. !  ! 
43. [Weston+13] !  x y VZ: r !  [Bordes&Weston14]
44. [Weston+13] !  !  -‐‑‒ [Bordes&Weston14]
45. [Bordes&Weston14]
46. Link prediction 1QA !  Link prediction ! 
47. QA [Bordes+14] !  _FV>H !  q t f(q) g(t) !  f(q), g(t) q, t
48. Memory networks: [Weston+15] !  I: I(x) !  G: m i = G(m i , I(x), m) !  O: o = O(I(x), m) !  R: r = R(o)
49. I: O: R: G:
50. !  !  !  !  _F\h1+3
51. !  Deep Learning -‐‑‒
52. -‐‑‒ !  !  -‐‑‒ !  -‐‑‒ !  -‐‑‒ !  !  factoid !  -‐‑‒ ! 
53. RNN [Peng&Yao15] !  RNN ! 
54. !  !  !  !  !  !  !  ! 
56. 1/4 !  [Mikolov+10] T. Mikolov, M. Karafiat, L. Burget, J. H. Cernocky, S. Khudanpur. Recurrent neural network based language model. Interspeech 2010. !  [ 15] . kUCIv7 !2<t. 2015. !  [ 15] . NLP Introduction based on Project Next NLP . PyData.Tokyo Meetup #5, 2015. !  [Bordes&Weston14] A. Bordes, J. Weston. Embedding Methods for Natural Language Processing. EMNLP2014 tutorial. !  [ 15] . j`<oCI^HGMW1+3=e. JSAI 2015 .
57. 2/4 !  [Bordes+11] A. Bordes, J. Weston, R. Collobert, Y. Bengio. Learning structured embeddings of knowledge bases. AAAI2011. !  [Bordes+13] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, O. Yakhnenko. Translating Embeddings for Modeling Multi-relational Data. NIPS 2013. !  [Fan+14] M. Fan, Q. Shou, E. Chang, T. F . Zheng. Transition-based Knowledge Graph Embedding with Relational Mapping Properties. PACLIC 2014. !  [Wang+14] Z. Wang, J. Zhang, J. Feng, Z. Chen. Knowledge Graph Embedding by Translating on Hyperplanes. AAAI 2014.
58. 3/4 !  [Socher+13] R. Socher, D. Chen, C. D. Manning, A. Y. Ng. Reasoning With Neural Tensor Networks for Knowledge Base Completion. NIPS 2013. !  [Yang+15] B. Yang, W. Yih, X. He, J. Gao, L. Deng. Embedding Entities and Relations for Learning and Inferenece in Knowledge Bases. ICLR 2015. !  [Nickel+11] M. Nickel, V. Tresp, H. P . Kriegel. A Three-Way Model for Collective Learning on Multi-Relational Data. ICML 2011.
59. 4/4 !  [Bordes+14] A. Bordes, J. Weston, N. Usunier. Open Question Answering with Weakly Supervised Embedding Models. ECML/PKDD 2014. !  [Weston+15] J. Weston, S. Chopra, A. Bordes. Memory Networks. ICLR 2015. !  [Peng&Yao15] B. Peng, K. Yao. Recurrent Neural Networks with External Memory for Language Understanding. arXiv:1506.00195, 2015.