Visualization for Text Analysis

Keynote presentation for Text Mining 2011

Transcript

1. http://joelf.com/other-stuff/dateme/ 1 Saturday, April 30, 2011
2. Visualization for Text Analysis John Goodall jgoodall@ornl.gov Oak Ridge National Laboratory Computational Sciences and Engineering Division 2 Saturday, April 30, 2011
3. Some research I will discuss were projects performed at Applied Visions, Inc. Secure Decisions division 3 Saturday, April 30, 2011
4. You have just developed the world’s greatest algorithm ... 4 Saturday, April 30, 2011
5. ... that identifies the meaning of everything ... 5 Saturday, April 30, 2011
6. ... mining the text from all human knowledge ... 6 Saturday, April 30, 2011
7. 7 Saturday, April 30, 2011
8. Now, you just need a user interface (to prove to the sponsor how great it is) 8 Saturday, April 30, 2011
9. Just show the result... 42 9 Saturday, April 30, 2011
10. Make it flashy... 10 Saturday, April 30, 2011
11. Make it flashy... 42 10 Saturday, April 30, 2011
12. Or just show users everything... 4390942 0911934 7572090 0614081 3862982 1328497 0424261 7847662 7851019 7689732 8076237 6254338 5578984 2243870 1841648 2598255 2472370 1359471 7298192 8284062 8575505 0284718 3735112 8427113 3607701 4656895 3669117 0719975 6805180 8138264 5364405 6781794 4677531 5626551 5998332 3358790 1060320 8009211 8069216 7966105 1976344 7101657 4296824 9781634 6357600 9575412 4378077 6142680 2275607 3421389 0451647 0556170 2875239 9225909 8962752 6614005 9790236 7315985 1037985 3127028 4959818 1551005 0192725 0513712 3648364 4689623 4147822 6176097 6447703 7016245 6146037 8958423 0281779 6398455 7868296 4839961 1594263 9842841 9631240 0623118 7774912 2622992 7140872 9017417 8677241 0465895 7665793 5078704 4924800 1146895 0671107 3757655 8816058 5499102 9774291 1988455 3534800 0812994 1957880 9463031 4683272 8067652 4852149 0590140 0536791 5056845 0099512 8296916 2500281 8258805 5229573 7091660 8480582 6050467 6521907 0339217 9274802 9888377 1167979 5040812 9906354 0423722 5802216 0108876 3826460 3433766 4683175 1375365 9773498 7589593 7890048 3777594 7126536 8700004 0287032 8091292 0467228 0649151 9790711 8553586 1365523 6853785 3278784 9425206 1977820 1550135 2580008 3326506 7440305 2383694 8096738 1927609 6950574 9310327 8399434 4399371 6451249 1608481 1806588 3620123 6993117 5865311 2504153 3135638 3670916 9884248 7999902 5896020 9555459 9936385 9183364 5317120 8495078 0429768 5554185 3696065 7184071 9634786 5003686 3102707 0035414 8510947 3101430 1743149 2933516 8689418 9075589 9782748 2217726 1626925 1062798 4381995 5487094 8132844 8730235 9046874 9315410 9859452 0700430 2682328 7400489 0704398 5850243 8124280 4102788 6597223 9417892 2530250 0628824 3144240 6575827 7341132 4408970 6889844 9340110 8249469 9425411 5701718 3305697 8307233 9921799 2362885 3164043 9505501 7421288 6128607 9712820 0745763 3283920 7084598 3760604 9961686 8222513 2470131 1839094 0401654 6890044 9340310 8249669 9425611 5701918 3305897 8307433 9921999 1508579 2058281 2629413 5277338 6046576 9700322 0987343 6734929 1633701 5370092 6670227 7111921 9861746 3573222 0421524 9819784 9420533 9969373 1394147 9357591 3740339 7886057 9247856 7626319 1159806 9932275 2973790 1671481 2936625 3251056 5250449 6920643 0328179 9761861 9342924 7943097 4270561 9471946 2956450 9568601 1609540 3731814 9222699 9093261 4079492 0886247 1912618 4577158 9039050 7096956 4880667 9457572 8923622 6542905 1024914 3090337 2450552 5252846 0891543 9781647 0817645 1324489 5213938 7515512 6707366 1119843 2596207 3446779 0704839 9491301 4430740 1271606 7805477 3093889 1172345 0004287 7868021 6239479 0606475 1263755 1215651 1675704 5599263 3408337 7329274 5990231 6743286 5774646 8387149 5014172 2766395 0556662 2576604 1544988 1438723 6621302 7777558 5029061 6372486 3914800 1218341 0396633 0777094 8374069 0555384 3002579 9961081 4091494 5771211 0092726 0358174 6974259 7310119 4088888 7050364 4914978 4965272 3652904 2896795 5144141 7353442 5760322 4781623 1656599 3697970 3715465 6042402 4578288 8612927 8680716 0122538 7570389 3263314 9135972 5750313 9044147 4965787 3226166 4907194 8375514 0129576 4586654 4776013 4214509 3342076 5461727 1804495 0612385 8706146 9039384 7097290 4881001 9457906 8923956 6115985 3035131 6321528 5653758 9963904 3239811 8939293 9722524 5966463 6973596 7736709 2505596 1808961 8907838 3363173 6510506 5091873 9518343 2921944 4487958 6975389 6883996 5672648 1864982 5037678 2535823 9021590 4770055 6772070 3413086 5197757 4335382 2192559 5226190 2400725 6565681 1220865 9544652 1839669 5693696 2913711 7897752 3878865 6563523 2500468 3940099 0562308 0018730 1900919 9692498 8754035 8579116 3173516 7071712 5536754 9835530 7799852 5224531 2826319 4981392 6832972 7411540 1037731 7548862 2214105 6275418 5261298 7481969 0773180 0079167 5898908 5532008 1967049 1559913 5044849 9552222 8429213 7111600 8913766 0805581 6439464 5355811 6987424 1768992 5958000 0383158 6317911 6931901 5212398 8368478 3113313 2646704 1707361 1958816 4969707 1521073 1668885 8583217 1468604 3833585 5744639 1174739 9152175 8930464 3558974 2964671 6158796 4706052 4052274 2822120 7113459 8061118 3972700 6495552 1485717 6587394 2270260 1978138 6871217 1214161 2101467 4015143 8593360 7206214 8419321 1374240 3901026 7186113 6944285 0097742 8226924 3214467 9595280 0100381 6947804 8392249 2883005 1111896 6006286 9923289 0289326 6385885 7946654 2992358 4223392 9932933 6453691 8961239 0344899 2942529 5536225 0262253 7894650 5157524 1385279 9212280 3355375 9493480 3578413 4439044 7861884 8796862 0677307 0999606 3746360 3255699 2119129 6769500 4692276 0020044 1047726 3286324 9332157 2204862 0111454 2259090 7094007 6159478 4706734 3625703 4405432 2354882 6173487 8739198 4973078 0242684 6420145 3454304 7721591 8899277 6772638 3413654 4771071 6345833 4996732 0012260 4457968 9762996 8489552 1536744 4848504 3097513 9466956 4660473 3233092 1923347 9888101 5208001 0073094 8456355 5602904 1702964 4090686 6197656 8082035 5021298 9782749 0391493 2908221 0455780 5628300 1047030 3712881 7321577 9400561 3220046 7464593 8446363 9865444 5012169 3618899 7389400 0415163 2677813 8493620 9831799 9505134 8463028 2618804 3642552 7158976 8452986 6881295 6951707 6687138 9644681 9214373 2502961 3088086 3730061 0075452 7604206 8770521 1759551 9793838 5702796 9077533 0899816 6371894 2489666 0559374 0928465 2469332 4811573 6840263 5634160 2061765 1563668 7451630 5719661 3775376 7334874 0383593 8350970 4355040 4011096 7154711 8869849 4215328 3779111 6484102 1329078 7419296 2523378 3329782 1953126 4740332 2150138 2852898 3376142 2172662 7962010 7066837 7367491 3580158 8796847 9737070 9654641 7303220 7187991 1639824 1292162 5072939 1345079 4017271 5879126 9968209 0538029 5606906 7802430 1087516 8661750 8441567 5219241 0301894 5786654 9529386 3881294 4796369 0243085 8118189 3897262 1394311 3385172 0259053 4716131 7252976 7819078 7472512 1254383 3152971 9588229 1336166 5717371 4413966 1466051 8289422 7538343 7220857 4623012 7027133 5872852 1243694 5492175 2254760 0526725 7945495 0681974 4988617 9205394 2772328 0385702 7925825 4721210 6189922 3841125 3301265 8119783 3685229 6471123 4093244 9719476 3482325 9634424 1555536 9152511 4041655 0562844 0291056 7912082 7698239 3200494 9381674 5350322 2447888 9489915 2386887 2460138 6938645 4797134 0030222 3407797 5368265 8620552 7159061 7806065 0236645 9607136 7296167 8676324 5251742 3284717 1521260 6381539 3230057 3216064 6192844 3203167 8957093 5503430 9916123 1594527 0860063 7141167 1633451 2567549 3974381 5022180 2189279 4346974 5712043 5476772 5443757 2460420 6725301 0086260 1500744 0847182 9691806 9436201 3655385 2636451 9303046 1934570 8567055 8427774 8196221 7312417 5274549 8394112 5212237 1790277 9184995 7024461 6511060 11 Saturday, April 30, 2011
13. Or let users query the data... SQL> 12 Saturday, April 30, 2011
14. Matching Game! • Show only a result • Show all of the data • Show query interface • No focus • No context • No overview 13 Saturday, April 30, 2011
15. Matching Game! • Show only a result • Show all of the data • Show query interface • No focus • No context • No overview 13 Saturday, April 30, 2011
16. London, 1854 14 Saturday, April 30, 2011
17. 14 Saturday, April 30, 2011
18. 15 Saturday, April 30, 2011
19. 16 Saturday, April 30, 2011
20. Visualization can convince your strongest opponents 17 Saturday, April 30, 2011
21. Russia, 1812 18 Saturday, April 30, 2011
22. 18 Saturday, April 30, 2011
23. 19 Saturday, April 30, 2011
24. Visualization can tell a story 20 Saturday, April 30, 2011
25. Information visualization is a set of technologies that use visual computing to amplify human cognition with abstract information. The use of computer-supported, interactive, visual representations of abstract data to amplify cognition. The communication of abstract data through the use of interactive visual interfaces. Information visualization utilizes computer graphics and interaction to assist humans in solving problems. Information visualization promises to help us speed our understanding and action in a world of increasing information volumes. Visual representations of the semantics, or meaning, of information. In contrast to scientific visualization, information visualization typically deals with nonnumeric, nonspatial, and high-dimensional data. The purpose of visualization is insight, not pictures InfoVis 21 Saturday, April 30, 2011
26. Information visualization is a set of technologies that use visual computing to amplify human cognition with abstract information. The use of computer-supported, interactive, visual representations of abstract data to amplify cognition. The communication of abstract data through the use of interactive visual interfaces. Information visualization utilizes computer graphics and interaction to assist humans in solving problems. Information visualization promises to help us speed our understanding and action in a world of increasing information volumes. Visual representations of the semantics, or meaning, of information. In contrast to scientific visualization, information visualization typically deals with nonnumeric, nonspatial, and high-dimensional data. The purpose of visualization is insight, not pictures 21 Saturday, April 30, 2011
27. Information visualization is a set of technologies that use visual computing to amplify human cognition with abstract  information. The use of computer-supported, interactive, visual representations of abstract data to amplify  cogni1on. The communica1on  of abstract data through the use of interactive visual interfaces. Information visualization utilizes computer graphics and interac1on  to assist humans in solving problems. Information visualization promises to help us speed our understanding and action in a world of increasing  informa1on  volumes. Visual representations of the semantics, or meaning, of information. In contrast to scientific visualization, information visualization typically deals with nonnumeric, nonspatial, and high-­‐dimensional  data. The purpose of visualization is insight, not pictures 22 Saturday, April 30, 2011
28. Information visualization is a set of technologies that use visual computing to amplify human cognition with abstract  information. The use of computer-supported, interactive, visual representations of abstract data to amplify  cogni1on. The communica1on  of abstract data through the use of interactive visual interfaces. Information visualization utilizes computer graphics and interac1on  to assist humans in solving problems. Information visualization promises to help us speed our understanding and action in a world of increasing  informa1on  volumes. Visual representations of the semantics, or meaning, of information. In contrast to scientific visualization, information visualization typically deals with nonnumeric, nonspatial, and high-­‐dimensional  data. The purpose of visualization is insight, not pictures vision highest bandwidth sense (100MB/s) 22 Saturday, April 30, 2011
29. Color Find red circle 23 Saturday, April 30, 2011
30. Color Find red circle 23 Saturday, April 30, 2011
31. Color Find red circle 23 Saturday, April 30, 2011
32. Color Find red circle 24 Saturday, April 30, 2011
33. Shape Find red circle 25 Saturday, April 30, 2011
34. Shape Find red circle 25 Saturday, April 30, 2011
35. Shape Find red circle 25 Saturday, April 30, 2011
36. Shape Find red circle 26 Saturday, April 30, 2011
37. Conjunction: Color and Shape Find red circle 27 Saturday, April 30, 2011
38. Conjunction: Color and Shape Find red circle 27 Saturday, April 30, 2011
39. Conjunction: Color and Shape Find red circle 27 Saturday, April 30, 2011
40. Conjunction: Color and Shape Find red circle ? 28 Saturday, April 30, 2011
41. http://www.gapminder.org Visualization can be used to explain ... 29 Saturday, April 30, 2011
42. Visualization can also be used to explore and analyze complex data sets 30 Saturday, April 30, 2011
43. 31 Saturday, April 30, 2011
44. Stocks by Sector 31 Saturday, April 30, 2011
45. h"p://www.smartmoney.com/map-­‐of-­‐the-­‐market/ April 2008 to April 2009 32 Saturday, April 30, 2011
46. h"p://www.smartmoney.com/map-­‐of-­‐the-­‐market/ November 2008 to April 2009 33 Saturday, April 30, 2011
47. h"p://www.smartmoney.com/map-­‐of-­‐the-­‐market/ April 2010 to April 2011 34 Saturday, April 30, 2011
48. Visualization can enable knowledge discovery 35 Saturday, April 30, 2011
49. Visualization and Text Mining 36 Saturday, April 30, 2011
50. Visualization of Text 37 Saturday, April 30, 2011
51. www.wordle.net 38 Saturday, April 30, 2011
52. http://petewarden.typepad.com/searchbrowser/2011/02/the-shape-of-my-favorite-books.html 39 Saturday, April 30, 2011
53. http://store.axismaps.com/product/manhattan 40 Saturday, April 30, 2011
54. http://hint.fm/projects/phrasenet/ 41 Saturday, April 30, 2011
55. Pretty? Useful? 42 Saturday, April 30, 2011
56. InfoVis Concepts in My Research • Simple visualizations are often the best • Multiple, coordinated visualization views • Visual information seeking mantra • Interaction to explore the data • Integration of analytics and visualization 43 Saturday, April 30, 2011
57. Visualization of Relationships in Text 44 Saturday, April 30, 2011
58. Copyright 2010, Applied Visions, Inc., Secure Decisions Division 45 Saturday, April 30, 2011
59. Copyright 2010, Applied Visions, Inc., Secure Decisions Division 46 Saturday, April 30, 2011
60. Copyright 2010, Applied Visions, Inc., Secure Decisions Division 47 Saturday, April 30, 2011
61. Copyright 2010, Applied Visions, Inc., Secure Decisions Division 48 Saturday, April 30, 2011
62. Visualization of Documents 49 Saturday, April 30, 2011
63. • Each source code file is represented as a block • Each block aggregates the vulnerabilities found • Flexible data > visual mappings (color/sort) Visualization Layout Copyright 2010, Applied Visions, Inc., Secure Decisions Division 50 Saturday, April 30, 2011
64. Copyright 2010, Applied Visions, Inc., Secure Decisions Division 51 Saturday, April 30, 2011
65. Copyright 2010, Applied Visions, Inc., Secure Decisions Division 52 Saturday, April 30, 2011
66. Copyright 2010, Applied Visions, Inc., Secure Decisions Division 53 Saturday, April 30, 2011
67. Copyright 2010, Applied Visions, Inc., Secure Decisions Division 54 Saturday, April 30, 2011
68. Copyright 2010, Applied Visions, Inc., Secure Decisions Division 55 Saturday, April 30, 2011
69. Copyright 2010, Applied Visions, Inc., Secure Decisions Division 56 Saturday, April 30, 2011
70. Future of InfoVis Data Visualization Data Analytics Visualization Data Analytics Visualization Then Now Next 57 Saturday, April 30, 2011
71. Resources 58 Saturday, April 30, 2011
72. Learning more about InfoVis • Conferences: IEEE InfoVis, EuroVis, VizSec • Journals: Information Visualization, IEEE Computer Graphics & Applications, IEEE Trans. on Visualization & Computer Graphics • Web: visualcomplexity.com, infovis-wiki.net, infosthetics.com, flowingdata.com, visualizing.org, eagereyes.org, ... 59 Saturday, April 30, 2011
73. Visualization Toolkits http://flare.prefuse.org/ http://prefuse.org/ http://processing.org/ Prefuse Flare Processing http://vis.stanford.edu/protovis/ Protovis d3 http://mbostock.github.com/d3/ Polymaps http://polymaps.org/ 60 Saturday, April 30, 2011
74. Summary • Visualization is about insights, not pretty pictures • Multiple, simple, linked visualizations are effective • Overview first, zoom and filter, details on demand • Interaction is key to generating insights • Tighter integration of analytics and visualization 61 Saturday, April 30, 2011
75. Questions? John Goodall jgoodall@ornl.gov Oak Ridge National Laboratory Computational Sciences and Engineering Division 62 Saturday, April 30, 2011