Вольф Дж. Лекция Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?

Лекция Джереми Вольфа на семинаре "Великая иллюзия - 4" в Санкт-Петербурге.

Анонс лекции: http://cogitoergo.ru/event/wolfe-lictures-2014/
Информация о семинаре: http://cogitoergo.ru/event/grand-illusion-4/

Transcript

1. Jeremy M Wolfe Ophthalmology & Radiology. Harvard Medical School Visual Attention Lab, Brigham & Women's Hospital Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?
2. What am I going to tell you? The world presents us with endless search tasks We pay people to do important search tasks
3. What am I going to tell you? Laboratory search tasks and models… …can tell us something about how experts deal with socially important search tasks.
4. And…. The behavior of experts…. …can tell use something about the basic rules of visual search.
5. Let’s begin with a few thank-yous Michelle Greene BWH to Stanford Karla Evans BWH to York, UK Dianne Georgian-Smith BWH Robyn Birdwell BWH Trafton Drew BWH
6. A talk with three parts Visual search is guided Visual attention is limited There is an important decision component in search
7. First, why do we have to search at all? Which of these plusses did you see in the title slide?
8. Let’s not rely on memory: Look for this one
9. Find: Not hard to SEE. Hard to FIND (Did you find both of them?)
10. It is hard to FIND because you need to BIND You need attention to bind features to objects I am a red, green, vertical, horizontal, pointy object
11. To recognize objects, we must bind features To bind features, we must attend to the object. So to find a specific object, we must search. Wolfe, J. M., & Bennett, S. C. (1997). Preattentive Object Files: Shapeless bundles of basic features. Vision Research, 37(1), 25-43.
12. CLAIM: Properties of the human “search engine” are the source of many error in medical image perception. Nodine, C. F., Mello-Thoms, C., Weinstein, S. P., Kundel, H. L., Conant, E. F., Heller-Savoy, R. E., et al. (2001). Blinded review of retrospectively visible unreported breast cancers: an eye-position analysis. Radiology, 221(1), 122-129. 20-30% Miss errors
13. What do we know about that human search engine that might shed light on visual search in tasks like radiology? http://www.adcmri.com/welcome/backendadmin/uploads/8c 24bdf59cb8b4ffa97cc7f592b39c51.gif
14. Visual Search is Guided 1st year 2nd year 3rd year Resident Expert Kundel, H. L., & La Follette, P. S., Jr. (1972). Visual search patterns and experience with radiological images. Radiology, 103(3), 523- 528. Kundel, H., L. . (2007). How to minimize perceptual error and maximize expertise in medical imaging. Paper presented at the Medical Imaging 2007: Image Perception, Observer Performance, and Technology Assessment. Random
15. Guidance in the lab Find the ‘5’ Some tasks are easy because guidance is perfect
16. Guidance in the lab Find the ‘5’ Some tasks are harder because you guide only to objects
17. Guidance in the lab Find the ‘green 5’ Some tasks are in- between
18. What do the data look like? Feature Search 0 6 12 18 500 1000 1500 2000 2500 3000 set size Conjunction Search 0 6 12 18 set size Spatial Configuration Search 0 6 12 18 set size Slopes Present: 43 msec/item Absent: 95 msec/item Slopes Present: 9.2 msec/item Absent: 26.1 msec/item Slopes Present: 1.0 msec/item Absent:-0.7 msec/item Wolfe, J. M., Palmer, E. M., & Horowitz , T. S. (2010). Reaction time distributions constrain models of visual search. Vision Res, 50, 1304-1311. Reaction time (msec)
19. Search is guided by a limited set of attributes Yes Yes No Yes No No* No* *I say….Others would argue. (They are wrong….probably) Yes Wolfe, J. M., & Horowitz, T. S. (2004). What attributes guide the deployment of visual attention and how do they do it? Nature Reviews Neuroscience, 5(6), 495-501.
20. There are two types of feature guidance Local differences create bottom-up guidance or “salience” Find the targets What grabbed your attention?
21. Find the target Bottom-up guidance is not enough
22. Give weight to what you want Find the red shallow tilted lines Top-down guidance
23. A weighted sum of different forms of guidance tells you where to deploy attention ωcolor ωorientation ωetc Σ Wolfe, J. M. (1994). Guided Search 2.0: A revised model of visual search. Psychonomic Bulletin and Review, 1(2), 202-238.
24. Guided Search for conjunctions – find a red vertical target
25. The intersection of Red and Vertical is a good place to find Red Verticals Red Vertical Σ Top-down Bottom-up
26. Why did guidance fail?
27. It is hard to set the weight on bottom-up salience to zero. Find the green verticals Some Rules of Guidance
28. Another Rule: Many simple properties do not guide Find the “plus” intersections (X-junctions among T-junctions)
29. 99 msec/item 47 31 14 .3 .6 D T Another Rule: Many simple properties do not guide Wolfe, J. M., & DiMase, J. S. (2003). Do intersections serve as basic features in visual search? Perception, 32(6), 645-656. Reaction Time (msec)
30. One more: Each attribute has its own rules Find two desaturated/pale targets
31. Lindsey, D. T., Brown, A. M., Reijnen, E., Rich, A. N., Kuzmova, Y., & Wolfe, J. M. (2010). Color Channels, not Color Appearance or Color Categories, Guide Visual Search for Desaturated Color Targets. Psychol Sci, 21(9), 1208-1214. Pink/peach/skin(?) is much faster than pale blue or green
32. http://www.diagnosticimaging.com/breast-imaging/image-iq-42-year-old-architectural-distortion-breast-tomo You don’t learn new basic features. You learn to guide more effectively. Architectural Distortions (et al) are not going to be a basic features
33. http://mentalfloss.com/article/19567/zany-history-mini-golf There is more to guidance than feature guidance Look for the golf balls (or “nodules” as we might call them)
34. http://mentalfloss.com/article/19567/zany-history-mini-golf How did you do? Sure Probably How about these five? Or this one?
35. Notice that the one you missed …is more salient than the two you found
36. http://mentalfloss.com/article/19567/zany-history-mini-golf Introducing “scene guidance” Sure Probably Syntactic guidance (Where is it possible)? Semantic guidance (Where is it sensible)?
37. So when a novice radiologist…… (eye tracking image from Elizabeth Krupinski)
38. …becomes an expert, a big piece of that is guidance
39. Today, you don’t search an image. You search a 3D volume of images Trafton Drew wanted to know what eye movements look like in X, Y & Z
40. Track eye movements in X & Y …while also tracking the slice as a measure of Z position. Our real question: How do radiologists move their eyes in a 3D volume?
41. One way to look at the data Let’s color code the quadrants
42. Time Depth Here we plot movement in Z with quadrants in XY color-coded for one expert radiologist
43. But here is another expert Depth Time
44. Drillers & Scanners Does it matter? We don’t have enough data to answer that yet. Drew, T., Vo, M. L.-H., Olwal, A., Jacobson, F., Seltzer, S. E., & Wolfe, J. M. (2013). Scanners and drillers: Characterizing expert visual search through volumetric images. Journal of Vision, 13(10).
45. One more case
46. There was something odd about this case
47. Not subtle but missed by 20 of 24 radiologists Note that radiologists were guiding to small white nodules
48. NOTE: This is not a criticism of radiologists! Just evidence that they, too, use the human search engine Drew, T., Vo, M. L.-H., & Wolfe, J. M. (2013). The Invisible Gorilla Strikes Again: Sustained Inattentional Blindness in Expert Observers. Psychological Science, 24(9), 1848–1853.
49. Volumetric datasets may encourage search errors This is related to the next topic: What are you really seeing right now? Previous data: Chest X-Ray
50. What do you really know about what you see? Try this extremely simple task
51. Ellsworth Kelley Study for colors for a large wall 1951 An artistic version of Wolfe, Reinecke, & Brawn. (2006). Why don’t we see changes? The role of attentional bottlenecks and limited visual memory. Visual Cognition, 19(4-8), 749-780.
52. Ellsworth Kelley Study for colors for a large wall 1951 Does the highlighted square change color?
53. Ellsworth Kelley Study for colors for a large wall 1951 Does the highlighted square change color? NO
54. Ellsworth Kelley Study for colors for a large wall 1951 Does the highlighted square change color? NO
55. Ellsworth Kelley Study for colors for a large wall 1951 Does the highlighted square change color? NO
56. Ellsworth Kelley Study for colors for a large wall 1951 Does the highlighted square change color? yes
57. Ellsworth Kelley Study for colors for a large wall 1951 Does the highlighted square change color? no
58. Claim: At any one moment you are aware of The gist / statistics (I see a bunch of colored squares) One object (oh, that one is red) And a theory (We will come back to that)
59. Did you “see” all those slices? We think you ‘saw’ the gist but not all the ‘objects’
60. Let me illustrate with some dancing chickens
65. Theory: I can see all of these dancing chickens And, of course, in some sense you CAN see them
66. Don’t discount gist: Radiologists use The Force
67. Experts have “feelings” (Don’t get me started about the book…. Actually, it is interesting but …)
68. Can radiologist beat chance in a glance? We ran an experiment Look here Flash a mammogram for 250 msec
69. Can radiologist beat chance in a glance? We ran an experiment Look here Flash a mammogram for 250 msec
70. Can radiologist beat chance in a glance? We ran an experiment Look here Flash a mammogram for 250 msec
71. Would you call back this patient? YES Call back No Don’t call back 0 100 Use a 100-pt rating scale
72. We tested 40+ radiologists at the Society for Breast Imaging “We” = Michelle Greene MIT Karla Evans BWH Dianne Georgian-Smith BWH Robyn Birdwell BWH
73. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 250 500 750 1000 2000 Here is how we are going to plot the data (standard ROC curve, if you are in the trade) Flash duration (msec) False Alarm Rate Hit Rate
74. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 250 500 750 1000 2000 And here are the results Flash duration (msec) False Alarm Rate Hit Rate Evans, K., Georgian-Smith, D., Tambouret, R., Birdwell, R., & Wolfe, J. (2013). The gist of the abnormal: Above-chance medical decision making in the blink of an eye. Psychonomic Bulletin & Review, 1-6.
75. A non-selective signal: Localization is at chance …. Independent of confidence
76. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 250 500 750 1000 2000 No one is suggesting that the radiologist should make a decision in a quarter second! Flash duration (msec) False Alarm Rate Hit Rate Experts doing the job
77. But there is a non-selective signal to exploit OBJECT Wolfe, J. M., Vo, M. L., Evans, K. K., & Greene, M. R. (2011). Visual search in scenes involves selective and nonselective pathways. Trends Cogn Sci, 15(2), 77-84.
78. Finally, a word about decisions in search When do is it time to quit?
79. Disease is rare in a screening population (low target prevalence). negative negative negative negative negative negative negative positive negative negative negative negative
80. When is it time to move to the next case? ..and does prevalence matter?
81. Not just a radiology problem Cervical cancer screening
82. Not just a medical Problem Airport Security
83. That is a knife of some sort
84. Let’s do an experiment Let’s take 20 bags with guns and knives
85. What is the effect of Prevalence? http://jamesthecomic.com/blog1/2010/05/03/top-10-reasons-why-your-luggage-gets-lost-or-damaged/ Let’s take 20 bags with guns and knives http://www.selectism.com/news/tag/luggage/page/4/ And put them in a stack of 40 bags (that’s 50% Prevalence)
86. What is the effect of Prevalence? http://jamesthecomic.com/blog1/2010/05/03/top-10-reasons-why-your-luggage-gets-lost-or-damaged/ Let’s take 20 bags with guns and knives http://www.selectism.com/news/tag/luggage/page/4/ And put them in a stack of 40 bags 50% Prevalence Or 1000 bags 2% Prevalence
87. When targets are present in half of the bags people miss about 20% of them NOTE: THESE ARE VOLUNTEERS, NOT AIRPORT SECURITY OFFICERS
88. When targets are present in 2% of the bags people miss over 40% of them! Same threats, just rarer
89. False alarm errors go the other way. Which kind of error do you want to minimize?
90. We did this in the breast cancer screening clinic of our hospital
91. thank-you to Karla Evans Robyn Birdwell
92. Basic Design Low Prevalence 100 cases (50 positive, 50 negative) inserted into normal workflow over the course of 9 months during which another 9826 other cases were screened. Estimated prevalence 0.8%. Data are the call back decisions. High Prevalence 100 cases (50 positive, 50 negative) each read by six radiologists (6 of 14 from the low prevalence arm). Prevalence is 50%. Reading the 100 cases took 3 hours. Data are the call back decisions and a 0-10 rating from negative to clearly abnormal.
93. The Key Result Miss error rates are substantially higher at low prevalence False alarm rates are somewhat lower at low prevalence Evans, K. K., Birdwell, R. L., & Wolfe, J. M. (2013). If You Don’t Find It Often, You Often Don’t Find It: Why Some Cancers Are Missed in Breast Cancer Screening. . PLoS ONE 8(5): e64366
94. But wait….there is more: Why doesn’t CAD work better? “The benefit would be greater if CADe were used more effectively, We have shown that radiologists only recognize a correct CADe prompt 30% of the time (Nishikawa, 2012). “ Nishikawa, R. M., Schmidt, R. A., Linver, M. N., Edwards, A. V., Papaioannou, J., & Stull, M. A. (2012). Clinically missed cancer: how effectively can radiologists use computer-aided detection? AJR Am J Roentgenol, 198(3), 708-716.
95. CC views Right Left Let’s do the prevalence math Prevalence = 0.3% Let’s say Sensitivity = 90% Specificity = 90% (D’ = 2.6) How excited are you about that mark?
96. Conclusions Low prevalence may be a significant source of false negative errors Low prevalence may be part of the reason why CAD is less effective than it should be.
97. Hypothesis: Progress is pushing us toward more attentional issues http://www.practicevelocity.com/urgent_care/2007/ 03/urgent-care-works-for-stern-family.html http://www.sciencephoto.com/media/439759/enlarge
98. Hypothesis about errors http://www.practicevelocity.com/urgent_care/2007/ 03/urgent-care-works-for-stern-family.html http://www.sciencephoto.com/media/439759/enlarge Can’t see it Can’t find it
99. Three summary conclusions Visual search is guided Visual attention is limited There is an important decision component in search
100. Melissa Vo Krista Ehinger Trafton Drew Sage Boettcher Jinxia Zhang Eric Chan Project Success Ray Farmer Project Success Bria Bugg CELEST Ali Cakal RSI Celeste Rousseau CELEST Beatriz Gil Gómez de Liaño Mardrid Stephanie Ding CELEST Lily Xue Gong Wheaton