Вольф Дж. Лекция Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?

Вольф Дж. Лекция Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?

Лекция Джереми Вольфа на семинаре "Великая иллюзия - 4" в Санкт-Петербурге.

Анонс лекции: http://cogitoergo.ru/event/wolfe-lictures-2014/
Информация о семинаре: http://cogitoergo.ru/event/grand-illusion-4/

09400c0210f32255467b2329260bd205?s=128

Cogito ergo ...

October 29, 2014
Tweet

Transcript

  1. Jeremy M Wolfe Ophthalmology & Radiology. Harvard Medical School Visual

    Attention Lab, Brigham & Women's Hospital Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?
  2. What am I going to tell you? The world presents

    us with endless search tasks We pay people to do important search tasks
  3. What am I going to tell you? Laboratory search tasks

    and models… …can tell us something about how experts deal with socially important search tasks.
  4. And…. The behavior of experts…. …can tell use something about

    the basic rules of visual search.
  5. Let’s begin with a few thank-yous Michelle Greene BWH to

    Stanford Karla Evans BWH to York, UK Dianne Georgian-Smith BWH Robyn Birdwell BWH Trafton Drew BWH
  6. A talk with three parts Visual search is guided Visual

    attention is limited There is an important decision component in search
  7. First, why do we have to search at all? Which

    of these plusses did you see in the title slide?
  8. Let’s not rely on memory: Look for this one

  9. Find: Not hard to SEE. Hard to FIND (Did you

    find both of them?)
  10. It is hard to FIND because you need to BIND

    You need attention to bind features to objects I am a red, green, vertical, horizontal, pointy object
  11. To recognize objects, we must bind features To bind features,

    we must attend to the object. So to find a specific object, we must search. Wolfe, J. M., & Bennett, S. C. (1997). Preattentive Object Files: Shapeless bundles of basic features. Vision Research, 37(1), 25-43.
  12. CLAIM: Properties of the human “search engine” are the source

    of many error in medical image perception. Nodine, C. F., Mello-Thoms, C., Weinstein, S. P., Kundel, H. L., Conant, E. F., Heller-Savoy, R. E., et al. (2001). Blinded review of retrospectively visible unreported breast cancers: an eye-position analysis. Radiology, 221(1), 122-129. 20-30% Miss errors
  13. What do we know about that human search engine that

    might shed light on visual search in tasks like radiology? http://www.adcmri.com/welcome/backendadmin/uploads/8c 24bdf59cb8b4ffa97cc7f592b39c51.gif
  14. Visual Search is Guided 1st year 2nd year 3rd year

    Resident Expert Kundel, H. L., & La Follette, P. S., Jr. (1972). Visual search patterns and experience with radiological images. Radiology, 103(3), 523- 528. Kundel, H., L. . (2007). How to minimize perceptual error and maximize expertise in medical imaging. Paper presented at the Medical Imaging 2007: Image Perception, Observer Performance, and Technology Assessment. Random
  15. Guidance in the lab Find the ‘5’ Some tasks are

    easy because guidance is perfect
  16. Guidance in the lab Find the ‘5’ Some tasks are

    harder because you guide only to objects
  17. Guidance in the lab Find the ‘green 5’ Some tasks

    are in- between
  18. What do the data look like? Feature Search 0 6

    12 18 500 1000 1500 2000 2500 3000 set size Conjunction Search 0 6 12 18 set size Spatial Configuration Search 0 6 12 18 set size Slopes Present: 43 msec/item Absent: 95 msec/item Slopes Present: 9.2 msec/item Absent: 26.1 msec/item Slopes Present: 1.0 msec/item Absent:-0.7 msec/item Wolfe, J. M., Palmer, E. M., & Horowitz , T. S. (2010). Reaction time distributions constrain models of visual search. Vision Res, 50, 1304-1311. Reaction time (msec)
  19. Search is guided by a limited set of attributes Yes

    Yes No Yes No No* No* *I say….Others would argue. (They are wrong….probably) Yes Wolfe, J. M., & Horowitz, T. S. (2004). What attributes guide the deployment of visual attention and how do they do it? Nature Reviews Neuroscience, 5(6), 495-501.
  20. There are two types of feature guidance Local differences create

    bottom-up guidance or “salience” Find the targets What grabbed your attention?
  21. Find the target Bottom-up guidance is not enough

  22. Give weight to what you want Find the red shallow

    tilted lines Top-down guidance
  23. A weighted sum of different forms of guidance tells you

    where to deploy attention ωcolor ωorientation ωetc Σ Wolfe, J. M. (1994). Guided Search 2.0: A revised model of visual search. Psychonomic Bulletin and Review, 1(2), 202-238.
  24. Guided Search for conjunctions – find a red vertical target

  25. The intersection of Red and Vertical is a good place

    to find Red Verticals Red Vertical Σ Top-down Bottom-up
  26. Why did guidance fail?

  27. It is hard to set the weight on bottom-up salience

    to zero. Find the green verticals Some Rules of Guidance
  28. Another Rule: Many simple properties do not guide Find the

    “plus” intersections (X-junctions among T-junctions)
  29. 99 msec/item 47 31 14 .3 .6 D T Another

    Rule: Many simple properties do not guide Wolfe, J. M., & DiMase, J. S. (2003). Do intersections serve as basic features in visual search? Perception, 32(6), 645-656. Reaction Time (msec)
  30. One more: Each attribute has its own rules Find two

    desaturated/pale targets
  31. Lindsey, D. T., Brown, A. M., Reijnen, E., Rich, A.

    N., Kuzmova, Y., & Wolfe, J. M. (2010). Color Channels, not Color Appearance or Color Categories, Guide Visual Search for Desaturated Color Targets. Psychol Sci, 21(9), 1208-1214. Pink/peach/skin(?) is much faster than pale blue or green
  32. http://www.diagnosticimaging.com/breast-imaging/image-iq-42-year-old-architectural-distortion-breast-tomo You don’t learn new basic features. You learn to

    guide more effectively. Architectural Distortions (et al) are not going to be a basic features
  33. http://mentalfloss.com/article/19567/zany-history-mini-golf There is more to guidance than feature guidance Look

    for the golf balls (or “nodules” as we might call them)
  34. http://mentalfloss.com/article/19567/zany-history-mini-golf How did you do? Sure Probably How about these

    five? Or this one?
  35. Notice that the one you missed …is more salient than

    the two you found
  36. http://mentalfloss.com/article/19567/zany-history-mini-golf Introducing “scene guidance” Sure Probably Syntactic guidance (Where is

    it possible)? Semantic guidance (Where is it sensible)?
  37. So when a novice radiologist…… (eye tracking image from Elizabeth

    Krupinski)
  38. …becomes an expert, a big piece of that is guidance

  39. Today, you don’t search an image. You search a 3D

    volume of images Trafton Drew wanted to know what eye movements look like in X, Y & Z
  40. Track eye movements in X & Y …while also tracking

    the slice as a measure of Z position. Our real question: How do radiologists move their eyes in a 3D volume?
  41. One way to look at the data Let’s color code

    the quadrants
  42. Time Depth Here we plot movement in Z with quadrants

    in XY color-coded for one expert radiologist
  43. But here is another expert Depth Time

  44. Drillers & Scanners Does it matter? We don’t have enough

    data to answer that yet. Drew, T., Vo, M. L.-H., Olwal, A., Jacobson, F., Seltzer, S. E., & Wolfe, J. M. (2013). Scanners and drillers: Characterizing expert visual search through volumetric images. Journal of Vision, 13(10).
  45. One more case

  46. There was something odd about this case

  47. Not subtle but missed by 20 of 24 radiologists Note

    that radiologists were guiding to small white nodules
  48. NOTE: This is not a criticism of radiologists! Just evidence

    that they, too, use the human search engine Drew, T., Vo, M. L.-H., & Wolfe, J. M. (2013). The Invisible Gorilla Strikes Again: Sustained Inattentional Blindness in Expert Observers. Psychological Science, 24(9), 1848–1853.
  49. Volumetric datasets may encourage search errors This is related to

    the next topic: What are you really seeing right now? Previous data: Chest X-Ray
  50. What do you really know about what you see? Try

    this extremely simple task
  51. Ellsworth Kelley Study for colors for a large wall 1951

    An artistic version of Wolfe, Reinecke, & Brawn. (2006). Why don’t we see changes? The role of attentional bottlenecks and limited visual memory. Visual Cognition, 19(4-8), 749-780.
  52. Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color?
  53. Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color? NO
  54. Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color? NO
  55. Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color? NO
  56. Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color? yes
  57. Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color? no
  58. Claim: At any one moment you are aware of The

    gist / statistics (I see a bunch of colored squares) One object (oh, that one is red) And a theory (We will come back to that)
  59. Did you “see” all those slices? We think you ‘saw’

    the gist but not all the ‘objects’
  60. Let me illustrate with some dancing chickens

  61. None
  62. None
  63. None
  64. None
  65. Theory: I can see all of these dancing chickens And,

    of course, in some sense you CAN see them
  66. Don’t discount gist: Radiologists use The Force

  67. Experts have “feelings” (Don’t get me started about the book….

    Actually, it is interesting but …)
  68. Can radiologist beat chance in a glance? We ran an

    experiment Look here Flash a mammogram for 250 msec
  69. Can radiologist beat chance in a glance? We ran an

    experiment Look here Flash a mammogram for 250 msec
  70. Can radiologist beat chance in a glance? We ran an

    experiment Look here Flash a mammogram for 250 msec
  71. Would you call back this patient? YES Call back No

    Don’t call back 0 100 Use a 100-pt rating scale
  72. We tested 40+ radiologists at the Society for Breast Imaging

    “We” = Michelle Greene MIT Karla Evans BWH Dianne Georgian-Smith BWH Robyn Birdwell BWH
  73. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6

    0.8 1.0 250 500 750 1000 2000 Here is how we are going to plot the data (standard ROC curve, if you are in the trade) Flash duration (msec) False Alarm Rate Hit Rate
  74. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6

    0.8 1.0 250 500 750 1000 2000 And here are the results Flash duration (msec) False Alarm Rate Hit Rate Evans, K., Georgian-Smith, D., Tambouret, R., Birdwell, R., & Wolfe, J. (2013). The gist of the abnormal: Above-chance medical decision making in the blink of an eye. Psychonomic Bulletin & Review, 1-6.
  75. A non-selective signal: Localization is at chance …. Independent of

    confidence
  76. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6

    0.8 1.0 250 500 750 1000 2000 No one is suggesting that the radiologist should make a decision in a quarter second! Flash duration (msec) False Alarm Rate Hit Rate Experts doing the job
  77. But there is a non-selective signal to exploit OBJECT Wolfe,

    J. M., Vo, M. L., Evans, K. K., & Greene, M. R. (2011). Visual search in scenes involves selective and nonselective pathways. Trends Cogn Sci, 15(2), 77-84.
  78. Finally, a word about decisions in search When do is

    it time to quit?
  79. Disease is rare in a screening population (low target prevalence).

    negative negative negative negative negative negative negative positive negative negative negative negative
  80. When is it time to move to the next case?

    ..and does prevalence matter?
  81. Not just a radiology problem Cervical cancer screening

  82. Not just a medical Problem Airport Security

  83. That is a knife of some sort

  84. Let’s do an experiment Let’s take 20 bags with guns

    and knives
  85. What is the effect of Prevalence? http://jamesthecomic.com/blog1/2010/05/03/top-10-reasons-why-your-luggage-gets-lost-or-damaged/ Let’s take 20

    bags with guns and knives http://www.selectism.com/news/tag/luggage/page/4/ And put them in a stack of 40 bags (that’s 50% Prevalence)
  86. What is the effect of Prevalence? http://jamesthecomic.com/blog1/2010/05/03/top-10-reasons-why-your-luggage-gets-lost-or-damaged/ Let’s take 20

    bags with guns and knives http://www.selectism.com/news/tag/luggage/page/4/ And put them in a stack of 40 bags 50% Prevalence Or 1000 bags 2% Prevalence
  87. When targets are present in half of the bags people

    miss about 20% of them NOTE: THESE ARE VOLUNTEERS, NOT AIRPORT SECURITY OFFICERS
  88. When targets are present in 2% of the bags people

    miss over 40% of them! Same threats, just rarer
  89. False alarm errors go the other way. Which kind of

    error do you want to minimize?
  90. We did this in the breast cancer screening clinic of

    our hospital
  91. thank-you to Karla Evans Robyn Birdwell

  92. Basic Design Low Prevalence 100 cases (50 positive, 50 negative)

    inserted into normal workflow over the course of 9 months during which another 9826 other cases were screened. Estimated prevalence 0.8%. Data are the call back decisions. High Prevalence 100 cases (50 positive, 50 negative) each read by six radiologists (6 of 14 from the low prevalence arm). Prevalence is 50%. Reading the 100 cases took 3 hours. Data are the call back decisions and a 0-10 rating from negative to clearly abnormal.
  93. The Key Result Miss error rates are substantially higher at

    low prevalence False alarm rates are somewhat lower at low prevalence Evans, K. K., Birdwell, R. L., & Wolfe, J. M. (2013). If You Don’t Find It Often, You Often Don’t Find It: Why Some Cancers Are Missed in Breast Cancer Screening. . PLoS ONE 8(5): e64366
  94. But wait….there is more: Why doesn’t CAD work better? “The

    benefit would be greater if CADe were used more effectively, We have shown that radiologists only recognize a correct CADe prompt 30% of the time (Nishikawa, 2012). “ Nishikawa, R. M., Schmidt, R. A., Linver, M. N., Edwards, A. V., Papaioannou, J., & Stull, M. A. (2012). Clinically missed cancer: how effectively can radiologists use computer-aided detection? AJR Am J Roentgenol, 198(3), 708-716.
  95. CC views Right Left Let’s do the prevalence math Prevalence

    = 0.3% Let’s say Sensitivity = 90% Specificity = 90% (D’ = 2.6) How excited are you about that mark?
  96. Conclusions Low prevalence may be a significant source of false

    negative errors Low prevalence may be part of the reason why CAD is less effective than it should be.
  97. Hypothesis: Progress is pushing us toward more attentional issues http://www.practicevelocity.com/urgent_care/2007/

    03/urgent-care-works-for-stern-family.html http://www.sciencephoto.com/media/439759/enlarge
  98. Hypothesis about errors http://www.practicevelocity.com/urgent_care/2007/ 03/urgent-care-works-for-stern-family.html http://www.sciencephoto.com/media/439759/enlarge Can’t see it Can’t

    find it
  99. Three summary conclusions Visual search is guided Visual attention is

    limited There is an important decision component in search
  100. Melissa Vo Krista Ehinger Trafton Drew Sage Boettcher Jinxia Zhang

    Eric Chan Project Success Ray Farmer Project Success Bria Bugg CELEST Ali Cakal RSI Celeste Rousseau CELEST Beatriz Gil Gómez de Liaño Mardrid Stephanie Ding CELEST Lily Xue Gong Wheaton