Вольф Дж. Лекция Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?

Вольф Дж. Лекция Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?

Лекция Джереми Вольфа на семинаре "Великая иллюзия - 4" в Санкт-Петербурге.

Анонс лекции: http://cogitoergo.ru/event/wolfe-lictures-2014/
Информация о семинаре: http://cogitoergo.ru/event/grand-illusion-4/

09400c0210f32255467b2329260bd205?s=128

Cogito ergo ...

October 29, 2014
Tweet

Transcript

  1. 1.

    Jeremy M Wolfe Ophthalmology & Radiology. Harvard Medical School Visual

    Attention Lab, Brigham & Women's Hospital Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?
  2. 2.

    What am I going to tell you? The world presents

    us with endless search tasks We pay people to do important search tasks
  3. 3.

    What am I going to tell you? Laboratory search tasks

    and models… …can tell us something about how experts deal with socially important search tasks.
  4. 5.

    Let’s begin with a few thank-yous Michelle Greene BWH to

    Stanford Karla Evans BWH to York, UK Dianne Georgian-Smith BWH Robyn Birdwell BWH Trafton Drew BWH
  5. 6.

    A talk with three parts Visual search is guided Visual

    attention is limited There is an important decision component in search
  6. 7.

    First, why do we have to search at all? Which

    of these plusses did you see in the title slide?
  7. 10.

    It is hard to FIND because you need to BIND

    You need attention to bind features to objects I am a red, green, vertical, horizontal, pointy object
  8. 11.

    To recognize objects, we must bind features To bind features,

    we must attend to the object. So to find a specific object, we must search. Wolfe, J. M., & Bennett, S. C. (1997). Preattentive Object Files: Shapeless bundles of basic features. Vision Research, 37(1), 25-43.
  9. 12.

    CLAIM: Properties of the human “search engine” are the source

    of many error in medical image perception. Nodine, C. F., Mello-Thoms, C., Weinstein, S. P., Kundel, H. L., Conant, E. F., Heller-Savoy, R. E., et al. (2001). Blinded review of retrospectively visible unreported breast cancers: an eye-position analysis. Radiology, 221(1), 122-129. 20-30% Miss errors
  10. 13.

    What do we know about that human search engine that

    might shed light on visual search in tasks like radiology? http://www.adcmri.com/welcome/backendadmin/uploads/8c 24bdf59cb8b4ffa97cc7f592b39c51.gif
  11. 14.

    Visual Search is Guided 1st year 2nd year 3rd year

    Resident Expert Kundel, H. L., & La Follette, P. S., Jr. (1972). Visual search patterns and experience with radiological images. Radiology, 103(3), 523- 528. Kundel, H., L. . (2007). How to minimize perceptual error and maximize expertise in medical imaging. Paper presented at the Medical Imaging 2007: Image Perception, Observer Performance, and Technology Assessment. Random
  12. 15.

    Guidance in the lab Find the ‘5’ Some tasks are

    easy because guidance is perfect
  13. 16.

    Guidance in the lab Find the ‘5’ Some tasks are

    harder because you guide only to objects
  14. 18.

    What do the data look like? Feature Search 0 6

    12 18 500 1000 1500 2000 2500 3000 set size Conjunction Search 0 6 12 18 set size Spatial Configuration Search 0 6 12 18 set size Slopes Present: 43 msec/item Absent: 95 msec/item Slopes Present: 9.2 msec/item Absent: 26.1 msec/item Slopes Present: 1.0 msec/item Absent:-0.7 msec/item Wolfe, J. M., Palmer, E. M., & Horowitz , T. S. (2010). Reaction time distributions constrain models of visual search. Vision Res, 50, 1304-1311. Reaction time (msec)
  15. 19.

    Search is guided by a limited set of attributes Yes

    Yes No Yes No No* No* *I say….Others would argue. (They are wrong….probably) Yes Wolfe, J. M., & Horowitz, T. S. (2004). What attributes guide the deployment of visual attention and how do they do it? Nature Reviews Neuroscience, 5(6), 495-501.
  16. 20.

    There are two types of feature guidance Local differences create

    bottom-up guidance or “salience” Find the targets What grabbed your attention?
  17. 22.

    Give weight to what you want Find the red shallow

    tilted lines Top-down guidance
  18. 23.

    A weighted sum of different forms of guidance tells you

    where to deploy attention ωcolor ωorientation ωetc Σ Wolfe, J. M. (1994). Guided Search 2.0: A revised model of visual search. Psychonomic Bulletin and Review, 1(2), 202-238.
  19. 25.

    The intersection of Red and Vertical is a good place

    to find Red Verticals Red Vertical Σ Top-down Bottom-up
  20. 27.

    It is hard to set the weight on bottom-up salience

    to zero. Find the green verticals Some Rules of Guidance
  21. 28.

    Another Rule: Many simple properties do not guide Find the

    “plus” intersections (X-junctions among T-junctions)
  22. 29.

    99 msec/item 47 31 14 .3 .6 D T Another

    Rule: Many simple properties do not guide Wolfe, J. M., & DiMase, J. S. (2003). Do intersections serve as basic features in visual search? Perception, 32(6), 645-656. Reaction Time (msec)
  23. 31.

    Lindsey, D. T., Brown, A. M., Reijnen, E., Rich, A.

    N., Kuzmova, Y., & Wolfe, J. M. (2010). Color Channels, not Color Appearance or Color Categories, Guide Visual Search for Desaturated Color Targets. Psychol Sci, 21(9), 1208-1214. Pink/peach/skin(?) is much faster than pale blue or green
  24. 39.

    Today, you don’t search an image. You search a 3D

    volume of images Trafton Drew wanted to know what eye movements look like in X, Y & Z
  25. 40.

    Track eye movements in X & Y …while also tracking

    the slice as a measure of Z position. Our real question: How do radiologists move their eyes in a 3D volume?
  26. 42.

    Time Depth Here we plot movement in Z with quadrants

    in XY color-coded for one expert radiologist
  27. 44.

    Drillers & Scanners Does it matter? We don’t have enough

    data to answer that yet. Drew, T., Vo, M. L.-H., Olwal, A., Jacobson, F., Seltzer, S. E., & Wolfe, J. M. (2013). Scanners and drillers: Characterizing expert visual search through volumetric images. Journal of Vision, 13(10).
  28. 47.

    Not subtle but missed by 20 of 24 radiologists Note

    that radiologists were guiding to small white nodules
  29. 48.

    NOTE: This is not a criticism of radiologists! Just evidence

    that they, too, use the human search engine Drew, T., Vo, M. L.-H., & Wolfe, J. M. (2013). The Invisible Gorilla Strikes Again: Sustained Inattentional Blindness in Expert Observers. Psychological Science, 24(9), 1848–1853.
  30. 49.

    Volumetric datasets may encourage search errors This is related to

    the next topic: What are you really seeing right now? Previous data: Chest X-Ray
  31. 50.
  32. 51.

    Ellsworth Kelley Study for colors for a large wall 1951

    An artistic version of Wolfe, Reinecke, & Brawn. (2006). Why don’t we see changes? The role of attentional bottlenecks and limited visual memory. Visual Cognition, 19(4-8), 749-780.
  33. 52.

    Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color?
  34. 53.

    Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color? NO
  35. 54.

    Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color? NO
  36. 55.

    Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color? NO
  37. 56.

    Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color? yes
  38. 57.

    Ellsworth Kelley Study for colors for a large wall 1951

    Does the highlighted square change color? no
  39. 58.

    Claim: At any one moment you are aware of The

    gist / statistics (I see a bunch of colored squares) One object (oh, that one is red) And a theory (We will come back to that)
  40. 59.

    Did you “see” all those slices? We think you ‘saw’

    the gist but not all the ‘objects’
  41. 61.
  42. 62.
  43. 63.
  44. 64.
  45. 65.

    Theory: I can see all of these dancing chickens And,

    of course, in some sense you CAN see them
  46. 68.

    Can radiologist beat chance in a glance? We ran an

    experiment Look here Flash a mammogram for 250 msec
  47. 69.

    Can radiologist beat chance in a glance? We ran an

    experiment Look here Flash a mammogram for 250 msec
  48. 70.

    Can radiologist beat chance in a glance? We ran an

    experiment Look here Flash a mammogram for 250 msec
  49. 71.

    Would you call back this patient? YES Call back No

    Don’t call back 0 100 Use a 100-pt rating scale
  50. 72.

    We tested 40+ radiologists at the Society for Breast Imaging

    “We” = Michelle Greene MIT Karla Evans BWH Dianne Georgian-Smith BWH Robyn Birdwell BWH
  51. 73.

    0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6

    0.8 1.0 250 500 750 1000 2000 Here is how we are going to plot the data (standard ROC curve, if you are in the trade) Flash duration (msec) False Alarm Rate Hit Rate
  52. 74.

    0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6

    0.8 1.0 250 500 750 1000 2000 And here are the results Flash duration (msec) False Alarm Rate Hit Rate Evans, K., Georgian-Smith, D., Tambouret, R., Birdwell, R., & Wolfe, J. (2013). The gist of the abnormal: Above-chance medical decision making in the blink of an eye. Psychonomic Bulletin & Review, 1-6.
  53. 76.

    0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6

    0.8 1.0 250 500 750 1000 2000 No one is suggesting that the radiologist should make a decision in a quarter second! Flash duration (msec) False Alarm Rate Hit Rate Experts doing the job
  54. 77.

    But there is a non-selective signal to exploit OBJECT Wolfe,

    J. M., Vo, M. L., Evans, K. K., & Greene, M. R. (2011). Visual search in scenes involves selective and nonselective pathways. Trends Cogn Sci, 15(2), 77-84.
  55. 79.

    Disease is rare in a screening population (low target prevalence).

    negative negative negative negative negative negative negative positive negative negative negative negative
  56. 80.

    When is it time to move to the next case?

    ..and does prevalence matter?
  57. 85.

    What is the effect of Prevalence? http://jamesthecomic.com/blog1/2010/05/03/top-10-reasons-why-your-luggage-gets-lost-or-damaged/ Let’s take 20

    bags with guns and knives http://www.selectism.com/news/tag/luggage/page/4/ And put them in a stack of 40 bags (that’s 50% Prevalence)
  58. 86.

    What is the effect of Prevalence? http://jamesthecomic.com/blog1/2010/05/03/top-10-reasons-why-your-luggage-gets-lost-or-damaged/ Let’s take 20

    bags with guns and knives http://www.selectism.com/news/tag/luggage/page/4/ And put them in a stack of 40 bags 50% Prevalence Or 1000 bags 2% Prevalence
  59. 87.

    When targets are present in half of the bags people

    miss about 20% of them NOTE: THESE ARE VOLUNTEERS, NOT AIRPORT SECURITY OFFICERS
  60. 88.

    When targets are present in 2% of the bags people

    miss over 40% of them! Same threats, just rarer
  61. 89.
  62. 92.

    Basic Design Low Prevalence 100 cases (50 positive, 50 negative)

    inserted into normal workflow over the course of 9 months during which another 9826 other cases were screened. Estimated prevalence 0.8%. Data are the call back decisions. High Prevalence 100 cases (50 positive, 50 negative) each read by six radiologists (6 of 14 from the low prevalence arm). Prevalence is 50%. Reading the 100 cases took 3 hours. Data are the call back decisions and a 0-10 rating from negative to clearly abnormal.
  63. 93.

    The Key Result Miss error rates are substantially higher at

    low prevalence False alarm rates are somewhat lower at low prevalence Evans, K. K., Birdwell, R. L., & Wolfe, J. M. (2013). If You Don’t Find It Often, You Often Don’t Find It: Why Some Cancers Are Missed in Breast Cancer Screening. . PLoS ONE 8(5): e64366
  64. 94.

    But wait….there is more: Why doesn’t CAD work better? “The

    benefit would be greater if CADe were used more effectively, We have shown that radiologists only recognize a correct CADe prompt 30% of the time (Nishikawa, 2012). “ Nishikawa, R. M., Schmidt, R. A., Linver, M. N., Edwards, A. V., Papaioannou, J., & Stull, M. A. (2012). Clinically missed cancer: how effectively can radiologists use computer-aided detection? AJR Am J Roentgenol, 198(3), 708-716.
  65. 95.

    CC views Right Left Let’s do the prevalence math Prevalence

    = 0.3% Let’s say Sensitivity = 90% Specificity = 90% (D’ = 2.6) How excited are you about that mark?
  66. 96.

    Conclusions Low prevalence may be a significant source of false

    negative errors Low prevalence may be part of the reason why CAD is less effective than it should be.
  67. 97.

    Hypothesis: Progress is pushing us toward more attentional issues http://www.practicevelocity.com/urgent_care/2007/

    03/urgent-care-works-for-stern-family.html http://www.sciencephoto.com/media/439759/enlarge
  68. 99.

    Three summary conclusions Visual search is guided Visual attention is

    limited There is an important decision component in search
  69. 100.

    Melissa Vo Krista Ehinger Trafton Drew Sage Boettcher Jinxia Zhang

    Eric Chan Project Success Ray Farmer Project Success Bria Bugg CELEST Ali Cakal RSI Celeste Rousseau CELEST Beatriz Gil Gómez de Liaño Mardrid Stephanie Ding CELEST Lily Xue Gong Wheaton