Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Вольф Дж. Лекция Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?

Вольф Дж. Лекция Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?

Лекция Джереми Вольфа на семинаре "Великая иллюзия - 4" в Санкт-Петербурге.

Анонс лекции: http://cogitoergo.ru/event/wolfe-lictures-2014/
Информация о семинаре: http://cogitoergo.ru/event/grand-illusion-4/

Cogito ergo ...

October 29, 2014
Tweet

More Decks by Cogito ergo ...

Other Decks in Science

Transcript

  1. Jeremy M Wolfe
    Ophthalmology & Radiology. Harvard Medical School
    Visual Attention Lab, Brigham & Women's Hospital
    Dancing chickens and gorillas in the lung:
    If I can see so much, why do I miss so much?

    View full-size slide

  2. What am I going to tell you?
    The world presents us
    with endless search tasks
    We pay people to do
    important search tasks

    View full-size slide

  3. What am I going to tell you?
    Laboratory search tasks
    and models…
    …can tell us something
    about how experts deal
    with socially important
    search tasks.

    View full-size slide

  4. And….
    The behavior of
    experts….
    …can tell use something
    about the basic rules of
    visual search.

    View full-size slide

  5. Let’s begin with a few thank-yous
    Michelle
    Greene
    BWH to Stanford
    Karla
    Evans
    BWH to York, UK
    Dianne
    Georgian-Smith
    BWH
    Robyn
    Birdwell
    BWH
    Trafton
    Drew
    BWH

    View full-size slide

  6. A talk with three parts
    Visual search is guided
    Visual attention is limited
    There is an important
    decision component in
    search

    View full-size slide

  7. First,
    why do
    we have
    to search
    at all?
    Which of these
    plusses did you see
    in the title slide?

    View full-size slide

  8. Let’s not rely on memory:
    Look for this one

    View full-size slide

  9. Find: Not hard to SEE. Hard to FIND (Did you find both of them?)

    View full-size slide

  10. It is hard to FIND
    because you need to BIND
    You need attention to bind features to objects
    I am a red, green, vertical,
    horizontal, pointy object

    View full-size slide

  11. To recognize objects, we must bind features
    To bind features, we must attend to the object.
    So to find a specific object, we must search.
    Wolfe, J. M., & Bennett, S. C. (1997). Preattentive Object Files: Shapeless bundles of basic
    features. Vision Research, 37(1), 25-43.

    View full-size slide

  12. CLAIM:
    Properties of the human “search engine”
    are the source of many error in medical
    image perception.
    Nodine, C. F., Mello-Thoms, C.,
    Weinstein, S. P., Kundel, H. L.,
    Conant, E. F., Heller-Savoy, R. E.,
    et al. (2001). Blinded review of
    retrospectively
    visible
    unreported
    breast cancers:
    an eye-position analysis.
    Radiology, 221(1), 122-129.
    20-30% Miss errors

    View full-size slide

  13. What do we know about that human
    search engine that might shed light on
    visual search in tasks like radiology?
    http://www.adcmri.com/welcome/backendadmin/uploads/8c
    24bdf59cb8b4ffa97cc7f592b39c51.gif

    View full-size slide

  14. Visual Search is Guided
    1st year 2nd year 3rd year Resident Expert
    Kundel, H. L., & La Follette, P. S., Jr. (1972). Visual search patterns
    and experience with radiological images. Radiology, 103(3), 523-
    528.
    Kundel, H., L. . (2007). How to minimize perceptual error and
    maximize expertise in medical imaging. Paper presented at the
    Medical Imaging 2007: Image Perception, Observer Performance,
    and Technology Assessment.
    Random

    View full-size slide

  15. Guidance in the lab
    Find the ‘5’
    Some tasks
    are easy
    because
    guidance is
    perfect

    View full-size slide

  16. Guidance in the lab
    Find the ‘5’
    Some tasks
    are harder
    because you
    guide only to
    objects

    View full-size slide

  17. Guidance in the lab
    Find the ‘green 5’
    Some tasks
    are in-
    between

    View full-size slide

  18. What do the data look like?
    Feature Search
    0 6 12 18
    500
    1000
    1500
    2000
    2500
    3000
    set size
    Conjunction Search
    0 6 12 18
    set size
    Spatial Configuration
    Search
    0 6 12 18
    set size
    Slopes
    Present: 43 msec/item
    Absent: 95 msec/item
    Slopes
    Present: 9.2 msec/item
    Absent: 26.1 msec/item
    Slopes
    Present: 1.0 msec/item
    Absent:-0.7 msec/item
    Wolfe, J. M., Palmer, E. M., & Horowitz , T. S. (2010). Reaction time distributions constrain
    models of visual search. Vision Res, 50, 1304-1311.
    Reaction time (msec)

    View full-size slide

  19. Search is guided by a limited set of attributes
    Yes Yes
    No
    Yes
    No No*
    No*
    *I say….Others would argue. (They are wrong….probably)
    Yes
    Wolfe, J. M., & Horowitz, T. S. (2004). What attributes guide the deployment of visual attention
    and how do they do it? Nature Reviews Neuroscience, 5(6), 495-501.

    View full-size slide

  20. There are
    two types of
    feature
    guidance
    Local
    differences
    create
    bottom-up
    guidance
    or “salience”
    Find the targets
    What grabbed your attention?

    View full-size slide

  21. Find the target
    Bottom-up guidance is not enough

    View full-size slide

  22. Give weight to what you want
    Find the red shallow tilted lines
    Top-down guidance

    View full-size slide

  23. A weighted sum of different
    forms of guidance tells you
    where to deploy attention
    ωcolor
    ωorientation
    ωetc
    Σ
    Wolfe, J. M. (1994). Guided Search 2.0: A revised model of visual search. Psychonomic Bulletin
    and Review, 1(2), 202-238.

    View full-size slide

  24. Guided Search for conjunctions – find a red vertical target

    View full-size slide

  25. The intersection of Red and
    Vertical is a good place to find
    Red Verticals
    Red
    Vertical
    Σ
    Top-down
    Bottom-up

    View full-size slide

  26. Why did guidance fail?

    View full-size slide

  27. It is hard to set the weight on bottom-up
    salience to zero.
    Find the green verticals
    Some Rules of Guidance

    View full-size slide

  28. Another Rule: Many simple properties do not guide
    Find the “plus” intersections (X-junctions among T-junctions)

    View full-size slide

  29. 99 msec/item
    47
    31
    14
    .3
    .6
    D
    T
    Another Rule: Many simple properties do not guide
    Wolfe, J. M., & DiMase, J. S. (2003). Do intersections serve as basic features in visual search? Perception, 32(6), 645-656.
    Reaction Time (msec)

    View full-size slide

  30. One more: Each attribute has its own rules
    Find two desaturated/pale targets

    View full-size slide

  31. Lindsey, D. T., Brown, A. M., Reijnen, E., Rich, A. N., Kuzmova, Y., & Wolfe, J. M. (2010). Color Channels, not
    Color Appearance or Color Categories, Guide Visual Search for Desaturated Color Targets. Psychol Sci, 21(9),
    1208-1214.
    Pink/peach/skin(?) is much faster
    than pale blue or green

    View full-size slide

  32. http://www.diagnosticimaging.com/breast-imaging/image-iq-42-year-old-architectural-distortion-breast-tomo
    You don’t learn new basic features.
    You learn to guide more effectively.
    Architectural
    Distortions (et al)
    are not going to
    be a basic
    features

    View full-size slide

  33. http://mentalfloss.com/article/19567/zany-history-mini-golf
    There is more to guidance than
    feature guidance
    Look for the
    golf balls
    (or “nodules”
    as we might
    call them)

    View full-size slide

  34. http://mentalfloss.com/article/19567/zany-history-mini-golf
    How did you do?
    Sure
    Probably
    How about
    these five?
    Or this one?

    View full-size slide

  35. Notice that the
    one you missed
    …is more salient
    than the two
    you found

    View full-size slide

  36. http://mentalfloss.com/article/19567/zany-history-mini-golf
    Introducing “scene guidance”
    Sure
    Probably
    Syntactic guidance (Where is it possible)?
    Semantic
    guidance
    (Where is it
    sensible)?

    View full-size slide

  37. So when a novice radiologist……
    (eye tracking image from Elizabeth Krupinski)

    View full-size slide

  38. …becomes an expert,
    a big piece of that is guidance

    View full-size slide

  39. Today, you don’t search an image.
    You search a 3D volume of images
    Trafton Drew wanted
    to know what eye movements look like in X, Y & Z

    View full-size slide

  40. Track eye
    movements
    in X & Y
    …while also
    tracking the
    slice as a
    measure of
    Z position.
    Our real question: How do radiologists
    move their eyes in a 3D volume?

    View full-size slide

  41. One way to look at the data
    Let’s color code the quadrants

    View full-size slide

  42. Time
    Depth
    Here we plot movement in Z with quadrants in
    XY color-coded for one expert radiologist

    View full-size slide

  43. But here is another expert
    Depth
    Time

    View full-size slide

  44. Drillers & Scanners
    Does it matter?
    We don’t have enough data to answer that yet.
    Drew, T., Vo, M. L.-H., Olwal, A., Jacobson, F., Seltzer, S. E., & Wolfe, J. M. (2013). Scanners and
    drillers: Characterizing expert visual search through volumetric images. Journal of Vision, 13(10).

    View full-size slide

  45. One more case

    View full-size slide

  46. There was something odd about this case

    View full-size slide

  47. Not subtle but
    missed by 20 of 24 radiologists
    Note that radiologists were
    guiding to small white nodules

    View full-size slide

  48. NOTE:
    This is not a
    criticism of
    radiologists!
    Just evidence that
    they, too, use the
    human search
    engine
    Drew, T., Vo, M. L.-H., & Wolfe, J. M. (2013).
    The Invisible Gorilla Strikes Again: Sustained
    Inattentional Blindness in Expert Observers.
    Psychological Science, 24(9), 1848–1853.

    View full-size slide

  49. Volumetric datasets may encourage search
    errors
    This is related to the next topic:
    What are you really seeing right now?
    Previous data:
    Chest X-Ray

    View full-size slide

  50. What do you really know about what
    you see?
    Try this extremely simple task

    View full-size slide

  51. Ellsworth Kelley
    Study for colors
    for a large wall
    1951
    An artistic version of Wolfe, Reinecke, & Brawn. (2006). Why don’t we see changes?
    The role of attentional bottlenecks and limited visual memory. Visual Cognition, 19(4-8), 749-780.

    View full-size slide

  52. Ellsworth Kelley
    Study for colors
    for a large wall
    1951
    Does the highlighted square change color?

    View full-size slide

  53. Ellsworth Kelley
    Study for colors
    for a large wall
    1951
    Does the highlighted square change color?
    NO

    View full-size slide

  54. Ellsworth Kelley
    Study for colors
    for a large wall
    1951
    Does the highlighted square change color?
    NO

    View full-size slide

  55. Ellsworth Kelley
    Study for colors
    for a large wall
    1951
    Does the highlighted square change color?
    NO

    View full-size slide

  56. Ellsworth Kelley
    Study for colors
    for a large wall
    1951
    Does the highlighted square change color?
    yes

    View full-size slide

  57. Ellsworth Kelley
    Study for colors
    for a large wall
    1951
    Does the highlighted square change color?
    no

    View full-size slide

  58. Claim: At any one moment you are aware of
    The gist / statistics
    (I see a bunch of
    colored squares)
    One object
    (oh, that one is red)
    And a theory
    (We will come back to
    that)

    View full-size slide

  59. Did you “see” all
    those slices?
    We think you ‘saw’
    the gist but not all
    the ‘objects’

    View full-size slide

  60. Let me illustrate with some dancing
    chickens

    View full-size slide

  61. Theory: I can see all of these dancing chickens
    And, of course, in some sense you CAN see them

    View full-size slide

  62. Don’t discount gist:
    Radiologists use The Force

    View full-size slide

  63. Experts have
    “feelings”
    (Don’t get me started
    about the book…. Actually,
    it is interesting but …)

    View full-size slide

  64. Can radiologist beat chance in a glance?
    We ran an experiment
    Look here
    Flash a mammogram for 250 msec

    View full-size slide

  65. Can radiologist beat chance in a glance?
    We ran an experiment
    Look here
    Flash a mammogram for 250 msec

    View full-size slide

  66. Can radiologist beat chance in a glance?
    We ran an experiment
    Look here
    Flash a mammogram for 250 msec

    View full-size slide

  67. Would you call back this patient?
    YES
    Call back
    No
    Don’t
    call back
    0 100
    Use a 100-pt rating scale

    View full-size slide

  68. We tested 40+ radiologists at the
    Society for Breast Imaging
    “We” =
    Michelle
    Greene
    MIT
    Karla
    Evans
    BWH
    Dianne
    Georgian-Smith
    BWH
    Robyn
    Birdwell
    BWH

    View full-size slide

  69. 0.0 0.2 0.4 0.6 0.8 1.0
    0.0
    0.2
    0.4
    0.6
    0.8
    1.0
    250
    500
    750
    1000
    2000
    Here is how we are going to plot the data
    (standard ROC curve, if you are in the trade)
    Flash duration (msec)
    False Alarm Rate
    Hit
    Rate

    View full-size slide

  70. 0.0 0.2 0.4 0.6 0.8 1.0
    0.0
    0.2
    0.4
    0.6
    0.8
    1.0
    250
    500
    750
    1000
    2000
    And here are the results
    Flash duration (msec)
    False Alarm Rate
    Hit
    Rate
    Evans, K., Georgian-Smith, D., Tambouret, R., Birdwell, R., & Wolfe, J. (2013). The gist
    of the abnormal: Above-chance medical decision making in the blink of an eye.
    Psychonomic Bulletin & Review, 1-6.

    View full-size slide

  71. A non-selective signal:
    Localization is at chance
    …. Independent of confidence

    View full-size slide

  72. 0.0 0.2 0.4 0.6 0.8 1.0
    0.0
    0.2
    0.4
    0.6
    0.8
    1.0
    250
    500
    750
    1000
    2000
    No one is suggesting that the radiologist
    should make a decision in a quarter second!
    Flash duration (msec)
    False Alarm Rate
    Hit
    Rate
    Experts doing the job

    View full-size slide

  73. But there is a non-selective
    signal to exploit
    OBJECT
    Wolfe, J. M., Vo, M. L., Evans, K.
    K., & Greene, M. R. (2011).
    Visual search in scenes involves
    selective and nonselective
    pathways. Trends Cogn Sci,
    15(2), 77-84.

    View full-size slide

  74. Finally, a
    word about
    decisions in
    search
    When do is it
    time to quit?

    View full-size slide

  75. Disease is rare in a screening population
    (low target prevalence).
    negative negative negative negative
    negative negative negative positive
    negative negative negative negative

    View full-size slide

  76. When is it time to move to the next case?
    ..and does prevalence matter?

    View full-size slide

  77. Not just a radiology problem
    Cervical cancer screening

    View full-size slide

  78. Not just a
    medical
    Problem
    Airport
    Security

    View full-size slide

  79. That is a
    knife of
    some
    sort

    View full-size slide

  80. Let’s do an experiment
    Let’s take 20 bags with guns and knives

    View full-size slide

  81. What is the effect of Prevalence?
    http://jamesthecomic.com/blog1/2010/05/03/top-10-reasons-why-your-luggage-gets-lost-or-damaged/
    Let’s take 20 bags with guns and knives
    http://www.selectism.com/news/tag/luggage/page/4/
    And put them in a stack of 40 bags
    (that’s 50% Prevalence)

    View full-size slide

  82. What is the effect of Prevalence?
    http://jamesthecomic.com/blog1/2010/05/03/top-10-reasons-why-your-luggage-gets-lost-or-damaged/
    Let’s take 20 bags with guns and knives
    http://www.selectism.com/news/tag/luggage/page/4/
    And put them in a stack of 40 bags
    50% Prevalence
    Or 1000
    bags
    2%
    Prevalence

    View full-size slide

  83. When targets are present in half of the
    bags people miss about 20% of them
    NOTE: THESE ARE VOLUNTEERS,
    NOT AIRPORT SECURITY OFFICERS

    View full-size slide

  84. When targets are present in 2% of the
    bags people miss over 40% of them!
    Same threats,
    just rarer

    View full-size slide

  85. False alarm errors go the other way.
    Which kind of
    error do you
    want to
    minimize?

    View full-size slide

  86. We did this
    in the
    breast
    cancer
    screening
    clinic of our
    hospital

    View full-size slide

  87. thank-you to
    Karla
    Evans
    Robyn
    Birdwell

    View full-size slide

  88. Basic Design
    Low Prevalence
    100 cases (50 positive, 50 negative) inserted into normal
    workflow over the course of 9 months during which
    another 9826 other cases were screened. Estimated
    prevalence 0.8%. Data are the call back decisions.
    High Prevalence
    100 cases (50 positive, 50 negative) each read by six
    radiologists (6 of 14 from the low prevalence arm).
    Prevalence is 50%. Reading the 100 cases took 3 hours.
    Data are the call back decisions and a 0-10 rating from
    negative to clearly abnormal.

    View full-size slide

  89. The Key Result
    Miss error rates are
    substantially higher at
    low prevalence
    False alarm rates are
    somewhat lower at low
    prevalence
    Evans, K. K., Birdwell, R. L., & Wolfe, J. M. (2013). If You Don’t Find It Often,
    You Often Don’t Find It: Why Some Cancers Are Missed in Breast Cancer
    Screening. . PLoS ONE 8(5): e64366

    View full-size slide

  90. But wait….there is more:
    Why doesn’t CAD work better?
    “The benefit would be greater if CADe were used more
    effectively, We have shown that radiologists only
    recognize a correct CADe prompt 30% of the time
    (Nishikawa, 2012). “
    Nishikawa, R. M., Schmidt, R. A., Linver, M. N., Edwards, A. V., Papaioannou, J.,
    & Stull, M. A. (2012). Clinically missed cancer: how effectively can radiologists
    use computer-aided detection? AJR Am J Roentgenol, 198(3), 708-716.

    View full-size slide

  91. CC views
    Right Left
    Let’s do the
    prevalence math
    Prevalence = 0.3%
    Let’s say
    Sensitivity = 90%
    Specificity = 90%
    (D’ = 2.6)
    How excited are you
    about that mark?

    View full-size slide

  92. Conclusions
    Low prevalence
    may be a significant
    source of false
    negative errors
    Low prevalence may
    be part of the reason
    why CAD is less
    effective than it
    should be.

    View full-size slide

  93. Hypothesis: Progress is
    pushing us toward more
    attentional issues
    http://www.practicevelocity.com/urgent_care/2007/
    03/urgent-care-works-for-stern-family.html http://www.sciencephoto.com/media/439759/enlarge

    View full-size slide

  94. Hypothesis about errors
    http://www.practicevelocity.com/urgent_care/2007/
    03/urgent-care-works-for-stern-family.html http://www.sciencephoto.com/media/439759/enlarge
    Can’t see it
    Can’t find it

    View full-size slide

  95. Three summary conclusions
    Visual search is guided
    Visual attention is limited
    There is an important
    decision component in
    search

    View full-size slide

  96. Melissa Vo
    Krista Ehinger
    Trafton Drew
    Sage Boettcher
    Jinxia Zhang
    Eric Chan
    Project Success
    Ray Farmer
    Project Success
    Bria Bugg
    CELEST
    Ali Cakal
    RSI
    Celeste Rousseau
    CELEST
    Beatriz Gil Gómez de Liaño
    Mardrid
    Stephanie Ding
    CELEST
    Lily Xue Gong
    Wheaton

    View full-size slide