Jeremy M Wolfe Ophthalmology & Radiology. Harvard Medical School Visual Attention Lab, Brigham & Women's Hospital Dancing chickens and gorillas in the lung: If I can see so much, why do I miss so much?
Let’s begin with a few thank-yous Michelle Greene BWH to Stanford Karla Evans BWH to York, UK Dianne Georgian-Smith BWH Robyn Birdwell BWH Trafton Drew BWH
To recognize objects, we must bind features To bind features, we must attend to the object. So to find a specific object, we must search. Wolfe, J. M., & Bennett, S. C. (1997). Preattentive Object Files: Shapeless bundles of basic features. Vision Research, 37(1), 25-43.
CLAIM: Properties of the human “search engine” are the source of many error in medical image perception. Nodine, C. F., Mello-Thoms, C., Weinstein, S. P., Kundel, H. L., Conant, E. F., Heller-Savoy, R. E., et al. (2001). Blinded review of retrospectively visible unreported breast cancers: an eye-position analysis. Radiology, 221(1), 122-129. 20-30% Miss errors
What do we know about that human search engine that might shed light on visual search in tasks like radiology? http://www.adcmri.com/welcome/backendadmin/uploads/8c 24bdf59cb8b4ffa97cc7f592b39c51.gif
Visual Search is Guided 1st year 2nd year 3rd year Resident Expert Kundel, H. L., & La Follette, P. S., Jr. (1972). Visual search patterns and experience with radiological images. Radiology, 103(3), 523- 528. Kundel, H., L. . (2007). How to minimize perceptual error and maximize expertise in medical imaging. Paper presented at the Medical Imaging 2007: Image Perception, Observer Performance, and Technology Assessment. Random
Search is guided by a limited set of attributes Yes Yes No Yes No No* No* *I say….Others would argue. (They are wrong….probably) Yes Wolfe, J. M., & Horowitz, T. S. (2004). What attributes guide the deployment of visual attention and how do they do it? Nature Reviews Neuroscience, 5(6), 495-501.
A weighted sum of different forms of guidance tells you where to deploy attention ωcolor ωorientation ωetc Σ Wolfe, J. M. (1994). Guided Search 2.0: A revised model of visual search. Psychonomic Bulletin and Review, 1(2), 202-238.
99 msec/item 47 31 14 .3 .6 D T Another Rule: Many simple properties do not guide Wolfe, J. M., & DiMase, J. S. (2003). Do intersections serve as basic features in visual search? Perception, 32(6), 645-656. Reaction Time (msec)
Lindsey, D. T., Brown, A. M., Reijnen, E., Rich, A. N., Kuzmova, Y., & Wolfe, J. M. (2010). Color Channels, not Color Appearance or Color Categories, Guide Visual Search for Desaturated Color Targets. Psychol Sci, 21(9), 1208-1214. Pink/peach/skin(?) is much faster than pale blue or green
http://www.diagnosticimaging.com/breast-imaging/image-iq-42-year-old-architectural-distortion-breast-tomo You don’t learn new basic features. You learn to guide more effectively. Architectural Distortions (et al) are not going to be a basic features
http://mentalfloss.com/article/19567/zany-history-mini-golf There is more to guidance than feature guidance Look for the golf balls (or “nodules” as we might call them)
http://mentalfloss.com/article/19567/zany-history-mini-golf Introducing “scene guidance” Sure Probably Syntactic guidance (Where is it possible)? Semantic guidance (Where is it sensible)?
Track eye movements in X & Y …while also tracking the slice as a measure of Z position. Our real question: How do radiologists move their eyes in a 3D volume?
Drillers & Scanners Does it matter? We don’t have enough data to answer that yet. Drew, T., Vo, M. L.-H., Olwal, A., Jacobson, F., Seltzer, S. E., & Wolfe, J. M. (2013). Scanners and drillers: Characterizing expert visual search through volumetric images. Journal of Vision, 13(10).
NOTE: This is not a criticism of radiologists! Just evidence that they, too, use the human search engine Drew, T., Vo, M. L.-H., & Wolfe, J. M. (2013). The Invisible Gorilla Strikes Again: Sustained Inattentional Blindness in Expert Observers. Psychological Science, 24(9), 1848–1853.
Ellsworth Kelley Study for colors for a large wall 1951 An artistic version of Wolfe, Reinecke, & Brawn. (2006). Why don’t we see changes? The role of attentional bottlenecks and limited visual memory. Visual Cognition, 19(4-8), 749-780.
Claim: At any one moment you are aware of The gist / statistics (I see a bunch of colored squares) One object (oh, that one is red) And a theory (We will come back to that)
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 250 500 750 1000 2000 Here is how we are going to plot the data (standard ROC curve, if you are in the trade) Flash duration (msec) False Alarm Rate Hit Rate
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 250 500 750 1000 2000 And here are the results Flash duration (msec) False Alarm Rate Hit Rate Evans, K., Georgian-Smith, D., Tambouret, R., Birdwell, R., & Wolfe, J. (2013). The gist of the abnormal: Above-chance medical decision making in the blink of an eye. Psychonomic Bulletin & Review, 1-6.
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 250 500 750 1000 2000 No one is suggesting that the radiologist should make a decision in a quarter second! Flash duration (msec) False Alarm Rate Hit Rate Experts doing the job
But there is a non-selective signal to exploit OBJECT Wolfe, J. M., Vo, M. L., Evans, K. K., & Greene, M. R. (2011). Visual search in scenes involves selective and nonselective pathways. Trends Cogn Sci, 15(2), 77-84.
What is the effect of Prevalence? http://jamesthecomic.com/blog1/2010/05/03/top-10-reasons-why-your-luggage-gets-lost-or-damaged/ Let’s take 20 bags with guns and knives http://www.selectism.com/news/tag/luggage/page/4/ And put them in a stack of 40 bags (that’s 50% Prevalence)
What is the effect of Prevalence? http://jamesthecomic.com/blog1/2010/05/03/top-10-reasons-why-your-luggage-gets-lost-or-damaged/ Let’s take 20 bags with guns and knives http://www.selectism.com/news/tag/luggage/page/4/ And put them in a stack of 40 bags 50% Prevalence Or 1000 bags 2% Prevalence
Basic Design Low Prevalence 100 cases (50 positive, 50 negative) inserted into normal workflow over the course of 9 months during which another 9826 other cases were screened. Estimated prevalence 0.8%. Data are the call back decisions. High Prevalence 100 cases (50 positive, 50 negative) each read by six radiologists (6 of 14 from the low prevalence arm). Prevalence is 50%. Reading the 100 cases took 3 hours. Data are the call back decisions and a 0-10 rating from negative to clearly abnormal.
The Key Result Miss error rates are substantially higher at low prevalence False alarm rates are somewhat lower at low prevalence Evans, K. K., Birdwell, R. L., & Wolfe, J. M. (2013). If You Don’t Find It Often, You Often Don’t Find It: Why Some Cancers Are Missed in Breast Cancer Screening. . PLoS ONE 8(5): e64366
But wait….there is more: Why doesn’t CAD work better? “The benefit would be greater if CADe were used more effectively, We have shown that radiologists only recognize a correct CADe prompt 30% of the time (Nishikawa, 2012). “ Nishikawa, R. M., Schmidt, R. A., Linver, M. N., Edwards, A. V., Papaioannou, J., & Stull, M. A. (2012). Clinically missed cancer: how effectively can radiologists use computer-aided detection? AJR Am J Roentgenol, 198(3), 708-716.
CC views Right Left Let’s do the prevalence math Prevalence = 0.3% Let’s say Sensitivity = 90% Specificity = 90% (D’ = 2.6) How excited are you about that mark?
Conclusions Low prevalence may be a significant source of false negative errors Low prevalence may be part of the reason why CAD is less effective than it should be.
Hypothesis: Progress is pushing us toward more attentional issues http://www.practicevelocity.com/urgent_care/2007/ 03/urgent-care-works-for-stern-family.html http://www.sciencephoto.com/media/439759/enlarge
Hypothesis about errors http://www.practicevelocity.com/urgent_care/2007/ 03/urgent-care-works-for-stern-family.html http://www.sciencephoto.com/media/439759/enlarge Can’t see it Can’t find it
Melissa Vo Krista Ehinger Trafton Drew Sage Boettcher Jinxia Zhang Eric Chan Project Success Ray Farmer Project Success Bria Bugg CELEST Ali Cakal RSI Celeste Rousseau CELEST Beatriz Gil Gómez de Liaño Mardrid Stephanie Ding CELEST Lily Xue Gong Wheaton
cogitoergo
tiffany_04192
yukizenimoto
mprove
kayceesrk
kamakiri1225
ecoopnet
s1ok69oo
rabernat
dasaptaerwin
shimacos
bob3bob3
enakai00
mclloyd
brianwarren
phodgson
frogandcode
shpigford
robhawkes
matthewcrist
carmenintech
danielanewman
philnash
jlugia
nonsquared