Никитин.pdf

Transcript

1. Evgeniy Nikitin Training medical neural networks with the real-world data
2. About us Celsus is analyzing medical images in order to 1) automate part of the radiology workflow to support faster decisions 2) reduce the risk of the errors Our team • Backend/frontend • Sales & Marketing • Data Team • ML team (11 ML engineers)
3. Are medical datasets really that bad?
4. Example - Mammography datasets © 2020, ООО “Медицинские скрининг системы” 4 Dataset 1 Classes: • Mass (malignant & benign) • Calcifications (malignant & benign) 1878 patients from 1 medical organization 1 annotator per image, non-verified Dataset 2 Classes: • Malignant mass • Benign mass • Malignant calcification • Benign calcification • Skin thickening 200 patients from 1 medical organization 3 annotators per image, verified Dataset 3 Patient-level annotations (benign/malignant) 5720 patients from ~30 medical organizations Unknown annotators
5. Example - Mammography datasets © 2020, ООО “Медицинские скрининг системы” 5 Dataset 1 Classes: • Mass (malignant & benign) • Calcifications (malignant & benign) 1878 patients from 1 medical organization 1 annotator per image, non-verified Dataset 2 Classes: • Malignant mass • Benign mass • Malignant calcification • Benign calcification • Skin thickening 200 patients from 1 medical organization 3 annotators per image, verified Dataset 3 Patient-level annotations (benign/malignant) 5720 patients from ~30 medical organizations Unknown annotators
6. Example - Mammography datasets © 2020, ООО “Медицинские скрининг системы” 6 Dataset 1 Classes: • Mass (malignant & benign) • Calcifications (malignant & benign) 1878 patients from 1 medical organization 1 annotator per image, non-verified Dataset 2 Classes: • Malignant mass • Benign mass • Malignant calcification • Benign calcification • Skin thickening 200 patients from 1 medical organization 3 annotators per image, verified Dataset 3 Patient-level annotations (benign/malignant) 5720 patients from ~30 medical organizations Unknown annotators
7. Example - Mammography datasets © 2020, ООО “Медицинские скрининг системы” 7 Dataset 1 Classes: • Mass (malignant & benign) • Calcifications (malignant & benign) 1878 patients from 1 medical organization 1 annotator per image, non-verified Dataset 2 Classes: • Malignant mass • Benign mass • Malignant calcification • Benign calcification • Skin thickening 200 patients from 1 medical organization 3 annotators per image, verified Dataset 3 Patient-level annotations (benign/malignant) 5720 patients from ~30 medical organizations Unknown annotators
8. Terminology © 2020, ООО “Медицинские скрининг системы” • X - input image or a series of images • y - target variable ◦ classification - normal/pathological ◦ sorting - risk score ◦ detection - bounding boxes ◦ segmentation - per-pixel masks
9. © 2020, ООО “Медицинские скрининг системы”
10. Problem sources 1) X a) Different scanners b) Different image acquisition protocols c) Human errors 2) Noisy y a) Inter-observer annotation variability b) Intra-observer annotation variability c) Unreliable data sources 3) Different label sets of y a) Different classes in the different datasets
11. Problem sources #2 4) Weak y a) Image-, volume-, patient-level annotations are cheaper and easier to acquire 5) Scarce y a) Annotation of medical images is expensive b) Some conditions are naturally rare
12. Types of solutions • Prevention • Data-centric solutions • Architectural modifications • Training process modifications • ...
13. Xs
14. Variations in X
15. Our solution Universal image preprocessor • Artifact removal • Contrast/brightness normalization • Correct color inversion & flip • Crop by the region of the interest • Unifying digitized film scans and digital scans • ….
16. Preprocessed images
17. Even better?
18. Out-of-Distribution in Production? • Classic CV methods • Unsupervised feature-based methods • Image quality branch • Confidence branch
19. Training “image quality” branch
20. Confidence branch https://arxiv.org/pdf/1802.04865.pdf
21. Different label spaces
22. Ambiguous “background” category © 2020, ООО “Медицинские скрининг системы” 22 Dataset 1 Bounding boxes: • Malignant mass • Benign mass • Malignant calcification • Benign calcification • Skin thickening Dataset 2 Bounding boxes: • Malignant mass • Benign mass We will penalize network for proposing non-annotated regions in Dataset 2 that contain objects!
23. Architectural modifications
24. Augmenting label spaces with pseudo labeling https://arxiv.org/pdf/2008.06614.pdf
25. Auxiliary classifiers for class decoupling Mass Malignant Benign p 1-p
26. Noisy labels
27. © 2020, ООО “Медицинские скрининг системы” 27
28. © 2020, ООО “Медицинские скрининг системы” 28
29. © 2020, ООО “Медицинские скрининг системы” 29
30. Reducing annotator variability • Prevention ◦ Creating and updating labeling guides ◦ Exams for annotators • Post factum ◦ Automatic resolution of minor conflicts ◦ Additional expert opinion on major conflicts
31. git merge for annotations © 2020, ООО “Медицинские скрининг системы” 31
32. Dealing with noisy annotations during training • Dropping or re-weighting noisy examples ◦ Usually doesn’t work well in the medical domain • Label smoothing ◦ Soft labels ◦ Smoothing for mask boundaries • Online noise correction ◦ Bounding box correction ◦ Mask refinement
33. https://arxiv.org/pdf/2003.01285.pdf
34. Weak labels
35. Multiple Instance Learning (MIL) https://arxiv.org/pdf/1911.05650.pdf
36. MIL upgrades • Other pooling techniques (LSEPooling) • Attention mechanisms • Aggregating bags of local instances
37. Training meta-models 1. Training detector/segmentator with existing small annotated datasets 2. Generating features for large weakly labeled dataset 3. Training simple meta-model to predict final target
38. Scarce labels
39. Unsupervised pre-training https://arxiv.org/pdf/2002.05709.pdf
40. Unsupervised pre-training in medicine
41. Domain adaptation & Transfer Learning
42. How to avoid going crazy?
43. Gold standard datasets It is absolutely necessary to create and update gold standard datasets: • Consensus labeling by at least 3 annotators • Preferably clinically verified • It must include all conditions that your system is supposed to detect But remember - every time you test your experiment on this dataset - you are one step closer to overfitting
44. Main takeaways • You need to stay strong if you want to work with the medical data
45. Main takeaways • You need to stay strong if you want to work with the medical data • It is both challenging and rewarding
46. Main takeaways • You need to stay strong if you want to work with the medical data • It is both challenging and rewarding • Read literature, but stay creative - every problem is unique
47. Main takeaways • You need to stay strong if you want to work with the medical data • It is both challenging and rewarding • Read literature, but stay creative - every problem is unique • Invest time and money into creating gold standard dataset for pre-release testing of all your crazy ideas