Using Text Analytics for Health to Extract Insights from COVID Papers Dataset

In this presentation, I show how different Azure resources can be used to extract insights from dataset of COVID scientific papers. The process includes the following:
- Extracting entities using Text Analytics for Health on Azure Batch
- Storing semi-structured data in Cosmos DB and doing SQL Queries
- Using SQL Queries in Cosmos DB Notebooks to gain further insights into data
- Adding Power BI dashboard for no-code exploration of data

Transcript

1. Using Text Analytics for Health to Extract Insights from CORD COVID Papers Dataset Dmitry Soshnikov, Cloud Developer Advocate, Microsoft http://soshnikov.com Victoria Soshnikova, Student, Phystech Lyceum vickie@soshnikov.com
2. CORD Papers Dataset https://allenai.org/data/cord-19 CORD-19 Dataset Contains over 400,000 scholarly articles about COVID-19 and the coronavirus family of viruses for use by the global research community 200,000 articles with full text https://www.kaggle.com/allen-institute-for-ai/ CORD-19-research-challenge
3. Main Idea: Use NLP tools to extract semi-structured data from papers, to enable semantically rich queries over the paper corpus. CORD Corpus Extracted JSON Cosmos DB Database Power BI Dashboard SQL Queries Azure Semantic Search NER Relations Text Analytics for Health
4. Text Analytics for Health (Preview)  Currently in Preview  Gated service, need to apply for usage (apply at https://aka.ms/csgate)  Should not be implemented or deployed in any production use.  Can be used through Web API or Container Service  Supports:  Named Entity Recognition (NER)  Relation Extraction  Entity Linking (Ontology Mapping)  Negation Detection
5. Entity Extraction + Entity Linking, Negation Detection
6. Relation Extraction
7. Using Text Analytics for Health Pip Install the Azure TextAnalytics SDK: pip install azure.ai.textanalytics==5.1.0b5 from azure.core.credentials import AzureKeyCredential from azure.ai.textanalytics import TextAnalyticsClient client = TextAnalyticsClient(endpoint=endpoint, credential=AzureKeyCredential(key), api_version="v3.1-preview.3") Create the client: documents = ["I have not been administered any aspirin, just 300 mg or favipiravir daily."] poller = client.begin_analyze_healthcare_entities(documents) result = poller.result() Do the call:
8. Analysis Result I have not been administered any aspirin, just 300 mg or favipiravir daily. HealthcareEntity(text=300 mg, category=Dosage, subcategory=None, length=6, offset=47, confidence_score=1.0, data_sources=None, related_entities={HealthcareEntity(text=favipiravir, category=MedicationName, subcategory=None, length=11, offset=57, confidence_score=1.0, data_sources=[HealthcareEntityDataSource(entity_id=C1138226, name=UMLS), HealthcareEntityDataSource(entity_id=J05AX27, name=ATC), HealthcareEntityDataSource(entity_id=DB12466, name=DRUGBANK), HealthcareEntityDataSource(entity_id=398131, name=MEDCIN), HealthcareEntityDataSource(entity_id=C462182, name=MSH), HealthcareEntityDataSource(entity_id=C81605, name=NCI), HealthcareEntityDataSource(entity_id=EW5GL2X7E0, name=NCI_FDA)], related_entities={}): 'DosageOfMedication'}) aspirin (C0004057) [MedicationName] 300 mg [Dosage] --DosageOfMedication--> favipiravir (C1138226) [MedicationName] favipiravir (C1138226) [MedicationName] daily [Frequency] --FrequencyOfMedication--> favipiravir (C1138226) [MedicationName]
9. Analyzing CORD Abstracts • All abstracts contained in CSV metadata file • Split 400k papers into chunks of 500 • Id, Title, Journal, Authors, Publication Date • Shuffle by date in order to get representative sample in each chunk • Enrich each json file with text analytics data • Entities, Relations • Parallel processing using VM pool / Azure Batch
10. Results of Text Analytics Processing { "gh690dai": { "id": "gh690dai", "title": "Beef and Pork Marketing Margins and Price Spreads during COVID-19", "authors": "Lusk, Jayson L.; Tonsor, Glynn T.; Schulz, Lee L.", "journal": "Appl Econ Perspect Policy", "abstract": "...", "publish_time": "2020-10-02", "entities": [ { "offset": 0, "length": 16, "text": "COVID-19-related", "category": "Diagnosis", "confidenceScore": 0.79, "isNegated": false },..] "relations": [ { "relationType": "TimeOfTreatment", "bidirectional": false, "source": { "uri": "#/documents/0/entities/15", "text": "previous year", "category": "Time", "isNegated": false, "offset": 704 }, "target": { "uri": "#/documents/0/entities/13", "text": "beef", "category": "TreatmentName", "isNegated": false, "offset": 642 }}]}, …
11. Storing Semi-Structured Data into Cosmos DB Cosmos DB – NoSQL universal solution Querying semi-structured data with SQL-like language Paper Paper Entity Entity Relation Collection … …
12. Cosmos DB SQL Queries Get mentioned dosages of a particular medication and papers they are mentioned in SELECT p.title, r.source.text FROM papers p JOIN r IN p.relations WHERE r.relationType='DosageOfMedication’ AND CONTAINS(r.target.text,'hydro')
13. Exploration: PowerBI
14. Exploration: PowerBI
15. Further Exploration: Jupyter in Cosmos DB SQL in Cosmos DB is somehow limited Good strategy: make query in Cosmos DB, export to Pandas Dataframe, final exploration in Python Jupyter support is built into Cosmos DB Makes exporting query results to DataFrame easy! %%sql --database CORD --container Papers --output meds SELECT e.text, e.isNegated, p.title, p.publish_time, ARRAY (SELECT VALUE l.id FROM l IN e.links WHERE l.dataSource='UMLS')[0] AS umls_id FROM papers p JOIN e IN p.entities WHERE e.category = 'MedicationName'
16. COVID Publication Activity 2020
17. Unique Medications by UMLS ID { 'C0020336': 'hydroxychloroquine', 'C0008269': 'chloroquine', 'C1609165': 'Tocilizumab', 'C4726677': 'remdesivir', 'C0052796': 'azithromycin', 'C0674432': 'lopinavir', 'C0292818': 'ritonavir', 'C0042866': 'vitamin D', 'C0011777': 'dexamethasone', 'C0019134': 'Heparin', 'C1138226': 'favipiravir', 'C0021641': 'insulin', 'C0030106': 'ozone', 'C0025815': 'methylprednisolone', 'C0023810': 'lipopolysaccharide’ } Top 15 by count
18. How Medication Strategies Change
19. How Medication Strategies Change
20. Term relations
21. Term Relations
22. Terms Co-occurence Treatment Medicine
23. Conclusions Text Mining for Medical Texts can be very valuable resource for gaining insights into large text corpus. ❶ ❷ A Range of Microsoft Technologies can be used to effectively make this a reality: • Text Analytics for Health to do NER and ontology mapping • Azure Batch / VM Pool to perform large-scale processing • Cosmos DB to store and query semistructured data • Power BI to explore the data interactively to gain insights • Cosmos DB Jupyter Notebooks to do deep dive into the data w/Python
24. © Copyright Microsoft Corporation. All rights reserved. For further collaboration: