Sofie Van Landeghem Core maintainer of spaCy Open Source Team Lead @ Explosion Belgian NLP meetup, June 2023 spacy-llm: Integrating Large Language Models into structured NLP pipelines

Sofie Van Landeghem, Belgian NLP meetup 2023 ➢ Free, open-source library ➢ Designed for production use ➢ Focus on developer productivity ➢ Free course: https://course.spacy.io https://github.com/explosion/spaCy 2 spaCy

Use-case: clinical trial results Hemodynamic Effects of Phenylephrine, Vasopressin, and Epinephrine in Children With Pulmonary Hypertension: A Pilot Study Abstract Objectives: During a pulmonary hypertensive crisis, the marked increase in pulmonary vascular resistance can result in acute right ventricular failure and death. Currently, there are no therapeutic guidelines for managing an acute crisis. This pilot study examined the hemodynamic effects of phenylephrine, arginine vasopressin, and epinephrine in pediatric patients with pulmonary hypertension. Design: In this prospective, open-label, nonrandomized pilot study, we enrolled pediatric patients previously diagnosed with pulmonary hypertensive who were scheduled electively for cardiac catheterization. Primary outcome was a change in the ratio of pulmonary-to-systemic vascular resistance. Baseline hemodynamic data were collected before and after the study drug was administered. Patients: Eleven of 15 participants were women, median age was 9.2 years (range, 1.7-14.9 yr), and median weight was 26.8 kg (range, 8.5-55.2 kg). Baseline mean pulmonary artery pressure was 49 ± 19 mm Hg, and mean indexed pulmonary vascular resistance was 10 ± 5.4 Wood units. Etiology of pulmonary hypertensive varied, and all were on systemic pulmonary hypertensive medications. Interventions: Patients 1-5 received phenylephrine 1 g/kg; patients 6-10 received arginine vasopressin 0.03 U/kg; and patients 11-15 received epinephrine 1 g/kg. μ μ Hemodynamics was measured continuously for up to 10 minutes following study drug administration. Measurements and main results: After study drug administration, the ratio of pulmonary-to-systemic vascular resistance decreased in three of five patients receiving phenylephrine, five of five patients receiving arginine vasopressin, and three of five patients receiving epinephrine. Although all three medications resulted in an increase in aortic pressure, only arginine vasopressin consistently resulted in a decrease in the ratio of systolic pulmonary artery-to-aortic pressure. Conclusions: This prospective pilot study of phenylephrine, arginine vasopressin, and epinephrine in pediatric patients with pulmonary hypertensive showed an increase in aortic pressure with all drugs although only vasopressin resulted in a consistent decrease in the ratio of pulmonary-to-systemic vascular resistance. Studies with more subjects are warranted to define optimal dosing strategies of these medications in an acute pulmonary hypertensive crisis. Stephanie L Siehr, Jeffrey A Feinstein, Weiguang Yang, Lynn F Peng, Michelle T Ogawa, Chandra Ramamoorthy. Pediatr Crit Care Med (2016) PMID: 27144689 3 Sofie Van Landeghem, Belgian NLP meetup 2023

Goal: Identify treatments and outcomes Patients: Eleven of 15 participants were women, median age was 9.2 years (range, 1.7-14.9 yr), and median weight was 26.8 kg (range, 8.5-55.2 kg). Baseline mean pulmonary artery pressure was 49 ± 19 mm Hg, and mean indexed pulmonary vascular resistance was 10 ± 5.4 Wood units. Etiology of pulmonary hypertensive varied, and all were on systemic pulmonary hypertensive medications. Interventions: Patients 1-5 received phenylephrine 1 g/kg; patients 6-10 received arginine vasopressin 0.03 μ U/kg; and patients 11-15 received epinephrine 1 g/kg. μ Hemodynamics was measured continuously for up to 10 minutes following study drug administration. Measurements and main results: After study drug administration, the ratio of pulmonary-to-systemic vascular resistance decreased in three of five patients receiving phenylephrine, five of five patients receiving arginine vasopressin, and three of five patients receiving epinephrine. Although all three medications resulted in an increase in aortic pressure, only arginine vasopressin consistently resulted in a decrease in the ratio of systolic pulmonary artery-to-aortic pressure. 4 Sofie Van Landeghem, Belgian NLP meetup 2023

spaCy pipelines ➢ A modular, pipeline approach for linguistic analysis ➢ Transforming unstructured text into structured data objects like spaCy’s Doc ORG 5 Sofie Van Landeghem, Belgian NLP meetup 2023

Pre-trained models $ python -m spacy download en_core_web_trf https://spacy.io/models nlp = spacy.load("en_core_web_trf") doc = nlp(text) for ent in doc.ents: print(ent.text, ent.label_) displacy.serve(doc, style="ent") 6 → The pre-trained English models do relatively well on generic English text → But they are not tailored to biomedical texts (drugs, patient groups etc) → We’ll have to train our own supervised NER/spancat model Sofie Van Landeghem, Belgian NLP meetup 2023

Annotate training data 7 Sofie Van Landeghem, Belgian NLP meetup 2023

Config file: capture all training settings [nlp] [nlp] lang = "en" pipeline = ["tok2vec","ner","spancat"] batch_size = 1000 [training] seed = 342 dropout = 0.1 max_steps = 20000 ... [components.spancat] factory = "spancat" spans_key = "sc" [components.spancat.model] @architectures = "spacy.SpanCategorizer.v1" [components.ner] factory = "ner" ... → A config file allows for serializability & reproducability of your NLP pipelines → spaCy has built-in architectures for NER, spancat, textcat, tagger, dependency parser, … → You can also implement and register your own models and components! https://github.com/explosion/projects/tree/v3/tutorials/rel_component 8 $ python -m spacy init config my_config.cfg --lang en --pipeline ner,spancat Sofie Van Landeghem, Belgian NLP meetup 2023

Training a supervised model $ python -m spacy train my_config.cfg --output ./my_output E # LOSS TOK2VEC LOSS NER ENTS_F ENTS_P ENTS_R SCORE --- ------ ------------ -------- ------ ------ ------ ------ 0 0 0.00 23.79 0.00 0.00 0.00 0.00 6 200 105.40 2586.38 37.21 57.14 27.59 0.37 14 400 255.98 360.81 40.00 47.62 34.48 0.40 23 600 60.01 47.55 34.04 44.44 27.59 0.34 33 800 35.52 20.49 40.00 47.62 34.48 0.40 45 1000 89.50 36.39 32.00 38.10 27.59 0.32 59 1200 47.41 22.91 43.90 75.00 31.03 0.44 ... Saves best & last trained model to the specified output directory. You can load it as an ‘nlp’ object to use for inference / further fine-tuning. nlp = spacy.load("my_output/model-best") doc = nlp(text) 9 Sofie Van Landeghem, Belgian NLP meetup 2023

spacy-llm: core concepts Integrate LLMs into production-ready, structured NLP pipelines ● Backends: ➢ External APIs, e.g. OpenAI, Cohere, Anthropic ➢ Open-source models, e.g. Dolly v2, OpenLLaMa, StableLM (via HuggingFace hub) ➢ Connect your favourite model by writing a custom backend! ● Tasks: ➢ Define prompt to send to the LLM ➢ Parse the LLM’s response and turn this into structured annotations on spaCy’s Doc objects ➢ Write a custom task definition for your specific use-case! https://github.com/explosion/spacy-llm 10 Sofie Van Landeghem, Belgian NLP meetup 2023

Zero-shot NER with spacy-llm [nlp] lang = "en" pipeline = ["llm"] [components] [components.llm] factory = "llm" [components.llm.backend] @llm_backends = "spacy.REST.v1" api = "OpenAI" [components.llm.backend.config] model: "gpt-3.5-turbo" [components.llm.task] @llm_tasks = "spacy.NER.v2" labels = "Drug,Dose" my_config.cfg 11 Assemble the pipeline and run it: from spacy_llm.util import assemble text = _read_trial(pmid=27144689) nlp = assemble(_MY_CONFIG_DIR / "my_config.cfg") doc = nlp(text) → No training data needed! Sofie Van Landeghem, Belgian NLP meetup 2023

Easily swap in other backends/tasks my_config.cfg 12 [components.llm.backend] @llm_backends = "spacy.DollyHF.v1" model = "databricks/dolly-v2-12b" [nlp] lang = "en" pipeline = ["llm"] [components] [components.llm] factory = "llm" [components.llm.backend] @llm_backends = "spacy.REST.v1" api = "OpenAI" [components.llm.backend.config] model: "gpt-3.5-turbo" [components.llm.task] @llm_tasks = "spacy.NER.v2" labels = "Drug,Dose" [components.llm.task] @llm_tasks = "spacy.NER.v2" labels = "Patient_group,Treatment" [components.llm.task] @llm_tasks = "spacy.TextCat.v2" labels = "Trial,Patent,News,Research" Sofie Van Landeghem, Belgian NLP meetup 2023

Writing a custom task INSTRUCTION = """ Summarize the trial results in a structured fashion like so: Patient group: Number of patients in the group: Treatment drug or substance: Treatment dose: Treatment frequency of administration: Treatment duration: Outcome: """ class TrialSummaryTask: def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[str]: for doc in docs: prompt = "Below this instruction, I will provide you with a clinical trial abstract. " prompt += INSTRUCTION + doc.text yield prompt def parse_responses(self, docs: Iterable[Doc], responses: Iterable[str]) -> Iterable[Doc]: ... @registry.llm_tasks("hedgy.TrialSummary.v1") def make_trial_task(): return TrialSummaryTask() 13 Sofie Van Landeghem, Belgian NLP meetup 2023

gpt-3.5-turbo (spacy-llm) Summarize the trial results in a structured fashion like so: Patient group: Number of patients in the group: Treatment drug or substance: Treatment dose: Treatment frequency of administration: Treatment duration: Outcome: Patient group: Pediatric patients with pulmonary hypertension Number of patients in the group: 15 Treatment drug or substance: Phenylephrine, arginine vasopressin, and epinephrine Treatment dose: Phenylephrine 1 μg/kg, arginine vasopressin 0.03 U/kg, epinephrine 1 μg/kg Treatment frequency of administration: Administered once Treatment duration: Hemodynamic measurements taken continuously for up to 10 minutes following drug administration Outcome: All three medications resulted in an increase in aortic pressure. Only arginine vasopressin consistently resulted in a decrease in the ratio of systolic pulmonary artery-to-aortic pressure. The ratio of pulmonary-to-systemic vascular resistance decreased in three out of five patients receiving phenylephrine, five out of five patients receiving arginine vasopressin, and three out of five patients receiving epinephrine. Studies with more patients are needed to define optimal dosing strategies for these medications in an acute pulmonary hypertensive crisis. Patients 1-5 received phenylephrine 1 g/kg; patients 6-10 received μ arginine vasopressin 0.03 U/kg; and patients 11-15 received epinephrine 1 g/kg μ (...) After study drug administration, the ratio of pulmonary-to-systemic vascular resistance decreased in three of five patients receiving phenylephrine, five of five patients receiving arginine vasopressin, and three of five patients receiving epinephrine. Although all three medications resulted in an increase in aortic pressure, only arginine vasopressin consistently resulted in a decrease in the ratio of systolic pulmonary artery-to-aortic pressure. GPT 3.5 [components.llm.task] @llm_tasks = "hedgy.TrialSummary.v1" Input text LLM output 14 Sofie Van Landeghem, Belgian NLP meetup 2023

Prompt engineering Summarize the trial results in a structured fashion. First, identify all patient groups with distinct treatments. Then, for each patient group, write down the following: Patient group: Number of patients in the group: Treatment drug or substance: Treatment dose: Treatment frequency of administration: Treatment duration: Outcome: GPT 3.5 [components.llm.task] @llm_tasks = "hedgy.TrialSummary.v2" Patient group: Phenylephrine group Number of patients in the group: 5 Treatment drug or substance: Phenylephrine Treatment dose: 1 μg/kg Treatment frequency of administration: One-time administration Treatment duration: Not specified Outcome: Ratio of pulmonary-to-systemic vascular resistance decreased in three out of five patients after study drug administration Patient group: Arginine vasopressin group Number of patients in the group: 5 Treatment drug or substance: Arginine vasopressin Treatment dose: 0.03 U/kg Treatment frequency of administration: One-time administration Treatment duration: Not specified Outcome: Ratio of pulmonary-to-systemic vascular resistance decreased in all five patients after study drug administration. Consistent decrease in the ratio of systolic pulmonary artery-to-aortic pressure noted. Patient group: Epinephrine group (...) Patients 1-5 received phenylephrine 1 g/kg μ ; patients 6-10 received arginine vasopressin 0.03 U/kg; and patients 11-15 received epinephrine 1 g/kg μ (...) After study drug administration, the ratio of pulmonary-to- systemic vascular resistance decreased in three of five patients receiving phenylephrine, five of five patients receiving arginine vasopressin, and three of five patients receiving epinephrine. LLM output Input text 15 Sofie Van Landeghem, Belgian NLP meetup 2023

Task: parse into structured fields def parse_responses(self, docs: Iterable[Doc], responses: Iterable[str]) -> Iterable[Doc]: for doc, response in zip(docs, responses): patient_groups = [] ... while ... patient_group = response[start_index:end_index].strip() patient_groups.append(patient_group) ... matcher.add("Patient_Group", [nlp.make_doc(text) for text in patient_groups]) ... matches = matcher(doc, as_spans=True) doc.ents = spacy.util.filter_spans(matches) yield doc 16 → Downstream processes can now use the LLM output in a structured way via the Doc object Sofie Van Landeghem, Belgian NLP meetup 2023

NLP is solved!

(or maybe not) 18

Reliability & robustness Patient group: Phenylephrine group Number of patients in the group: 5 Treatment drug or substance: Phenylephrine 1 μg/kg Treatment dose: As mentioned above Number of patients in the group: 15 Treatment drug or substance: Group 1: Patient 1-5 received phenylephrine 1 μg/kg Group 2: Patient 6-10 received arginine vasopressin 0.03 U/kg Group 3: Patient 11-15 received epinephrine 1 μg/kg Treatment frequency of administration “Administered once” “Single administration” “One-time dose” “One time” “Single dose” “One-time administration” “once” openai.error.RateLimitError 19 Sofie Van Landeghem, Belgian NLP meetup 2023

Performance trade-offs Accuracy Inference speed Memory usage Reliability / reproducibility Maintainability Customizability Runtime cost Annotation / implementation cost Compute power Quick prototype Interpretability Data privacy 20 Sofie Van Landeghem, Belgian NLP meetup 2023

Performance trade-offs (1) Rules/patterns Supervised ML Large Language Models 21 Sofie Van Landeghem, Belgian NLP meetup 2023

Performance trade-offs (2) Closed source LLMs Open source LLMs 22 Note: make sure to inspect the license and the terms of use! Sofie Van Landeghem, Belgian NLP meetup 2023

Ex 1: LLM-assisted annotation LLM zero-shot predictions https://prodigy.ai/features/large-language-models Manual curation Evaluation data - Measure pipeline performance Training data - Train a supervised model 23 Examples for few-shot learning - Tune the LLM Sofie Van Landeghem, Belgian NLP meetup 2023

Ex 2: Pre-process texts PII NER LLM ➢ Avoid sending sensitive data to third parties ➢ Recognize & replace Personal Identifiable Information 24 Sofie Van Landeghem, Belgian NLP meetup 2023

Ex 3: Filter input texts TextCat NER ➢ Only send texts/sentences with certain topics/entities to the LLM ➢ Avoid inducing unncessary costs ➢ Adjust prompt according to earlier classification and/or identified entities ➢ ... LLM 25 Sofie Van Landeghem, Belgian NLP meetup 2023

Ex 4: Post-process LLM responses LLM Entity linking ➢ Normalize the (free-text) LLM responses ➢ Connect to a knowledge base (e.g. through entity linking) ➢ Make the (unpredictable) LLM responses more robust for ingestion by downstream processes ➢ ... 26 Rules Sofie Van Landeghem, Belgian NLP meetup 2023

[email protected] https://twitter.com/OxyKodit https://www.linkedin.com/in/sofievanlandeghem/ https://github.com/explosion/spaCy https://github.com/explosion/spacy-llm https://explosion.ai/ Thanks