Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Large Language Models: From Prototype to Production (PyData London keynote)

Large Language Models: From Prototype to Production (PyData London keynote)

Large Language Models (LLMs) have shown some impressive capabilities and their impact is the topic of the moment. What will the future look like? Are we going to only talk to bots? Will prompting replace programming? Or are we just hyping up unreliable parrots and burning money? In this talk, I'll present visions for NLP in the age of LLMs and a pragmatic, practical approach for how to use Large Language Models to ship more successful NLP projects from prototype to production today.

Ines Montani
PRO

June 03, 2023
Tweet

More Decks by Ines Montani

Other Decks in Technology

Transcript

  1. Ines Montani
    Explosion
    LARGE LANGUAGE
    LARGE LANGUAGE MODELS ✨ CHATGPT " ARTIFICIAL INTELLIGENCE # MACHINE LEARNING ✨
    PROTOTYPE TO PRODUCTION
    MODELS FROM
    LLAMA $ NATURAL LANGUAGE PROCESSING %
    ✨ OPEN SOURCE & PYTHON ' PROMPT ENGINEERING ⚙ ZERO-SHOT LEARNING ) GPT-4
    EVALUATION * COPILOT + GENERATIVE AI ,
    Ines Montani - Explosion

    View Slide

  2. SPACY
    SPACY.IO & @SPACY_IO ✍ SPACY.TV / GITHUB.COM/EXPLOSION/SPACY
    Open-source library for
    industrial-strength Natural
    Language Processing
    140m+
    downloads

    View Slide

  3. SPACY
    SPACY.IO & @SPACY_IO ✍ SPACY.TV / GITHUB.COM/EXPLOSION/SPACY
    Open-source library for
    industrial-strength Natural
    Language Processing
    140m+
    downloads
    ChatGPT can write spaCy code!

    View Slide

  4. PRODIGY
    Modern scriptable annotation
    tool for machine learning
    developers
    PRODIGY.AI & GITHUB.COM/EXPLOSION/PRODIGY-RECIPES
    8k+
    users
    700+
    companies

    View Slide

  5. PRODIGY
    Modern scriptable annotation
    tool for machine learning
    developers
    PRODIGY.AI & GITHUB.COM/EXPLOSION/PRODIGY-RECIPES
    8k+
    users
    700+
    companies

    View Slide

  6. THE HISTORY OF FUTURE TECHNOLOGY

    View Slide

  7. THE HISTORY OF FUTURE TECHNOLOGY
    How people in 1900
    imagined the year 2000

    View Slide

  8. THE HISTORY OF FUTURE TECHNOLOGY
    How people in 1900
    imagined the year 2000

    View Slide

  9. THE HISTORY OF FUTURE TECHNOLOGY

    View Slide

  10. THE HISTORY OF FUTURE TECHNOLOGY
    manual calculation vs. calculator

    View Slide

  11. THE HISTORY OF FUTURE TECHNOLOGY
    manual calculation vs. calculator

    View Slide

  12. THE HISTORY OF FUTURE TECHNOLOGY
    “knocker-uppers” vs. alarm clock
    manual calculation vs. calculator

    View Slide

  13. THE HISTORY OF FUTURE TECHNOLOGY

    View Slide

  14. THE HISTORY OF FUTURE TECHNOLOGY
    human assistant
    vs. calendar apps
    Calendly
    Fantastical

    View Slide

  15. THE HISTORY OF FUTURE TECHNOLOGY
    human assistant
    vs. calendar apps
    Calendly
    Fantastical
    WHAT’S NEXT?

    View Slide

  16. NLP IN THE AGE OF LLMS

    View Slide

  17. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %

    View Slide

  18. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "

    View Slide

  19. VISION #1 dialogue
    is all you need
    %
    2 LLM
    3 user
    actions or information
    natural language input

    View Slide

  20. VISION #1 dialogue
    is all you need
    %
    2 LLM
    3 user
    actions or information
    natural language input
    LLM is the system
    and needs to manage
    the whole interaction

    View Slide

  21. VISION #2 prompting
    is all you need
    "
    2 LLM
    4 text % prompt
    5 system
    3 user
    6 structured data

    View Slide

  22. VISION #2 prompting
    is all you need
    "
    2 LLM
    4 text % prompt
    5 system
    3 user
    LLM replaces the
    specific ML model
    6 structured data

    View Slide

  23. VISION #3 modern
    practical NLP
    -
    7 developer 8 code
    2 LLM 9 training data
    5 system
    3 user
    6 structured data
    ⚙ ML system

    View Slide

  24. VISION #3 modern
    practical NLP
    -
    7 developer 8 code
    2 LLM 9 training data
    5 system
    3 user
    6 structured data
    ⚙ ML system
    LLM helps with
    building the pipeline

    View Slide

  25. VISION #3 modern
    practical NLP
    -
    7 developer 8 code
    2 LLM 9 training data
    5 system
    3 user
    6 structured data
    ⚙ ML system
    LLM helps with
    building the pipeline

    View Slide

  26. VISION #3 modern
    practical NLP
    -
    7 developer 8 code
    2 LLM 9 training data
    5 system
    3 user
    6 structured data
    ⚙ ML system
    LLM helps with
    building the pipeline

    View Slide

  27. COMPANY
    COMPANY
    MONEY
    INVESTOR
    “Hooli raises $5m to
    revolutionize search,
    led by ACME Ventures”
    5923214
    1681056
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    Database

    View Slide

  28. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA

    View Slide

  29. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition

    View Slide

  30. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation

    View Slide

  31. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    custom database lookup

    View Slide

  32. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    custom database lookup
    currency normalization

    View Slide

  33. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    custom database lookup
    currency normalization
    entity relation extraction

    View Slide

  34. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    custom database lookup
    currency normalization
    entity relation extraction
    6 structured data
    2 LLM
    7 developer
    quick prototype

    View Slide

  35. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    custom database lookup
    currency normalization
    entity relation extraction
    6 structured data
    2 LLM
    7 developer
    quick prototype

    fast to develop,
    slow to run, hard
    to improve

    View Slide

  36. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -

    View Slide

  37. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data

    View Slide

  38. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data
    humans in
    the loop

    View Slide

  39. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data fast prototyping
    humans in
    the loop

    View Slide

  40. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data fast prototyping
    humans in
    the loop
    powered by
    open source

    View Slide

  41. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data fast prototyping
    humans in
    the loop
    powered by
    open source
    conversational
    and graphical
    interfaces

    View Slide

  42. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data fast prototyping
    humans in
    the loop
    powered by
    open source
    robust
    evaluation
    conversational
    and graphical
    interfaces

    View Slide

  43. A CASE FOR LLM PRAGMATISM
    EXPLOSION.AI/BLOG/AGAINST-LLM-MAXIMALISM
    NOOO YOU CAN'T JUST MIX
    UP ALL THE STEPS OF YOUR TASK
    AND ASK AN LLM TO DO IT ALL.
    HOW WILL YOU EVER MAKE A RELIABLE
    AND EXTENSIBLE SYSTEM THAT WAY?
    HAHA LLM GO BRRR

    View Slide

  44. A CASE FOR LLM PRAGMATISM
    EXPLOSION.AI/BLOG/AGAINST-LLM-MAXIMALISM
    NOOO YOU CAN'T JUST MIX
    UP ALL THE STEPS OF YOUR TASK
    AND ASK AN LLM TO DO IT ALL.
    HOW WILL YOU EVER MAKE A RELIABLE
    AND EXTENSIBLE SYSTEM THAT WAY?
    HAHA LLM GO BRRR
    avoid coupling prediction tasks
    to arbitrary business logic

    View Slide

  45. A CASE FOR LLM PRAGMATISM
    EXPLOSION.AI/BLOG/AGAINST-LLM-MAXIMALISM
    NOOO YOU CAN'T JUST MIX
    UP ALL THE STEPS OF YOUR TASK
    AND ASK AN LLM TO DO IT ALL.
    HOW WILL YOU EVER MAKE A RELIABLE
    AND EXTENSIBLE SYSTEM THAT WAY?
    HAHA LLM GO BRRR
    avoid coupling prediction tasks
    to arbitrary business logic
    design modular solutions

    View Slide

  46. A CASE FOR LLM PRAGMATISM
    EXPLOSION.AI/BLOG/AGAINST-LLM-MAXIMALISM
    NOOO YOU CAN'T JUST MIX
    UP ALL THE STEPS OF YOUR TASK
    AND ASK AN LLM TO DO IT ALL.
    HOW WILL YOU EVER MAKE A RELIABLE
    AND EXTENSIBLE SYSTEM THAT WAY?
    HAHA LLM GO BRRR
    avoid coupling prediction tasks
    to arbitrary business logic
    design modular solutions
    prototype modules with LLMs

    View Slide

  47. A CASE FOR LLM PRAGMATISM
    EXPLOSION.AI/BLOG/AGAINST-LLM-MAXIMALISM
    NOOO YOU CAN'T JUST MIX
    UP ALL THE STEPS OF YOUR TASK
    AND ASK AN LLM TO DO IT ALL.
    HOW WILL YOU EVER MAKE A RELIABLE
    AND EXTENSIBLE SYSTEM THAT WAY?
    HAHA LLM GO BRRR
    avoid coupling prediction tasks
    to arbitrary business logic
    design modular solutions
    prototype modules with LLMs
    evaluate alternatives

    View Slide

  48. TRADE-OFFS
    performance on
    the bar exam
    kentlaw.iit.edu

    View Slide

  49. TRADE-OFFS
    performance on
    the bar exam
    kentlaw.iit.edu
    OpenAI API
    latency
    promptlayer.com

    View Slide

  50. TRADE-OFFS
    Supervised 1 LLM 2
    accuracy words/s accuracy words/s
    Textcat on SST2
    (Stanford Sentiment Treebank)
    0.9 4019 0.9 <100
    Textcat on Banking77
    (intent recognition)
    0.9 3234 0.7 <100
    NER on AnEm
    (anatomical entity mentions)
    0.7 5146 0.1 <100
    1. RoBERTa-base with spaCy, 2. text-davinci-003 zero-shot
    ongoing experiments
    comparing LLMS to
    task-specific models
    performance on
    the bar exam
    kentlaw.iit.edu
    OpenAI API
    latency
    promptlayer.com

    View Slide

  51. LLM-POWERED NLP IN PRACTICE
    LLM-powered collaborative data development environment
    7

    View Slide

  52. LLM-POWERED NLP IN PRACTICE
    LLM-powered collaborative data development environment
    7
    Assign labeling tasks to LLMs
    "

    View Slide

  53. LLM-POWERED NLP IN PRACTICE
    LLM-powered collaborative data development environment
    7
    Assign labeling tasks to LLMs
    "
    Review label decisions, correct errors
    ;

    View Slide

  54. LLM-POWERED NLP IN PRACTICE
    LLM-powered collaborative data development environment
    7
    Assign labeling tasks to LLMs
    "
    Review label decisions, correct errors
    ;
    Tune prompts and compare LLMs empirically
    6

    View Slide

  55. LLM-POWERED NLP IN PRACTICE
    LLM-powered collaborative data development environment
    7
    Assign labeling tasks to LLMs
    "
    Review label decisions, correct errors
    ;
    Tune prompts and compare LLMs empirically
    6
    Build data sets to train and evaluate e icient, production-ready pipelines
    +

    View Slide

  56. 8
    PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS

    View Slide

  57. 8
    correct
    mistakes
    PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS

    View Slide

  58. PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    correct
    mistakes

    View Slide

  59. add correct
    answer to prompt
    to tune it
    PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    correct
    mistakes

    View Slide

  60. PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS

    View Slide

  61. PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    query LLM and
    parse response

    View Slide

  62. PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    query LLM and
    parse response
    tune prompt
    if needed

    View Slide

  63. GITHUB.COM/EXPLOSION/SPACY-LLM
    TOWARDS STRUCTURED DATA
    Prompt Template
    2 LLM
    London is bigger
    than Berlin
    LOCATION:
    London, Berlin
    LOCATION

    View Slide

  64. GITHUB.COM/EXPLOSION/SPACY-LLM
    TOWARDS STRUCTURED DATA
    Prompt Template
    2 LLM
    London is bigger
    than Berlin
    LOCATION:
    London, Berlin
    LOCATION

    View Slide

  65. SPECIFIC
    Task-specific models
    powered by LLMS
    IS BETTER.

    View Slide

  66. SMALLER & FASTER
    Task-specific models
    powered by LLMS
    IS BETTER.

    View Slide

  67. PRIVATE
    Task-specific models
    powered by LLMS
    IS BETTER.

    View Slide

  68. BETTER
    Task-specific models
    powered by LLMS
    IS BETTER.

    View Slide

  69. THANK YOU!
    - Explosion – explosion.ai
    < spaCy – spacy.io
    ✨ Prodigy – prodigy.ai
    = Twitter – @_inesmontani
    > Mastodon – @[email protected]
    ? LinkedIn

    View Slide