Large Language Models: From Prototype to Production (PyData London keynote)

Large Language Models (LLMs) have shown some impressive capabilities and their impact is the topic of the moment. What will the future look like? Are we going to only talk to bots? Will prompting replace programming? Or are we just hyping up unreliable parrots and burning money? In this talk, I'll present visions for NLP in the age of LLMs and a pragmatic, practical approach for how to use Large Language Models to ship more successful NLP projects from prototype to production today.

Transcript

1. Ines Montani
   Explosion
   LARGE LANGUAGE
   LARGE LANGUAGE MODELS ✨ CHATGPT " ARTIFICIAL INTELLIGENCE # MACHINE LEARNING ✨
   PROTOTYPE TO PRODUCTION
   MODELS FROM
   LLAMA $ NATURAL LANGUAGE PROCESSING %
   ✨ OPEN SOURCE & PYTHON ' PROMPT ENGINEERING ⚙ ZERO-SHOT LEARNING ) GPT-4
   EVALUATION * COPILOT + GENERATIVE AI ,
   Ines Montani - Explosion
   

   View Slide

2. SPACY
   SPACY.IO & @SPACY_IO ✍ SPACY.TV / GITHUB.COM/EXPLOSION/SPACY
   Open-source library for
   industrial-strength Natural
   Language Processing
   140m+
   downloads
   

   View Slide

3. SPACY
   SPACY.IO & @SPACY_IO ✍ SPACY.TV / GITHUB.COM/EXPLOSION/SPACY
   Open-source library for
   industrial-strength Natural
   Language Processing
   140m+
   downloads
   ChatGPT can write spaCy code!
   

   View Slide

4. PRODIGY
   Modern scriptable annotation
   tool for machine learning
   developers
   PRODIGY.AI & GITHUB.COM/EXPLOSION/PRODIGY-RECIPES
   8k+
   users
   700+
   companies
   

   View Slide

5. PRODIGY
   Modern scriptable annotation
   tool for machine learning
   developers
   PRODIGY.AI & GITHUB.COM/EXPLOSION/PRODIGY-RECIPES
   8k+
   users
   700+
   companies
   

   View Slide

6. THE HISTORY OF FUTURE TECHNOLOGY
   

   View Slide

7. THE HISTORY OF FUTURE TECHNOLOGY
   How people in 1900
   imagined the year 2000
   

   View Slide

8. THE HISTORY OF FUTURE TECHNOLOGY
   How people in 1900
   imagined the year 2000
   

   View Slide

9. THE HISTORY OF FUTURE TECHNOLOGY
   

   View Slide

10. THE HISTORY OF FUTURE TECHNOLOGY
    manual calculation vs. calculator
    

    View Slide

11. THE HISTORY OF FUTURE TECHNOLOGY
    manual calculation vs. calculator
    

    View Slide

12. THE HISTORY OF FUTURE TECHNOLOGY
    “knocker-uppers” vs. alarm clock
    manual calculation vs. calculator
    

    View Slide

13. THE HISTORY OF FUTURE TECHNOLOGY
    

    View Slide

14. THE HISTORY OF FUTURE TECHNOLOGY
    human assistant
    vs. calendar apps
    Calendly
    Fantastical
    

    View Slide

15. THE HISTORY OF FUTURE TECHNOLOGY
    human assistant
    vs. calendar apps
    Calendly
    Fantastical
    WHAT’S NEXT?
    

    View Slide

16. NLP IN THE AGE OF LLMS
    

    View Slide

17. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    

    View Slide

18. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    

    View Slide

19. VISION #1 dialogue
    is all you need
    %
    2 LLM
    3 user
    actions or information
    natural language input
    

    View Slide

20. VISION #1 dialogue
    is all you need
    %
    2 LLM
    3 user
    actions or information
    natural language input
    LLM is the system
    and needs to manage
    the whole interaction
    

    View Slide

21. VISION #2 prompting
    is all you need
    "
    2 LLM
    4 text % prompt
    5 system
    3 user
    6 structured data
    

    View Slide

22. VISION #2 prompting
    is all you need
    "
    2 LLM
    4 text % prompt
    5 system
    3 user
    LLM replaces the
    specific ML model
    6 structured data
    

    View Slide

23. VISION #3 modern
    practical NLP
    -
    7 developer 8 code
    2 LLM 9 training data
    5 system
    3 user
    6 structured data
    ⚙ ML system
    

    View Slide

24. VISION #3 modern
    practical NLP
    -
    7 developer 8 code
    2 LLM 9 training data
    5 system
    3 user
    6 structured data
    ⚙ ML system
    LLM helps with
    building the pipeline
    

    View Slide

25. VISION #3 modern
    practical NLP
    -
    7 developer 8 code
    2 LLM 9 training data
    5 system
    3 user
    6 structured data
    ⚙ ML system
    LLM helps with
    building the pipeline
    

    View Slide

26. VISION #3 modern
    practical NLP
    -
    7 developer 8 code
    2 LLM 9 training data
    5 system
    3 user
    6 structured data
    ⚙ ML system
    LLM helps with
    building the pipeline
    

    View Slide

27. COMPANY
    COMPANY
    MONEY
    INVESTOR
    “Hooli raises $5m to
    revolutionize search,
    led by ACME Ventures”
    5923214
    1681056
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    Database
    

    View Slide

28. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    

    View Slide

29. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    

    View Slide

30. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    

    View Slide

31. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    custom database lookup
    

    View Slide

32. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    custom database lookup
    currency normalization
    

    View Slide

33. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    custom database lookup
    currency normalization
    entity relation extraction
    

    View Slide

34. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    custom database lookup
    currency normalization
    entity relation extraction
    6 structured data
    2 LLM
    7 developer
    quick prototype
    

    View Slide

35. Database
    CLASSIC NLP PROBLEM: EXTRACT STRUCTURED DATA
    named entity recognition
    entity disambiguation
    custom database lookup
    currency normalization
    entity relation extraction
    6 structured data
    2 LLM
    7 developer
    quick prototype
    ⚡
    fast to develop,
    slow to run, hard
    to improve
    

    View Slide

36. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    

    View Slide

37. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data
    

    View Slide

38. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data
    humans in
    the loop
    

    View Slide

39. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data fast prototyping
    humans in
    the loop
    

    View Slide

40. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data fast prototyping
    humans in
    the loop
    powered by
    open source
    

    View Slide

41. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data fast prototyping
    humans in
    the loop
    powered by
    open source
    conversational
    and graphical
    interfaces
    

    View Slide

42. NLP IN THE AGE OF LLMS
    SQL
    is all you need
    dialogue
    is all you need
    0 %
    lots of humans
    is all you need
    prompting
    is all you need
    1 "
    modern
    practical
    NLP
    -
    structured data fast prototyping
    humans in
    the loop
    powered by
    open source
    robust
    evaluation
    conversational
    and graphical
    interfaces
    

    View Slide

43. A CASE FOR LLM PRAGMATISM
    EXPLOSION.AI/BLOG/AGAINST-LLM-MAXIMALISM
    NOOO YOU CAN'T JUST MIX
    UP ALL THE STEPS OF YOUR TASK
    AND ASK AN LLM TO DO IT ALL.
    HOW WILL YOU EVER MAKE A RELIABLE
    AND EXTENSIBLE SYSTEM THAT WAY?
    HAHA LLM GO BRRR
    

    View Slide

44. A CASE FOR LLM PRAGMATISM
    EXPLOSION.AI/BLOG/AGAINST-LLM-MAXIMALISM
    NOOO YOU CAN'T JUST MIX
    UP ALL THE STEPS OF YOUR TASK
    AND ASK AN LLM TO DO IT ALL.
    HOW WILL YOU EVER MAKE A RELIABLE
    AND EXTENSIBLE SYSTEM THAT WAY?
    HAHA LLM GO BRRR
    avoid coupling prediction tasks
    to arbitrary business logic
    

    View Slide

45. A CASE FOR LLM PRAGMATISM
    EXPLOSION.AI/BLOG/AGAINST-LLM-MAXIMALISM
    NOOO YOU CAN'T JUST MIX
    UP ALL THE STEPS OF YOUR TASK
    AND ASK AN LLM TO DO IT ALL.
    HOW WILL YOU EVER MAKE A RELIABLE
    AND EXTENSIBLE SYSTEM THAT WAY?
    HAHA LLM GO BRRR
    avoid coupling prediction tasks
    to arbitrary business logic
    design modular solutions
    

    View Slide

46. A CASE FOR LLM PRAGMATISM
    EXPLOSION.AI/BLOG/AGAINST-LLM-MAXIMALISM
    NOOO YOU CAN'T JUST MIX
    UP ALL THE STEPS OF YOUR TASK
    AND ASK AN LLM TO DO IT ALL.
    HOW WILL YOU EVER MAKE A RELIABLE
    AND EXTENSIBLE SYSTEM THAT WAY?
    HAHA LLM GO BRRR
    avoid coupling prediction tasks
    to arbitrary business logic
    design modular solutions
    prototype modules with LLMs
    

    View Slide

47. A CASE FOR LLM PRAGMATISM
    EXPLOSION.AI/BLOG/AGAINST-LLM-MAXIMALISM
    NOOO YOU CAN'T JUST MIX
    UP ALL THE STEPS OF YOUR TASK
    AND ASK AN LLM TO DO IT ALL.
    HOW WILL YOU EVER MAKE A RELIABLE
    AND EXTENSIBLE SYSTEM THAT WAY?
    HAHA LLM GO BRRR
    avoid coupling prediction tasks
    to arbitrary business logic
    design modular solutions
    prototype modules with LLMs
    evaluate alternatives
    

    View Slide

48. TRADE-OFFS
    performance on
    the bar exam
    kentlaw.iit.edu
    

    View Slide

49. TRADE-OFFS
    performance on
    the bar exam
    kentlaw.iit.edu
    OpenAI API
    latency
    promptlayer.com
    

    View Slide

50. TRADE-OFFS
    Supervised 1 LLM 2
    accuracy words/s accuracy words/s
    Textcat on SST2
    (Stanford Sentiment Treebank)
    0.9 4019 0.9 <100
    Textcat on Banking77
    (intent recognition)
    0.9 3234 0.7 <100
    NER on AnEm
    (anatomical entity mentions)
    0.7 5146 0.1 <100
    1. RoBERTa-base with spaCy, 2. text-davinci-003 zero-shot
    ongoing experiments
    comparing LLMS to
    task-specific models
    performance on
    the bar exam
    kentlaw.iit.edu
    OpenAI API
    latency
    promptlayer.com
    

    View Slide

51. LLM-POWERED NLP IN PRACTICE
    LLM-powered collaborative data development environment
    7
    

    View Slide

52. LLM-POWERED NLP IN PRACTICE
    LLM-powered collaborative data development environment
    7
    Assign labeling tasks to LLMs
    "
    

    View Slide

53. LLM-POWERED NLP IN PRACTICE
    LLM-powered collaborative data development environment
    7
    Assign labeling tasks to LLMs
    "
    Review label decisions, correct errors
    ;
    

    View Slide

54. LLM-POWERED NLP IN PRACTICE
    LLM-powered collaborative data development environment
    7
    Assign labeling tasks to LLMs
    "
    Review label decisions, correct errors
    ;
    Tune prompts and compare LLMs empirically
    6
    

    View Slide

55. LLM-POWERED NLP IN PRACTICE
    LLM-powered collaborative data development environment
    7
    Assign labeling tasks to LLMs
    "
    Review label decisions, correct errors
    ;
    Tune prompts and compare LLMs empirically
    6
    Build data sets to train and evaluate e icient, production-ready pipelines
    +
    

    View Slide

56. 8
    PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    

    View Slide

57. 8
    correct
    mistakes
    PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    

    View Slide

58. PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    correct
    mistakes
    

    View Slide

59. add correct
    answer to prompt
    to tune it
    PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    correct
    mistakes
    

    View Slide

60. PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    

    View Slide

61. PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    query LLM and
    parse response
    

    View Slide

62. PRODIGY.AI/FEATURES/LARGE-LANGUAGE-MODELS
    query LLM and
    parse response
    tune prompt
    if needed
    

    View Slide

63. GITHUB.COM/EXPLOSION/SPACY-LLM
    TOWARDS STRUCTURED DATA
    Prompt Template
    2 LLM
    London is bigger
    than Berlin
    LOCATION:
    London, Berlin
    LOCATION
    

    View Slide

64. GITHUB.COM/EXPLOSION/SPACY-LLM
    TOWARDS STRUCTURED DATA
    Prompt Template
    2 LLM
    London is bigger
    than Berlin
    LOCATION:
    London, Berlin
    LOCATION
    

    View Slide

65. SPECIFIC
    Task-specific models
    powered by LLMS
    IS BETTER.
    

    View Slide

66. SMALLER & FASTER
    Task-specific models
    powered by LLMS
    IS BETTER.
    

    View Slide

67. PRIVATE
    Task-specific models
    powered by LLMS
    IS BETTER.
    

    View Slide

68. BETTER
    Task-specific models
    powered by LLMS
    IS BETTER.
    

    View Slide

69. THANK YOU!
    - Explosion – explosion.ai
    < spaCy – spacy.io
    ✨ Prodigy – prodigy.ai
    = Twitter – @_inesmontani
    > Mastodon – @ines@sigmoid.social
    ? LinkedIn
    

    View Slide