Upgrade to Pro — share decks privately, control downloads, hide ads and more …

spaCy meets Transformers

spaCy meets Transformers

Huge transformer models like BERT, GPT-2 and XLNet have set a new standard for accuracy on almost every Natural Language Processing leaderboard. However, these models are very new, and most of the software ecosystem surrounding them is oriented towards the many opportunities for further research that they provide. In this talk, I’ll describe how you can now use these models in spaCy, a popular library for putting Natural Language Processing to work on real problems. I’ll also discuss the many opportunities that new transfer learning technologies can offer production NLP, regardless of which specific software packages you choose to get the job done.

Matthew Honnibal

October 12, 2019
Tweet

More Decks by Matthew Honnibal

Other Decks in Technology

Transcript

  1. spaCy meets
    Transformers
    Matthew Honnibal
    Explosion

    View full-size slide

  2. Matthew Honnibal
    CO-FOUNDER
    PhD in Computer Science in 2009.
    10 years publishing research on state-of-the-
    art natural language understanding systems.
    Left academia in 2014 to develop spaCy.
    Ines Montani
    CO-FOUNDER
    Programmer and front-end developer with
    degree in media science and linguistics.
    Has been working on spaCy since its first
    release. Lead developer of Prodigy.

    View full-size slide

  3. 100k+ users worldwide
    15k stars on GitHub
    400 contributors
    60+ extension packages
    https://spacy.io

    View full-size slide

  4. https://prodi.gy

    View full-size slide

  5. 2500+ users, including
    250+ companies
    1200+ forum members
    https://prodi.gy

    View full-size slide

  6. ELMo
    ULMFiT
    BERT

    View full-size slide

  7. ELMo
    ULMFiT
    BERT

    View full-size slide

  8. ELMo
    ULMFiT
    BERT

    View full-size slide

  9. Tokenization alignment

    View full-size slide

  10. spaCy’s NLP pipeline

    View full-size slide

  11. Processing pipeline

    View full-size slide

  12. Processing pipeline
    with shared representations

    View full-size slide

  13. Processing pipeline
    without shared representations

    View full-size slide

  14. Modular
    architecture
    • Functions should be small
    and self-contained
    • Avoid state and side-
    effects
    • Lots of systems from
    fewer parts
    Speed and
    accuracy
    • Small functions make you
    repeat work
    • Without state, models
    lose information
    • ML models aren’t really
    interchangeable anyway

    View full-size slide

  15. What you can do
    with transformers

    View full-size slide

  16. Transformers:
    Pros
    • Easy network design
    • Great accuracy
    • Need few annotated
    examples
    Transformers:
    Cons
    • Slow / expensive
    • Need large batches
    • Bleeding edge

    View full-size slide

  17. github.com/explosion/spacy-transformers

    View full-size slide

  18. • pip install spacy-transformers
    • Supports textcat, aligned tokenization, custom models
    • Coming soon: NER, tagging, dependency parsing
    • Coming soon: RPC for the transformer components
    • Coming soon: Transformers support in Prodigy
    Conclusion

    View full-size slide

  19. Thank you!
    Explosion
    explosion.ai
    Follow us on Twitter
    @honnibal
    @explosion_ai

    View full-size slide