Pro Yearly is on sale from $80 to $50! »

Introducing elasticsearch-fixtures

Introducing elasticsearch-fixtures

These are the slides for a presentation I gave at the Python User Group Singapore September 2019 meetup.

It shows how to use a tiny little library that I wrote to create test-fixtures for a Elasticsearch index: https://github.com/TheArtling/elasticsearch-fixtures

3949eadbf8213b05e415415b2c17eaee?s=128

Martin Brochhaus

September 24, 2019
Tweet

Transcript

  1. INTRODUCING ELASTICSEARCH-FIXTURES MARTIN BROCHHAUS

  2. LET'S STAY IN TOUCH ▸ CTO of The Artling (theartling.com)

    ▸ PyCon SG Organizer (pycon.sg) ▸ Twitter: @mbrochh
  3. WHAT IS ELASTICSEARCH? ▸ Open Source ▸ Distributed database for

    all types of data ▸ Data is stored as JSON documents ▸ Simple REST API ▸ Inverted Index, very fast full-text search
  4. SHOULD YOU USE IT? ▸ At The Artling we used

    PostgreSQL with Django for 4 years ▸ As the DB grew, search became slower and slower ▸ We added PostgreSQL fulltext search ▸ We tried Elasticsearch with elasticsearch-django ▸ Search speed improved up to 10x ▸ [show example "fairy of the forest"]
  5. HOW TO INSTALL ELASTICSEARCH? ▸ brew tap elastic/tap ▸ brew

    install elastic/tap/elasticsearch-full ▸ brew services start elasticsearch
  6. HOW TO INSTALL KIBANA? ▸ brew tap elastic/tap ▸ brew

    install elastic/tap/kibana-full ▸ brew services start kibana ▸ browse to http://localhost:5601
  7. INTRODUCING: ELASTICSEARCH-FIXTURES ▸ You can simply install it via pip:

    ▸ mkvirtualenv es-fixtures ▸ pip install elasticsearch-fixtures ▸ pip install pytest ipython
  8. TRY IT OUT IN IPDB! ▸ ipython ▸ from elasticsearch_fixtures.es_mixer

    import ESMixer ▸ es_mixer = ESMixer('http://localhost:9200') ▸ es_mixer.blend('testindex', name='Alice') ▸ es_mixer.blend('testindex', name='Bob')
  9. LOOK AT IT IN KIBANA ▸ browse to http://localhost:5601 ▸

    add the new indexpattern for 'testindex' ▸ look at the index in the Discover tab ▸ wipe the index: ▸ es_mixer.wipe_index('testindex') ▸ refresh Discover tab
  10. LET'S SAY WE WANT TO TEST THE FOLLOWING PROGRAM

  11. OUR TEST COULD LOOK LIKE THIS

  12. RUN THE TEST ▸ py.test ▸ it should work ▸

    run it again ▸ it should not work ▸ problem: the data remains inside the testindex between tests
  13. WIPE INDEX AT THE BEGINNING OF EACH TEST SESSION ▸

    create conftest.py

  14. LET'S HAVE A LOOK AT THE CODE ▸ README is

    here: https://github.com/ TheArtling/elasticsearch-fixtures ▸ Code is here: https://github.com/ TheArtling/elasticsearch-fixtures/ blob/master/elasticsearch_fixtures/ es_mixer.py

  15. CAVEATS ▸ This solution is far from perfect ▸ It

    is good enough for our narrow usecase ▸ If you run "py.test -n 4" (parallel) your tests can randomly fail because you can't be sure in what order your tests will run, so in different runs, different data will be visible for each test ▸ All fixtures that you create in your test cases will still be present in all following test cases -- you have to come up with very smart test data so that searches in one test don't affect the results that you expect in another test (i.e. search for unique strings like SearchResultTest0001-Object1)
  16. FOR THOSE WHO DON'T KNOW DJANGO MIXER ▸ It allows

    you to create test fixtures like so: ▸ from mixer.backend.django import Mixer
 mixer = Mixer()
 mixer.blend('products.Product')
 mixer.blend('products.Product', title="Foobar")
  17. TIP: UNIQUE IDS WITH DJANGO MIXER ▸ Our Django codebase

    has hundreds of tests ▸ Most tests create Product fixtures via "mixer.blend('products.Product')" ▸ We use an in-memory SQLite3 DB, so each test-case has it's own empty database ▸ PROBLEM: Elasticsearch does not have an in-memory mode, so test fixtures pollute the DB and bleed into other tests ▸ PROBLEM: When running in parallel, up to four tests will simultaneously try to create Product fixtures and they all try to use ID=1, but only one such ID can exist in Elasticsearch at a time
  18. TIP: UNIQUE IDS WITH DJANGO MIXER

  19. DEMO PACKAGE ON GITHUB ▸ The code shown in this

    presentation can be found here: https:// github.com/mbrochh/introducing- elasticsearch-fixtures
  20. THANK YOU! ▸ CTO of The Artling (theartling.com) ▸ PyCon

    SG Organizer (pycon.sg) ▸ Twitter: @mbrochh