Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introducing elasticsearch-fixtures

Introducing elasticsearch-fixtures

These are the slides for a presentation I gave at the Python User Group Singapore September 2019 meetup.

It shows how to use a tiny little library that I wrote to create test-fixtures for a Elasticsearch index: https://github.com/TheArtling/elasticsearch-fixtures

Martin Brochhaus

September 24, 2019

More Decks by Martin Brochhaus

Other Decks in Programming


  1. LET'S STAY IN TOUCH ▸ CTO of The Artling (theartling.com)

    ▸ PyCon SG Organizer (pycon.sg) ▸ Twitter: @mbrochh
  2. WHAT IS ELASTICSEARCH? ▸ Open Source ▸ Distributed database for

    all types of data ▸ Data is stored as JSON documents ▸ Simple REST API ▸ Inverted Index, very fast full-text search
  3. SHOULD YOU USE IT? ▸ At The Artling we used

    PostgreSQL with Django for 4 years ▸ As the DB grew, search became slower and slower ▸ We added PostgreSQL fulltext search ▸ We tried Elasticsearch with elasticsearch-django ▸ Search speed improved up to 10x ▸ [show example "fairy of the forest"]
  4. HOW TO INSTALL ELASTICSEARCH? ▸ brew tap elastic/tap ▸ brew

    install elastic/tap/elasticsearch-full ▸ brew services start elasticsearch
  5. HOW TO INSTALL KIBANA? ▸ brew tap elastic/tap ▸ brew

    install elastic/tap/kibana-full ▸ brew services start kibana ▸ browse to http://localhost:5601
  6. INTRODUCING: ELASTICSEARCH-FIXTURES ▸ You can simply install it via pip:

    ▸ mkvirtualenv es-fixtures ▸ pip install elasticsearch-fixtures ▸ pip install pytest ipython
  7. TRY IT OUT IN IPDB! ▸ ipython ▸ from elasticsearch_fixtures.es_mixer

    import ESMixer ▸ es_mixer = ESMixer('http://localhost:9200') ▸ es_mixer.blend('testindex', name='Alice') ▸ es_mixer.blend('testindex', name='Bob')
  8. LOOK AT IT IN KIBANA ▸ browse to http://localhost:5601 ▸

    add the new indexpattern for 'testindex' ▸ look at the index in the Discover tab ▸ wipe the index: ▸ es_mixer.wipe_index('testindex') ▸ refresh Discover tab
  9. RUN THE TEST ▸ py.test ▸ it should work ▸

    run it again ▸ it should not work ▸ problem: the data remains inside the testindex between tests

    here: https://github.com/ TheArtling/elasticsearch-fixtures ▸ Code is here: https://github.com/ TheArtling/elasticsearch-fixtures/ blob/master/elasticsearch_fixtures/ es_mixer.py

  11. CAVEATS ▸ This solution is far from perfect ▸ It

    is good enough for our narrow usecase ▸ If you run "py.test -n 4" (parallel) your tests can randomly fail because you can't be sure in what order your tests will run, so in different runs, different data will be visible for each test ▸ All fixtures that you create in your test cases will still be present in all following test cases -- you have to come up with very smart test data so that searches in one test don't affect the results that you expect in another test (i.e. search for unique strings like SearchResultTest0001-Object1)

    you to create test fixtures like so: ▸ from mixer.backend.django import Mixer
 mixer = Mixer()
 mixer.blend('products.Product', title="Foobar")
  13. TIP: UNIQUE IDS WITH DJANGO MIXER ▸ Our Django codebase

    has hundreds of tests ▸ Most tests create Product fixtures via "mixer.blend('products.Product')" ▸ We use an in-memory SQLite3 DB, so each test-case has it's own empty database ▸ PROBLEM: Elasticsearch does not have an in-memory mode, so test fixtures pollute the DB and bleed into other tests ▸ PROBLEM: When running in parallel, up to four tests will simultaneously try to create Product fixtures and they all try to use ID=1, but only one such ID can exist in Elasticsearch at a time
  14. DEMO PACKAGE ON GITHUB ▸ The code shown in this

    presentation can be found here: https:// github.com/mbrochh/introducing- elasticsearch-fixtures
  15. THANK YOU! ▸ CTO of The Artling (theartling.com) ▸ PyCon

    SG Organizer (pycon.sg) ▸ Twitter: @mbrochh