Introducing elasticsearch-fixtures

These are the slides for a presentation I gave at the Python User Group Singapore September 2019 meetup.

It shows how to use a tiny little library that I wrote to create test-fixtures for a Elasticsearch index: https://github.com/TheArtling/elasticsearch-fixtures

Transcript

1. INTRODUCING ELASTICSEARCH-FIXTURES MARTIN BROCHHAUS
2. LET'S STAY IN TOUCH ▸ CTO of The Artling (theartling.com) ▸ PyCon SG Organizer (pycon.sg) ▸ Twitter: @mbrochh
3. WHAT IS ELASTICSEARCH? ▸ Open Source ▸ Distributed database for all types of data ▸ Data is stored as JSON documents ▸ Simple REST API ▸ Inverted Index, very fast full-text search
4. SHOULD YOU USE IT? ▸ At The Artling we used PostgreSQL with Django for 4 years ▸ As the DB grew, search became slower and slower ▸ We added PostgreSQL fulltext search ▸ We tried Elasticsearch with elasticsearch-django ▸ Search speed improved up to 10x ▸ [show example "fairy of the forest"]
5. HOW TO INSTALL ELASTICSEARCH? ▸ brew tap elastic/tap ▸ brew install elastic/tap/elasticsearch-full ▸ brew services start elasticsearch
6. HOW TO INSTALL KIBANA? ▸ brew tap elastic/tap ▸ brew install elastic/tap/kibana-full ▸ brew services start kibana ▸ browse to http://localhost:5601
7. INTRODUCING: ELASTICSEARCH-FIXTURES ▸ You can simply install it via pip: ▸ mkvirtualenv es-fixtures ▸ pip install elasticsearch-fixtures ▸ pip install pytest ipython
8. TRY IT OUT IN IPDB! ▸ ipython ▸ from elasticsearch_fixtures.es_mixer import ESMixer ▸ es_mixer = ESMixer('http://localhost:9200') ▸ es_mixer.blend('testindex', name='Alice') ▸ es_mixer.blend('testindex', name='Bob')
9. LOOK AT IT IN KIBANA ▸ browse to http://localhost:5601 ▸ add the new indexpattern for 'testindex' ▸ look at the index in the Discover tab ▸ wipe the index: ▸ es_mixer.wipe_index('testindex') ▸ refresh Discover tab
10. LET'S SAY WE WANT TO TEST THE FOLLOWING PROGRAM
11. OUR TEST COULD LOOK LIKE THIS
12. RUN THE TEST ▸ py.test ▸ it should work ▸ run it again ▸ it should not work ▸ problem: the data remains inside the testindex between tests
13. WIPE INDEX AT THE BEGINNING OF EACH TEST SESSION ▸ create conftest.py

14. LET'S HAVE A LOOK AT THE CODE ▸ README is here: https://github.com/ TheArtling/elasticsearch-fixtures ▸ Code is here: https://github.com/ TheArtling/elasticsearch-fixtures/ blob/master/elasticsearch_fixtures/ es_mixer.py

15. CAVEATS ▸ This solution is far from perfect ▸ It is good enough for our narrow usecase ▸ If you run "py.test -n 4" (parallel) your tests can randomly fail because you can't be sure in what order your tests will run, so in different runs, different data will be visible for each test ▸ All fixtures that you create in your test cases will still be present in all following test cases -- you have to come up with very smart test data so that searches in one test don't affect the results that you expect in another test (i.e. search for unique strings like SearchResultTest0001-Object1)
16. FOR THOSE WHO DON'T KNOW DJANGO MIXER ▸ It allows you to create test fixtures like so: ▸ from mixer.backend.django import Mixer
 mixer = Mixer()
 mixer.blend('products.Product')
 mixer.blend('products.Product', title="Foobar")
17. TIP: UNIQUE IDS WITH DJANGO MIXER ▸ Our Django codebase has hundreds of tests ▸ Most tests create Product fixtures via "mixer.blend('products.Product')" ▸ We use an in-memory SQLite3 DB, so each test-case has it's own empty database ▸ PROBLEM: Elasticsearch does not have an in-memory mode, so test fixtures pollute the DB and bleed into other tests ▸ PROBLEM: When running in parallel, up to four tests will simultaneously try to create Product fixtures and they all try to use ID=1, but only one such ID can exist in Elasticsearch at a time
18. TIP: UNIQUE IDS WITH DJANGO MIXER
19. DEMO PACKAGE ON GITHUB ▸ The code shown in this presentation can be found here: https:// github.com/mbrochh/introducing- elasticsearch-fixtures
20. THANK YOU! ▸ CTO of The Artling (theartling.com) ▸ PyCon SG Organizer (pycon.sg) ▸ Twitter: @mbrochh