INTRODUCING ELASTICSEARCH-FIXTURES MARTIN BROCHHAUS 

LET'S STAY IN TOUCH ▸ CTO of The Artling (theartling.com) ▸ PyCon SG Organizer (pycon.sg) ▸ Twitter: @mbrochh 

WHAT IS ELASTICSEARCH? ▸ Open Source ▸ Distributed database for all types of data ▸ Data is stored as JSON documents ▸ Simple REST API ▸ Inverted Index, very fast full-text search 

SHOULD YOU USE IT? ▸ At The Artling we used PostgreSQL with Django for 4 years ▸ As the DB grew, search became slower and slower ▸ We added PostgreSQL fulltext search ▸ We tried Elasticsearch with elasticsearch-django ▸ Search speed improved up to 10x ▸ [show example "fairy of the forest"] 

HOW TO INSTALL ELASTICSEARCH? ▸ brew tap elastic/tap ▸ brew install elastic/tap/elasticsearch-full ▸ brew services start elasticsearch 

HOW TO INSTALL KIBANA? ▸ brew tap elastic/tap ▸ brew install elastic/tap/kibana-full ▸ brew services start kibana ▸ browse to http://localhost:5601 

INTRODUCING: ELASTICSEARCH-FIXTURES ▸ You can simply install it via pip: ▸ mkvirtualenv es-fixtures ▸ pip install elasticsearch-fixtures ▸ pip install pytest ipython 

TRY IT OUT IN IPDB! ▸ ipython ▸ from elasticsearch_fixtures.es_mixer import ESMixer ▸ es_mixer = ESMixer('http://localhost:9200') ▸ es_mixer.blend('testindex', name='Alice') ▸ es_mixer.blend('testindex', name='Bob') 

LOOK AT IT IN KIBANA ▸ browse to http://localhost:5601 ▸ add the new indexpattern for 'testindex' ▸ look at the index in the Discover tab ▸ wipe the index: ▸ es_mixer.wipe_index('testindex') ▸ refresh Discover tab 

LET'S SAY WE WANT TO TEST THE FOLLOWING PROGRAM 

OUR TEST COULD LOOK LIKE THIS 

RUN THE TEST ▸ py.test ▸ it should work ▸ run it again ▸ it should not work ▸ problem: the data remains inside the testindex between tests 

WIPE INDEX AT THE BEGINNING OF EACH TEST SESSION ▸ create conftest.py
 

LET'S HAVE A LOOK AT THE CODE ▸ README is here: https://github.com/ TheArtling/elasticsearch-fixtures ▸ Code is here: https://github.com/ TheArtling/elasticsearch-fixtures/ blob/master/elasticsearch_fixtures/ es_mixer.py
 

CAVEATS ▸ This solution is far from perfect ▸ It is good enough for our narrow usecase ▸ If you run "py.test -n 4" (parallel) your tests can randomly fail because you can't be sure in what order your tests will run, so in different runs, different data will be visible for each test ▸ All fixtures that you create in your test cases will still be present in all following test cases -- you have to come up with very smart test data so that searches in one test don't affect the results that you expect in another test (i.e. search for unique strings like SearchResultTest0001-Object1) 

FOR THOSE WHO DON'T KNOW DJANGO MIXER ▸ It allows you to create test fixtures like so: ▸ from mixer.backend.django import Mixer
 mixer = Mixer()
 mixer.blend('products.Product')
 mixer.blend('products.Product', title="Foobar") 

TIP: UNIQUE IDS WITH DJANGO MIXER ▸ Our Django codebase has hundreds of tests ▸ Most tests create Product fixtures via "mixer.blend('products.Product')" ▸ We use an in-memory SQLite3 DB, so each test-case has it's own empty database ▸ PROBLEM: Elasticsearch does not have an in-memory mode, so test fixtures pollute the DB and bleed into other tests ▸ PROBLEM: When running in parallel, up to four tests will simultaneously try to create Product fixtures and they all try to use ID=1, but only one such ID can exist in Elasticsearch at a time 

TIP: UNIQUE IDS WITH DJANGO MIXER 

DEMO PACKAGE ON GITHUB ▸ The code shown in this presentation can be found here: https:// github.com/mbrochh/introducing- elasticsearch-fixtures 

THANK YOU! ▸ CTO of The Artling (theartling.com) ▸ PyCon SG Organizer (pycon.sg) ▸ Twitter: @mbrochh