Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Integration and Functional Testing with LiveServerTestCase, Selenium and More.

7617093fdae20c6fe4b0f7fe954a8aad?s=47 Julien Phalip
September 05, 2013

Integration and Functional Testing with LiveServerTestCase, Selenium and More.

Talk given at DjangoCon US 2013. Chicago, IL.
The video: http://www.youtube.com/watch?v=BSYg9-sxSjM

7617093fdae20c6fe4b0f7fe954a8aad?s=128

Julien Phalip

September 05, 2013
Tweet

Transcript

  1. ~ Julien Phalip Integration and Functional Testing with LiveServerTestCase, Selenium

    and More. September 5th, 2013 DJANGOCON US 2013
  2. CONNECTED PERSONAL OBJECTS 5/2012 About me... ‣ Worked with Django

    since 2007 (version 0.96). ‣ Django core committer since 2011.
  3. CONNECTED PERSONAL OBJECTS 5/2012 www.nurun.com

  4. CONNECTED PERSONAL OBJECTS 5/2012 Definitions (subject to debate)

  5. CONNECTED PERSONAL OBJECTS 5/2012 Definitions (subject to debate) ‣ Unit

    testing Ensure that small parts work in isolation.
  6. CONNECTED PERSONAL OBJECTS 5/2012 Definitions (subject to debate) ‣ Unit

    testing Ensure that small parts work in isolation. ‣ Integration testing Ensure that those parts work well together.
  7. CONNECTED PERSONAL OBJECTS 5/2012 Definitions (subject to debate) ‣ Unit

    testing Ensure that small parts work in isolation. ‣ Integration testing Ensure that those parts work well together. ‣ Functional testing Ensure that the application works, from a user’s perspective.
  8. CONNECTED PERSONAL OBJECTS 5/2012 Testing in Django (the traditional way)

  9. CONNECTED PERSONAL OBJECTS 5/2012 Testing in Django (the traditional way)

    from django.test import TestCase class MyTests(TestCase): def test_hello(self): r = self.client.get('/hello') self.assertTrue('Hello' in r.content)
  10. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase

  11. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase DJANGO THREAD

  12. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase Application code DJANGO THREAD

  13. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase TestCase Application code DJANGO THREAD

  14. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase Database TestCase Application code DJANGO

    THREAD
  15. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database

    TestCase Application code DJANGO THREAD
  16. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database

    TestCase Application code MockRequest DJANGO THREAD
  17. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database

    TestCase Application code MockRequest DJANGO THREAD
  18. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database

    TestCase Application code MockRequest HttpResponse DJANGO THREAD
  19. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database

    TestCase Application code MockRequest HttpResponse DJANGO THREAD
  20. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database

    TestCase Application code MockRequest HttpResponse DJANGO THREAD
  21. CONNECTED PERSONAL OBJECTS 5/2012 Pretty good but incomplete...

  22. CONNECTED PERSONAL OBJECTS 5/2012 Pretty good but incomplete... ‣ Does

    not exercise the full http protocol.
  23. CONNECTED PERSONAL OBJECTS 5/2012 Pretty good but incomplete... ‣ Does

    not exercise the full http protocol. ‣ Does not allow to test the user interface.
  24. And then came LiveServerTestCase...

  25. And then came LiveServerTestCase...

  26. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase

  27. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase LiveServerTestCase DJANGO THREAD #1

  28. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase LiveServerTestCase Application code DJANGO THREAD

    #2 DJANGO THREAD #1 Live http server
  29. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client LiveServerTestCase Application

    code DJANGO THREAD #2 DJANGO THREAD #1 Live http server
  30. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase

    Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server
  31. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase

    Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server
  32. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase

    http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server
  33. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase

    http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
  34. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase

    http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
  35. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase

    http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
  36. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase

    http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
  37. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase

    http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
  38. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase

    http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
  39. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase Database LiveServerTestCase http python-requests Application

    code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
  40. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase Database LiveServerTestCase http Application code

    DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
  41. CONNECTED PERSONAL OBJECTS 5/2012 Some important details ‣ Live server

    listens to localhost:8081 by default ‣ Address can be overridden: ./manage.py test --liveserver=localhost:8082 ./manage.py test --liveserver=localhost:9000-9200 ‣ Address accessible from the test’s code: self.live_server_url
  42. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests

  43. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests ‣ Testing REST

    APIs:
  44. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests ‣ Testing REST

    APIs: import requests class MyTests(LiveServerTestCase): def test_api_user(self): r = requests.get( self.live_server_url + '/api/user/1/') self.assertEqual(r.status_code, 200) self.assertEqual(r.json, {'username': 'john.doe', 'email': 'john@doe.com'})
  45. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests

  46. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests ‣ Testing OAUTH

    workflows:
  47. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests ‣ Testing OAUTH

    workflows: import requests from requests_oauthlib import OAuth2 from oauthlib.oauth2 import BackendApplicationClient class MyTests(LiveServerTestCase): def test_oauth(self): backend = BackendApplicationClient('mybackend') oauth2 = OAuth2(client=backend, token=TOKEN) r = requests.get( self.live_server_url + '/oauth', auth=oauth2) self.assertEqual(r.status_code, 200) self.assertTrue('Hello' in r.content)
  48. Browser automators

  49. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators?

  50. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer

    to real Web user environment.
  51. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer

    to real Web user environment. ‣ Media & static files (CSS/images/etc.) implicitly get loaded.
  52. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer

    to real Web user environment. ‣ Media & static files (CSS/images/etc.) implicitly get loaded. ‣ Javascript code & Ajax calls implicitly get executed.
  53. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer

    to real Web user environment. ‣ Media & static files (CSS/images/etc.) implicitly get loaded. ‣ Javascript code & Ajax calls implicitly get executed. ‣ User interactions can be tested.
  54. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer

    to real Web user environment. ‣ Media & static files (CSS/images/etc.) implicitly get loaded. ‣ Javascript code & Ajax calls implicitly get executed. ‣ User interactions can be tested. ‣ Browser compatibility can be tested.
  55. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer

    to real Web user environment. ‣ Media & static files (CSS/images/etc.) implicitly get loaded. ‣ Javascript code & Ajax calls implicitly get executed. ‣ User interactions can be tested. ‣ Browser compatibility can be tested. ‣ It’s fun!
  56. CONNECTED PERSONAL OBJECTS 5/2012 Selenium

  57. CONNECTED PERSONAL OBJECTS 5/2012 Selenium ‣ API to become W3C

    standard (currently in draft)
  58. CONNECTED PERSONAL OBJECTS 5/2012 Selenium ‣ API to become W3C

    standard (currently in draft) ‣ Client libraries in Java, C#, .NET, Ruby, PHP, Perl, Javascript and Python.
  59. CONNECTED PERSONAL OBJECTS 5/2012 Selenium ‣ API to become W3C

    standard (currently in draft) ‣ Client libraries in Java, C#, .NET, Ruby, PHP, Perl, Javascript and Python. ‣ Works across multiple browsers (Firefox, Chrome, IE, Safari, Opera)
  60. CONNECTED PERSONAL OBJECTS 5/2012 Selenium ‣ API to become W3C

    standard (currently in draft) ‣ Client libraries in Java, C#, .NET, Ruby, PHP, Perl, Javascript and Python. ‣ Works across multiple browsers (Firefox, Chrome, IE, Safari, Opera) ‣ Easy to install: pip install selenium
  61. CONNECTED PERSONAL OBJECTS 5/2012 from django.test import LiveServerTestCase from selenium.webdriver.firefox.webdriver

    import \ WebDriver class MyTests(LiveServerTestCase):
  62. CONNECTED PERSONAL OBJECTS 5/2012 from django.test import LiveServerTestCase from selenium.webdriver.firefox.webdriver

    import \ WebDriver class MyTests(LiveServerTestCase): def setUp(self): self.driver = WebDriver() super(MyTests, self).setUp() def tearDown(self): self.driver.quit() super(MyTests, self).tearDown()
  63. CONNECTED PERSONAL OBJECTS 5/2012 from django.test import LiveServerTestCase from selenium.webdriver.firefox.webdriver

    import \ WebDriver class MyTests(LiveServerTestCase): def setUp(self): self.driver = WebDriver() super(MyTests, self).setUp() def tearDown(self): self.driver.quit() super(MyTests, self).tearDown()
  64. CONNECTED PERSONAL OBJECTS 5/2012 from django.test import LiveServerTestCase from selenium.webdriver.firefox.webdriver

    import \ WebDriver class MyTests(LiveServerTestCase): ...
  65. CONNECTED PERSONAL OBJECTS 5/2012 from django.test import LiveServerTestCase from selenium.webdriver.firefox.webdriver

    import \ WebDriver class MyTests(LiveServerTestCase): def test_hello(self): self.driver.get( self.live_server_url + '/hello') e = self.driver.find_element_by_id('btn') e.click() ... ...
  66. CONNECTED PERSONAL OBJECTS 5/2012 DEMO Django Core Test Suite

  67. Tips and goodies

  68. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle

  69. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ https://github.com/bfirsh/needle

  70. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ https://github.com/bfirsh/needle

    ‣ Based on Selenium.
  71. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ https://github.com/bfirsh/needle

    ‣ Based on Selenium. ‣ Tests visuals by comparing screenshots.
  72. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ Test

    the looks of an entire page...
  73. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ Test

    the looks of an entire page...
  74. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ Or

    just parts of it.
  75. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ Or

    just parts of it.
  76. CONNECTED PERSONAL OBJECTS 5/2012 from needle.cases import NeedleTestCase from from

    selenium.webdriver import firefox class MyTests(NeedleTestCase, LiveServerTestCase): driver_command_executor = firefox.ExtensionConnection( "127.0.0.1", firefox.FirefoxProfile()) def test_homepage_menu(self): self.driver.get(self.live_server_url) self.assertScreenshot('ul.menu', 'menu') Testing visuals with Needle
  77. CONNECTED PERSONAL OBJECTS 5/2012 ====================================================================== FAIL: test_homepage_menu (myapp.tests.MyTests) ---------------------------------------------------------------------- Traceback

    (most recent call last): File "myapp/tests.py", line 115, in test_homepage_menu self.assertScreenshot('ul.menu', 'menu') File "/.virtualenvs/myproject/src/needle/needle/cases.py", line 99, in assertScreenshot pass File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/ python2.6/contextlib.py", line 23, in __exit__ self.gen.next() File "/.virtualenvs/myproject/src/needle/needle/cases.py", line 160, in compareScreenshot % (filename, distance)) AssertionError: The saved screenshot for 'menu' did not match the screenshot captured (by a distance of 26.17) Testing visuals with Needle
  78. CONNECTED PERSONAL OBJECTS 5/2012 Test coverage

  79. CONNECTED PERSONAL OBJECTS 5/2012 Test coverage ‣ Yes, it does

    work too.
  80. CONNECTED PERSONAL OBJECTS 5/2012 Test coverage ‣ Yes, it does

    work too. $ coverage run manage.py test \ myapp.SeleniumTests $ coverage report
  81. CONNECTED PERSONAL OBJECTS 5/2012 Headless mode for continuous integration

  82. CONNECTED PERSONAL OBJECTS 5/2012 Headless mode for continuous integration ‣

    Install a browser and fonts: $ sudo apt-get install -y firefox xfonts-100dpi xfonts-75dpi xfonts- scalable xfonts-cyrillic
  83. CONNECTED PERSONAL OBJECTS 5/2012 Headless mode for continuous integration ‣

    Install a browser and fonts: $ sudo apt-get install -y firefox xfonts-100dpi xfonts-75dpi xfonts- scalable xfonts-cyrillic ‣ Install Xvfb: $ sudo apt-get install -y xvfb
  84. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Either with the Xvfb command:

    $ Xvfb -ac :99 2>/dev/null & $ DISPLAY=:99 ./manage.py test Headless mode for continuous integration
  85. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Either with the Xvfb command:

    $ Xvfb -ac :99 2>/dev/null & $ DISPLAY=:99 ./manage.py test ‣ ... or with Jenkins: Xvfb plugin. Headless mode for continuous integration
  86. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Either with the Xvfb command:

    $ Xvfb -ac :99 2>/dev/null & $ DISPLAY=:99 ./manage.py test ‣ ... or with Jenkins: Xvfb plugin. ‣ ... or with Travis: before_install: - "export DISPLAY=:99.0" - "sh -e /etc/init.d/xvfb start" Headless mode for continuous integration
  87. CONNECTED PERSONAL OBJECTS 5/2012 ‣ ... or with PyVirtualDisplay: from

    pyvirtualdisplay import Display class MyTests(LiveServerTestCase) def setUp(self): self.display = Display( 'xvfb', visible=1,size=(1280, 1024)) self.display.start() def tearDown(self): self.display.stop() Headless mode for continuous integration
  88. CONNECTED PERSONAL OBJECTS 5/2012 Sauce Labs

  89. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Runs remotely in the cloud

    (no local browser needed). Sauce Labs
  90. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Runs remotely in the cloud

    (no local browser needed). ‣ Gives access to multiple browsers (IE, FF, Opera, Chrome, Safari...) and multiple platforms (Windows, Linux, OSX, Android, iPad, iPhone). Sauce Labs
  91. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Runs remotely in the cloud

    (no local browser needed). ‣ Gives access to multiple browsers (IE, FF, Opera, Chrome, Safari...) and multiple platforms (Windows, Linux, OSX, Android, iPad, iPhone). ‣ Allows tests to be run in parallel. Sauce Labs
  92. CONNECTED PERSONAL OBJECTS 5/2012

  93. CONNECTED PERSONAL OBJECTS 5/2012

  94. Not just Selenium...

  95. CONNECTED PERSONAL OBJECTS 5/2012 Ghost.py ‣ http://jeanphix.me/Ghost.py/

  96. CONNECTED PERSONAL OBJECTS 5/2012 Ghost.py from ghost import Ghost class

    MyTests(LiveServerTestCase): def setUp(self): self.ghost = Ghost() super(MyTests, self).setUp() def test_hello(self): self.ghost.open(self.live_server_url + '/hello') self.assertTrue('Hello' in self.ghost.content) ‣ http://jeanphix.me/Ghost.py/
  97. CONNECTED PERSONAL OBJECTS 5/2012 CasperJS ‣ http://casperjs.org/ ‣ https://github.com/dobarkod/django-casper

  98. CONNECTED PERSONAL OBJECTS 5/2012 CasperJS from casper.tests import CasperTestCase class

    MyTests(CasperTestCase): def test_something(self): self.assertTrue(self.casper('test.js')) ‣ http://casperjs.org/ ‣ https://github.com/dobarkod/django-casper
  99. CONNECTED PERSONAL OBJECTS 5/2012 Splinter ‣ http://splinter.cobrateam.info

  100. CONNECTED PERSONAL OBJECTS 5/2012 Splinter from splinter import Browser class

    MyTests(LiveServerTestCase): def setUp(self): self.browser = Browser() super(MyTests, self).setUp() def test_hello(self): self.browser.visit(self.live_server_url + '/hello') self.assertTrue( self.browser.is_text_present('Hello')) ‣ http://splinter.cobrateam.info
  101. Epilogue...

  102. CONNECTED PERSONAL OBJECTS 5/2012 A word of caution...

  103. CONNECTED PERSONAL OBJECTS 5/2012 A word of caution... ‣ Selenium

    tests can be a little flaky.
  104. CONNECTED PERSONAL OBJECTS 5/2012 A word of caution... ‣ Selenium

    tests can be a little flaky. ‣ Integration & functional tests are slow. Use them with moderation.
  105. CONNECTED PERSONAL OBJECTS 5/2012 A word of caution... ‣ Selenium

    tests can be a little flaky. ‣ Integration & functional tests are slow. Use them with moderation. ‣ Use LiveServerTestCase or Selenium only for what the dummy client cannot already achieve.
  106. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important.

  107. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important.

    ‣ Increase your confidence in your code.
  108. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important.

    ‣ Increase your confidence in your code. ‣ Increase your test coverage.
  109. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important.

    ‣ Increase your confidence in your code. ‣ Increase your test coverage. ‣ Test the integration frontend/backend.
  110. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important.

    ‣ Increase your confidence in your code. ‣ Increase your test coverage. ‣ Test the integration frontend/backend. ‣ Ensure the user interface works.
  111. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important.

    ‣ Increase your confidence in your code. ‣ Increase your test coverage. ‣ Test the integration frontend/backend. ‣ Ensure the user interface works. ‣ Have fun in the process!
  112. CONNECTED PERSONAL OBJECTS 5/2012 Photo credits ‣ http://www.flickr.com/photos/t0msk/3148160756/ ‣ http://www.flickr.com/photos/loozrboy/7311771718/

    ‣ http://www.flickr.com/photos/johnonolan/5836211096/ ‣ http://www.flickr.com/photos/randomskk/3057595640/
  113. @julienphalip http:/ /julienphalip.com http:/ /nurun.com Thank you!