Automating Django Functional Tests Using Selenium on Cloud

테스트 코드는 작성한 코드가 제대로 동작하고 있는지 점검하는 코드이다. 경우의 수가 많아질수록 사람이 직접 테스트를 하기 어려워지므로, 테스트 코드를 작성해서 최소한의 코드 안정성을 항상 담보해둘 수 있다. 리팩토링할 때는 테스트 코드를 통해 바뀐 코드에 문제가 있는지 금방 확인할 수 있어 특히 유용하다고 할 수 있다.

Django는 자체적으로 테스트를 위한 기능을 제공하고 있다. 파이썬의 unittest 모듈을 확장해서 유용한 기능을 제공하고, 테스트 유닛마다 DB를 새로 생성해 각 테스트 단위의 독립성을 보장한다. 이를 통해 모델(model), 폼(form), 뷰(view) 단위에서 단위 테스트(unit test)를 작성할 수 있다. 하지만 실제 웹상에서 사용자의 동작에 반응하는 기능 테스트(functional test)를 하기는 매우 어려우므로, 이를 위한 별도의 테스트 프레임워크가 필요하다.

Selenium은 브라우저상에서 직접 사용자의 동작을 에뮬레이션할 수 있는 프레임워크로, Python+Django 환경에서 기능 테스트를 수행하기에 알맞은 기능을 제공한다. 웹드라이버(web driver)를 이용해서 실제 브라우저를 동작시키고, 각 페이지의 DOM에 존재하는 객체를 실제로 조작하는 과정을 파이썬 코드로 작성할 수 있다. 개발자가 원하는 방식으로 다양한 기능 테스트 코드를 작성하고 실행함으로써, 웹페이지와 사용자의 상호작용을 직접 테스트할 수 있다. Django가 제공하는 테스트 프레임워크와 결합하면 보다 더 촘촘한 테스트 망을 구축할 수 있다.

Selenium을 이용한 기능 테스트를 작성할 때 가장 중요한 점 중 하나는, 조작하고자 하는 DOM 객체가 준비(ready)될 때까지 기다리는 것이다. 기능 테스트는 단위 테스트와는 다르게 실제 웹브라우저 상에서 이루어지므로, 브라우저에서 DOM이 작동하는 과정에 대한 고려가 필요하다. 예를 들어, 어떤 페이지를 방문하자마자 DOM 객체에 명령을 내린다고 생각해보자. CPU가 빨리 동작하는 환경이라면 다행히 문제가 없을 수도 있겠지만, 그렇지 않고 객체가 아직 준비되지 않았다면 'undefined'가 반환되면서 적절한 테스트가 불가능하게 될 수 있다. 이런 상황을 피하고자 Selenium은 DOM 객체가 로딩될 때까지 기다리는 두 가지 방법을 가지고 있다. Implicit Wait(암시적 기다림)와 Explicit Wait(명시적 기다림)라고 부르는 방법인데, 이번 세션에서는 이들 기다림과 그 차이에 대해 알아볼 것이다. (결론만 말하면 Implicit Wait는 사용하지 마세요.) 이 외에 테스트를 작성하면서 얻은 나름의 팁에 대해서도 최대한 설명하고자 한다.

잘 작성한 테스트가 있다고 하더라도 개발자가 매번 테스트하는 것은 여간 귀찮은 일이 아닐 수 없다. 그래서 우리가 사용하고 있는 테스트 자동화에 대해서도 간단하게 이야기하고자 한다. 작성된 Selenium 기능 테스트를 클라우드(AWS) 상에 올려두고, 단위 테스트는 Github 커밋이 올라올 때마다, 기능 테스트는 하루에 한 번 자동으로 수행되도록 하고 있다. 이 테스트 환경 구축에 관한 개인적인 경험을 나누려고 한다.

Transcript

1. Automating Django Functional Tests Using Selenium on Cloud Jonghyun Park, Lablup Inc. 2016/08/14 (Sun) @PyCon APAC 2016
2. Contents and target audience • How to functional tests on Django using Selenium. • Running functional tests automatically on AWS EC2. • And some tips on writing functional tests (if time permits). • Target audience • Python beginners - intermediates. • Who have written some tests in Python/Django • … but not much.
3. About speaker • Physics (~4 yr) • Single-molecule Biophysics (~8 yr) • Optical microscope • DNA, protein • IT newbie (~1.2 yr) • HTML, CSS, Javascript, Polymer (Google) • Python-Django
4. About speaker CodeOnWeb (codeonweb.com)
5. Code Test General remarks about code testing Tests on CodeOnWeb
6. Test code • Code that tests code. def login(username, password, ...): ... if success: return True else: return False def test_login(): result = login('test', '1' , ...) assertTrue(result) Test login.py test_login.py
7. Why tests code? • How do we know our code is okay even if we are messing around all the parts of the code? • Ensure minimal code integrity when refactoring. • Man cannot test all features by hand. • Pre-define requirements for a feature.
8. Test-Driven Development (TDD) https://goo.gl/IPwERJ Some people actually do!
9. Writing tests is good for newbie • Can understand the logic of each function/module • … w/o destroying real service whatever you do. • Defeat bugs & improves codes, if you are lucky. https://goo.gl/sU2D2M There are stupid codes, but not stupid test codes! like me
10. Kinds of tests for CodeOnWeb • Unit tests • Test of independent unit of behavior. • e.g., function, class method. • Test layer: model / form / view. • Functional tests • Test of independent piece of functionality. • May test many functions or methods. • Test of actual user interaction on the web browser. lenium
11. Using Selenium Functional testing using selenium with Django
12. Selenium • Browser emulator. • Automate user interaction on the browser. • Can be utilized for testing … and other things. python manage.py test functional_tests.test_login_registration .LoginTest Browser (Firefox, Chrome, …) Selenium Webdriver
13. Test scheme using Selenium Install Selenium pip install selenium Get browser through webdriver browser = webdriver.Firefox() Visit web page browser.get(url) Do whatever you want Test response is expected browser.find_element_by_id( 'lst-ib').send_keys('haha') self.assertEqual(...) self.assertTrue(...)
14. Example: automating googling import time from selenium import webdriver from selenium.webdriver.common.keys import Keys # Get browser through webdriver browser = webdriver.Firefox() # Visit Google! browser.get('https://google.com/') # Enter keyword and enter inp = browser.find_element_by_id('lst-ib') inp.send_keys('PyCon 2016') inp.send_keys(Keys.RETURN) # Just wait a little time.sleep(3) # Close browser window browser.quit() python functional_tests/test_pycon_google1.py This is not a test case!
15. Convert to Django’s test case import time from django.contrib.staticfiles.testing import StaticLiveServerTestCase from selenium import webdriver from selenium.webdriver.common.keys import Keys class GoogleTest(StaticLiveServerTestCase): def setUp(self): self.browser = webdriver.Firefox() self.browser.set_window_size(1024, 768) def tearDown(self): time.sleep(3) self.browser.quit() def test_google_search(self): self.browser.get('https://google.com/') inp = self.browser.find_element_by_id('lst-ib') inp.send_keys('PyCon 2016') inp.send_keys(Keys.RETURN) live django server + collect static files python functional_tests/test_pycon_google2.py (X) python manage.py test functional_tests.test_pycon_google2 (O)
16. Finding elements browser.find_element_by_id('loginForm') .find_element_by_css_selector('input[name="username"]') .find_element_by_name('username') .find_element_by_class_name('login') .find_element_by_tag_name('input') .find_element_by_tag_link_text('Go to Google') .find_element_by_tag_partial_link_text('Go to') .find_element_by_xpath('//form[@id=" loginForm "]') <html> <body> <form id="loginForm" class="login"> <input name="username" type="text" /> <input name="password" type="password" /> <input name="continue" type="submit" value="Login" /> </form> <a href=”https://google.com/">Go to Google</a> </body> <html> browser.find_elements_by_id('loginForm') ...
17. Interacting with elements item = browser.find_element_by_id('entry-item') item.click() ui_type = item.get_attribute('ui-type') browser.execute_script( 'arguments[0].setAttribute("disabled", "true")', item, ) browser.execute_script( 'arguments[0].querySelector("paper-menu").selected = arguments[1]', item, '2' ) Not that difficult if you know Javascript.
18. Let’s write a login test def test_user_login(self): # Store login count of the user user = get_user_model().objects.get(username='einstein') logincount = user.profile.logincount # visit root url ('/') self.browser.get(self.live_server_url) # Try to login with email and password self.browser.find_element_by_name('email').send_keys('einstein@lablup.com') self.browser.find_element_by_name('password').send_keys('1') self.browser.find_element_by_css_selector('fieldset paper-button').click() # Confirm login count is increased by 1 user = get_user_model().objects.get(username='einstein') self.assertEqual(user.profile.logincount, logincount + 1) # Check visit correct url target_url = self.live_server_url + '/accounts/profile/usertype/' self.assertEqual(self.browser.current_url, target_url) python manage.py test functional_tests.test_pycon_login
19. Waits in Selenium Waiting is very important
20. Selenium, the beauty of waiting • Wait what? • Page loading • Elements loading • Elements visibility • Ajax response • DB operation • … https://goo.gl/5ykwc4
21. Broken login example def test_user_login(self): # Store login count of the user user = get_user_model().objects.get(username='einstein') logincount = user.profile.logincount # visit root url ('/') self.browser.get(self.live_server_url) # Try to login with email and password self.browser.find_element_by_name('email').send_keys('einstein@lablup.com') self.browser.find_element_by_name('password').send_keys('1') self.browser.find_element_by_css_selector('fieldset paper-button').click() # Confirm login count is increased by 1 user = get_user_model().objects.get(username='einstein') self.assertEqual(user.profile.logincount, logincount + 1) # Check visit correct url target_url = self.live_server_url + '/accounts/profile/usertype/' self.assertEqual(self.browser.current_url, target_url) python manage.py test functional_tests.test_pycon_login
22. Selenium do not wait automatically login.html submit() views.py … user.profile.logincount += 1 user.profile.save() ... render logged_in.html logged_in.html test_pycon_login.py user.profile.logincount Which one is faster?
23. Selenium do not wait automatically browser.get("http://www.google.com") Dependent on several factors, including the OS/Browser combination, WebDriver may or may not wait for the page to load.
24. Waiting methods in Selenium • Implicit wait • Explicit wait • Stupid wait • It is highly un-recommended to wait for specific amounts of time. • If target is loaded in 5 s, we waste the remaining time. • If target is not loaded in 5 s, the test fails (slow system). import time time.sleep(5) Provided by Selenium
25. Implicit wait • Official DOC: • Easy to use, global effect. • Applies only for element waiting. An implicit wait is to tell WebDriver to poll the DOM for a certain amount of time when trying to find an element or elements if they are not immediately available. The default setting is . Once set, the implicit wait is set for the life of the WebDriver object instance. browser = webdriver.Firefox() browser.implicitly_wait(10) # in seconds browser.get("http://somedomain/url_that_delays_loading") myDynamicElement = browser.find_element_by_id("myDynamicElement")
26. Implicit wait • Waiting behavior is determined on the “remote” side of the webdriver. • Different waiting behavior on different OS, browser, versions, etc. • Return element immediately • Wait until timeout • … def implicitly_wait(self, time_to_wait): self.execute(Command.IMPLICIT_WAIT, {'ms': float(time_to_wait) * 1000}) WebDriver RemoteConnection.execute() Browser (Firefox, Chrome, …) Selenium Webdriver
27. Explicit wait • Official DOC: • A little messy, local effect. An explicit wait is code you define to wait for a certain condition to occur before proceeding further in the code. …WebDriverWait in combination with ExpectedCondition is one way this can be accomplished. from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait import selenium.webdriver.support.expected_conditions as EC ff = webdriver.Firefox() ff.get("http://somedomain/url_that_delays_loading") try: element = WebDriverWait(ff, 10).until( EC.presence_of_element_located((By.ID, "myDynamicElement")) ) finally: ff.quit()
28. Explicit wait • Highly flexible (variety of expected conditions) EC.title_is(...) .title_contains(...) .presence_of_element_located(...) .visibility_of_element_located(...) .visibility_of(...) .presence_of_all_elements_located(...) .text_to_be_present_in_element(...) .text_to_be_present_in_element_value(...) .frame_to_be_available_and_switch_to_it(...) .invisibility_of_element_located(...) .element_to_be_clickable(...) .staleness_of(...) .element_to_be_selected(...) .element_located_to_be_selected(...) .element_selection_state_to_be(...) .element_located_selection_state_to_be(...) .alert_is_present(...)
29. Explicit wait • Highly flexible (not limited for waiting element) • Waiting is controlled on the “local” side. • Defined behavior. def until(self, method, message=''): end_time = time.time() + self._timeout while(True): try: value = method(self._driver) if value: return value except self._ignored_exceptions: pass time.sleep(self._poll) if(time.time() > end_time): break raise TimeoutException(message) condition = lambda _: self.browser.execute_script('return document.readyState') == 'complete' WebDriverWait(self.browser, 30).until(condition) Browser (Firefox, Chrome, …) Selenium Webdriver
30. Use explicit wait • Use explicit wait! • In good conditions (OS, browser, version), implicit wait can simplify test codes. • Be aware that the condition may change in the future. • I think implicit wait is not very stable way to go. • Never use stupid wait. • Except in a very limited cases (like presentation?).
31. Running Tests on Cloud (AWS EC2) Functional tests: automatically once a day
32. Run selenium on AWS • No screen • Virtual display • Slow machine (low cost) • Give timeout generously • That’s all!
33. Run selenium on AWS # Install headless java sudo apt-get install openjdk-6-jre-headless # Fonts sudo apt-get install xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic # Headless X11 magic is here sudo apt-get install xvfb # We still demand X11 core sudo apt-get install xserver-xorg-core # Firefox installation sudo apt-get install firefox # Download Selenium server wget http://selenium.googlecode.com/files/selenium-server-standalone-2.31.0.jar # Run Selenium server Xvfb :0 -screen 0 1440x900x16 2>&1 >/dev/null & export DISPLAY=:0 nohup xvfb-run java -jar selenium-server-standalone-2.53.0.jar > selenium.log & http://goo.gl/GRbztO + crontab
34. Test automation for CodeOnWeb Unit tests Functional tests Local1 Test server (on EC2) git commit; git push Actual codes Github Github-Auto-Deploy (listen hook ➡ run scripts) hook git pull post-merge hook Run DB migration Run unit tests listen err no err Error posting script Run functional tests err no err Unit tests Functional tests Local2 git commit; git push Actual codes once a day comment on issue … +Slack/Teams integration
35. Slack integration is also possible
36. Some Tips Most of them is on the web.
37. Faster password hasher • Use fastest password hasher (MD5) in testing. • Django’s default: PBKDF2 algorithm with SHA256 hash. • It’s more secure, but quite slow. • In our case, this reduces the test running time by • half (for unit tests). • ~10% (for functional tests). • Never use this in production server. PASSWORD_HASHERS = [ 'django.contrib.auth.hashers.MD5PasswordHasher', ] settings.py
38. Shorter login from django.conf import settings from django.contrib.auth import get_user_model, BACKEND_SESSION_KEY, SESSION_KEY, HASH_SESSION_KEY from django.contrib.sessions.backends.db import SessionStore def login_browser(self, username='einstein'): # Set fake session user = get_user_model().objects.get(username=username) session = SessionStore() session[SESSION_KEY] = user.pk session[BACKEND_SESSION_KEY] = 'django.contrib.auth.backends.ModelBackend' session[HASH_SESSION_KEY] = user.get_session_auth_hash() session.save() # To set cookies, the browser should visit a page (due to same origin policy). # It is faster to visit dummy page than login page itself. self.browser.get(self.live_server_url + '/selenium_dummy') self.browser.add_cookie({ 'name': settings.SESSION_COOKIE_NAME, 'value': session.session_key, 'secure': False, 'path': '/' }) Tests for logged-in user is requred.
39. Fluently wait • Wait for page load. class CommonMethods: @contextmanager def wait_for_page_load(self, timeout=TIMEOUT): old_element = self.browser.find_element_by_tag_name('html') yield WebDriverWait(self.browser, timeout).until(EC.staleness_of(old_element)) # Wait until ready state is complete self.wait_until( lambda _: self.browser.execute_script('return document.readyState') == 'complete' ) ... with self.wait_for_page_load(): self.browser.get(self.live_server_url + '/dashboard/entry/' self.browser.get(self.live_server_url + '/dashboard/entry/') ?
40. Fluently wait • Simpler explicit wait. class CommonMethods: ... def wait_until(self, condition): WebDriverWait(self.browser, TIMEOUT).until(condition) element = self.wait_until( EC.element_to_be_clickable((By.ID, 'myDynamicElement')) ) element = WebDriverWait(ff, 10).until( EC.presence_of_element_located((By.ID, "myDynamicElement")) )
41. Make your own useful scripts class CommonMethods: ... def clear_input(self, element): self.browser.execute_script('arguments[0].value = ""', element) def click_button(self, element): self.browser.execute_script('arguments[0].click()', element) self.clear_input(self.elm['email_input']) Using send_keys(str) appends str, not replacing Using selenium’s click() sometimes not working (animated buttons, …) self.click_button(self.elm['submit_button'])
42. Separate page implementation from test codes CommonMethods WebPageBase FunctionalTestBase LogInPage SignupPage … LogInTest SignupProcessTest … /accounts/login/ /accounts/signup/
43. Web page classes class LogInPage(WebPageBase): def __init__(self, browser): super(LogInPage, self).__init__(browser) self.elm.update({ 'email_input': browser.find_element_by_name('email'), 'pwd_input': browser.find_element_by_name('password'), 'submit_button': browser.find_element_by_css_selector('fieldset paper-button'), ... }) def login(self, email, password): self.elm['email_input'].send_keys(email) self.elm['pwd_input'].send_keys(password) with self.wait_for_page_load(): self.click_button(self.elm['submit_button']) def click_signup_button(self): with self.wait_for_page_load(): self.click_button(self.elm['signup_link']) def click_forgot_password_button(self): with self.wait_for_page_load(): self.click_button(self.elm['forgot_pwd']) class SignUpPage(WebPageBase): ... pre-define static elements modularize representative features
44. Test classes using web page classes class LoginTest(FunctionalTestBase): def test_user_login(self): # Store login count of the user user = get_user_model().objects.get(username='einstein') logincount = user.profile.logincount # Visit login page and login with self.wait_for_page_load(): self.browser.get(self.live_server_url) login_page = LogInPage(self.browser) login_page.login('einstein@lablup.com', '0000') # Check login count is increased by 1 user = get_user_model().objects.get(username='einstein') self.assertEqual(user.profile.logincount, logincount + 1) # Check expected url target_url = self.live_server_url + '/dashboard/' self.assertEqual(self.browser.current_url, target_url) class SignupProcessTest(FunctionalTestBase): def test_user_signup(self): # User clicks signup button at login page with self.wait_for_page_load(): self.browser.get(self.live_server_url) login_page = LogInPage(self.browser) login_page.click_signup_button() ... use web page classes / methods
45. Summary • Using Selenium for Django functional tests. • Wait is important in Selenium. •Use explicit wait. • Automate tests on AWS EC2. • Give structure to your tests. • Make your useful scripts.
46. Thank you! • Staffs of APAC PyCon 2016 • … And ALL of YOU! http://www.lablup.com/ https://codeonweb.com/ https://kid.codeonweb.com/
47. Additional Information
48. Hierarchy of Django unit testing classes https://goo.gl/yDZlCw
49. Set Github webhook for unit test • Repository settings -Webhooks & services. • Add webhook for push. • When commit is pushed, send webhook to a test server. Test server
50. Process webhook on AWS EC2 • Create an EC2 instance for testing server. • Catch webhook and execute a command. • Github-Auto-Deploy (Python2). { "port": 9092, "repositories": [{ "url": "https://github.com/lablup/neumann", "path": "/home/jpark/neumann", "deploy": "git pull" }] } GitAutoDeploy.conf.json python2 GitAutoDeploy.py
51. Set post-merge hook • Run tests and record errors. • If error occurred, comment on a Github issue. • github3 package. #!/bin/bash /home/jpark/.pyenv/versions/neumann/bin/python3 publish_unittest_errors.py .git/hooks/post-merge