PyCon Siberia 2016. Не доверяйте тестам!

Каждый программист рано или поздно начинает писать тесты на свой код. В какой-то момент он начинает задумываться о том, насколько его тесты хороши. В своем докладе я расскажу о том, какие инструменты для проверки качества тестов существуют, как они работают и почему они обманывают нас.

Transcript

1. Цыганов Иван Positive Technologies Не доверяйте тестам!
2. Обо мне ✤ Спикер PyCon Russia 2016, PiterPy#2 и PiterPy#3 ✤ Люблю OpenSource ✤ Не умею frontend
3. ✤ 15 лет практического опыта на рынке ИБ ✤ Более 650 сотрудников в 9 странах ✤ Каждый год находим более 200 уязвимостей нулевого дня ✤ Проводим более 200 аудитов безопасности в крупнейших компаниях мира ежегодно
5. MaxPatrol ✤ Тестирование на проникновение (Pentest) ✤ Системные проверки (Audit) ✤ Соответствие стандартам (Compliance) ✤ Одна из крупнейших баз знаний в мире Система контроля защищенности и соответствия стандартам.
6. ✤ Тестирование на проникновение (Pentest) ✤ Системные проверки (Audit) ✤ Соответствие стандартам (Compliance) ✤ Одна из крупнейших баз знаний в мире Система контроля защищенности и соответствия стандартам. ✤ Системные проверки (Audit) MaxPatrol
7. > 50 000 строк кода
8. Зачем тестировать? ✤ Уверенность, что написанный код работает ✤ Ревью кода становится проще ✤ Гарантия, что ничего не сломалось при изменениях
9. есть тесты != код протестирован
10. Давайте писать тесты! def get_total_price(cart_prices): if len(cart_prices) == 0: return result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return result['TotalPrice'] - result.get('Discount')
11. Плохой тест def get_total_price(cart_prices): if len(cart_prices) == 0: return result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return result['TotalPrice'] - result.get('Discount') def test_get_total_price(): assert get_total_price([90, 10]) == 75
12. Неожиданные данные >>> balance = 1000 >>> >>> goods = [] >>> >>> balance -= get_total_price(goods) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for -=: 'int' and 'NoneType' >>>
13. есть тесты == есть тесты
14. Как сделать тесты лучше? ✤ Проверить покрытие кода тестами ✤ Попробовать мутационное тестирование
15. coverage.py ✤ Позволяет проверить покрытие кода тестами ✤ Есть плагин для pytest
16. coverage.py ✤ Позволяет проверить покрытие кода тестами ✤ Есть плагин для pytest ✤ В основном работает
17. coverage.ini [report]
 show_missing = True
 precision = 2 py.test --cov-config=coverage.ini --cov=target test.py
18. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount') def test_get_total_price(): assert get_total_price([90, 10]) == 75 Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 1 85.71% 2
19. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount') def test_get_total_price(): assert get_total_price([90, 10]) == 75 Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 1 85.71% 2 2 if len(cart_prices) == 0:
20. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount') def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 0 100.00%
21. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount') def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 0 100.00%
22. >>> get_total_price([90]) 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount')
23. >>> get_total_price([90]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 9, in get_total_price TypeError: unsupported operand type(s) for -: 'int' and 'NoneType' >>> 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount')
25. coverage.ini [report]
 show_missing = True
 precision = 2
 [run]
 branch = True py.test --cov-config=coverage.ini --cov=target test.py
26. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 7 0 4 1 90.91% 6 ->9
27. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 7 0 4 1 90.91% 6 ->9
28. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 7 0 4 0 100.00%
30. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price) 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
31. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price) def test_get_total_price(): assert get_total_price([90, 10]) == 75 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 6 1 4 1 80.00% 3, 2 ->3
32. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 6 0 4 0 100.00%
33. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 6 0 4 0 100.00% 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
35. Как считать coverage? Все строки Реально выполненные строки - Непокрытые строки =
36. Все строки Source coverage.parser.PythonParser Statements
37. coverage.parser.PythonParser ✤ Обходит все токены и отмечает «интересные» факты ✤ Компилирует код. Обходит code-object и сохраняет номера строк
38. Обход токенов ✤ Запоминает определения классов ✤ «Сворачивает» многострочные выражения ✤ Исключает комментарии
39. Обход байткода ✤ Полностью повторяет метод dis.findlinestarts ✤ Анализирует code_obj.co_lnotab ✤ Генерирует пару (номер байткода, номер строки)
40. Как считать coverage --branch? Все переходы Реально выполненные переходы - Непокрытые переходы =
41. Все переходы Source coverage.parser.AstArcAnalyzer (from_line, to_line) coverage.parser.PythonParser
42. coverage.parser.AstArcAnalyzer ✤ Обходит AST с корневой ноды ✤ Обрабатывает отдельно каждый тип нод отдельно
43. Обработка ноды class While(stmt): _fields = ( 'test', 'body', 'orelse', ) while i<10: print(i) i += 1
44. Обработка ноды class While(stmt): _fields = ( 'test', 'body', 'orelse', ) while i<10: print(i) i += 1 else: print('All done')
45. Выполненные строки sys.settrace(tracefunc) Set the system’s trace function, which allows you to implement a Python source code debugger in Python. Trace functions should have three arguments: frame, event, and arg. frame is the current stack frame. event is a string: 'call', 'line', 'return', 'exception', 'c_call', 'c_return', or 'c_exception'. arg depends on the event type.
46. PyTracer «call» event ✤ Сохраняем данные предыдущего контекста ✤ Начинаем собирать данные нового контекста ✤ Учитываем особенности генераторов
47. PyTracer «line» event ✤ Запоминаем выполняемую строку ✤ Запоминаем переход между строками
48. PyTracer «return» event ✤ Отмечаем выход из контекста ✤ Помним о том, что yield это тоже return
49. Отчет ✤ Что выполнялось ✤ Что должно было выполниться ✤ Ругаемся
50. Зачем такие сложности? 1 for i in some_list: 2 if i == 'Hello': 3 print(i + ' World!') 4 elif i == 'Skip': 5 continue 6 else: 7 break 8 else: 9 print(r'¯\_(ツ)_/¯')
51. Серебряная пуля?
52. Не совсем…
53. Что может пойти не так? 1 def make_dict(a,b,c): 2 return { 3 'a': a, 4 'b': b if a>1 else 0, 5 'c': [ 6 i for i in range(c) if i<(a*10) 7 ] 6 }
55. Мутационное тестирование ✤ Берем тестируемый код ✤ Мутируем ✤ Тестируем мутантов нашими тестами ✤ Тест не упал -> плохой тест
56. Мутационное тестирование ✤ Берем тестируемый код ✤ Мутируем ✤ Тестируем мутантов нашими тестами ✤ Если тест не упал -> это плохой тест ✤ Тест не упал -> плохой тест
57. Идея def mul(a, b): return a * b def test_mul(): assert mul(2, 2) == 4
58. Идея def mul(a, b): return a * b def test_mul(): assert mul(2, 2) == 4 def mul(a, b): return a ** b
59. Идея def mul(a, b): return a * b def test_mul(): assert mul(2, 2) == 4 def mul(a, b): return a + b def mul(a, b): return a ** b
60. Идея def mul(a, b): return a * b def test_mul(): assert mul(2, 2) == 4 assert mul(2, 3) == 6 def mul(a, b): return a + b def mul(a, b): return a ** b
61. Tools MutPy ✤ Проект заброшен cosmic-ray ✤ Активно развивается ✤ Требует RabbitMQ
62. Реализация Source NodeTransformer compile run test
63. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] / 0.25 8 …
64. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 9 return result['TotalPrice'] + result.get(‘Discount’, 0) …
65. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 2 if (not len(cart_prices) == 0): 3 return 0 …
66. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 2 if len(cart_prices) == 1: 3 return 0 …
67. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 2 if len(cart_prices) == 0: 3 return 1 …
68. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 5 result = {'': sum(cart_prices)} …
69. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 9 return result[‘some_key'] - result.get(‘Discount’, 0)
70. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
71. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(self): self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90) [*] Mutation score [0.50795 s]: 96.4% - all: 28 - killed: 27 (96.4%) - survived: 1 (3.6%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
72. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(self): self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90) [*] Mutation score [0.50795 s]: 96.4% - all: 28 - killed: 27 (96.4%) - survived: 1 (3.6%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%) - survived: 1 (3.6%)
73. … ---------------------------------------------------------- 1: def get_total_price(cart_prices): 2: if len(cart_prices) == 0: ~3: pass 4: 5: result = {'TotalPrice': sum(cart_prices)} 6: if len(cart_prices) >= 2: 7: result['Discount'] = result['TotalPrice'] * 0.25 8: ---------------------------------------------------------- [0.00968 s] survived - [# 26] SDL target:5 : … [*] Mutation score [0.50795 s]: 96.4% - all: 28 - killed: 27 (96.4%) - survived: 1 (3.6%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
74. 1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(self): self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90) [*] Mutation score [0.44658 s]: 100.0% - all: 23 - killed: 23 (100.0%) - survived: 0 (0.0%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
75. Идея имеет право на жизнь и работает! Но требует много ресурсов.
76. 1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90 Name Stmts Miss Cover Missing -------------------------------------------- target.py 5 0 100.00%
77. 1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 5 0 2 0 100.00%
78. 1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 5 0 2 0 100.00%
79. Есть тесты != код протестирован
80. Есть тесты != код протестирован Качество тестов важнее количества
81. Есть тесты != код протестирован Качество тестов важнее количества 100% coverage - не повод расслабляться
84. Simple app app = Flask(__name__) @app.route('/get_total_discount', methods=['POST']) def get_total_discount(): cart_prices = json.loads(request.form['cart_prices']) result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return jsonify(result['TotalPrice'] - result.get('Discount', 0)) flask_app.py
85. pip install pytest-flask @pytest.fixture def app(): from flask_app import app return app def test_get_total_discount(client): get_total_discount = lambda prices: client.post( '/get_total_discount', data=dict(cart_prices=json.dumps(prices)) ).json assert get_total_discount([90, 10]) == 75 assert get_total_discount( []) == 0 assert get_total_discount([90]) == 90 test_flask_app.py
86. pip install pytest-flask Name Stmts Miss Cover Missing ----------------------------------------------- flask_app.py 9 0 100.00% py.test --cov-config=coverage.ini \ --cov=flask_app \ test_flask_app.py Name Stmts Miss Branch BrPart Cover Missing ------------------------------------------------------------- flask_app.py 9 0 2 0 100.00% py.test --cov-config=coverage_branch.ini \ --cov=flask_app \ test_flask_app.py
87. mutpy class FlaskTestCase(unittest.TestCase): def setUp(self): self.app = flask_app.app.test_client() def post(self, path, data): return json.loads(self.app.post(path, data=data).data.decode('utf-8')) def test_get_total_discount(self): get_total_discount = lambda prices: self.post( '/get_total_discount', data=dict(cart_prices=json.dumps(prices)) ) self.assertEqual(get_total_discount([90, 10]), 75) unittest_flask_app.py
88. mutpy [*] Mutation score [0.39122 s]: 100.0% - all: 27 - killed: 1 (3.7%) - survived: 0 (0.0%) - incompetent: 26 (96.3%) - timeout: 0 (0.0%) mut.py --target flask_app --unit-test unittest_flask_app
89. mutpy [*] Mutation score [0.39122 s]: 100.0% - all: 27 - killed: 1 (3.7%) - survived: 0 (0.0%) - incompetent: 26 (96.3%) - timeout: 0 (0.0%) mut.py --target flask_app --unit-test unittest_flask_app
90. mutpy def _matching_loader_thinks_module_is_package(loader, mod_name): #... raise AttributeError( ('%s.is_package() method is missing but is required by Flask of ' 'PEP 302 import hooks. If you do not use import hooks and ' 'you encounter this error please file a bug against Flask.') % loader.__class__.__name__)
91. mutpy def _matching_loader_thinks_module_is_package(loader, mod_name): #... raise AttributeError( ('%s.is_package() method is missing but is required by Flask of ' 'PEP 302 import hooks. If you do not use import hooks and ' 'you encounter this error please file a bug against Flask.') % loader.__class__.__name__) class InjectImporter: def __init__(self, module): # ... def find_module(self, fullname, path=None): # ... def load_module(self, fullname): # ... def install(self): # ... def uninstall(cls): # ...
92. mutpy class InjectImporter: def __init__(self, module): # ... def find_module(self, fullname, path=None): # ... def load_module(self, fullname): # ... def install(self): # ... def uninstall(cls): # … def is_package(self, fullname): # ...
93. mutpy [*] Mutation score [1.14206 s]: 100.0% - all: 27 - killed: 25 (92.6%) - survived: 0 (0.0%) - incompetent: 2 (7.4%) - timeout: 0 (0.0%) mut.py --target flask_app --unit-test unittest_flask_app
95. Simple app import json from django.http import HttpResponse def index(request): cart_prices = json.loads(request.POST['cart_prices']) result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return HttpResponse(result['TotalPrice'] - result.get('Discount', 0)) django_root/billing/views.py
96. pip install pytest-django class TestCase1(TestCase): def test_get_total_price(self): get_total_price = lambda items: json.loads( self.client.post( '/billing/', data={'cart_prices': json.dumps(items)} ).content.decode('utf-8') ) self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90) django_root/billing/tests.py
97. pip install pytest-django Name Stmts Miss Cover Missing --------------------------------------------------- billing/views.py 8 0 100.00% py.test --cov-config=coverage.ini \ --cov=billing.views \ billing/tests.py Name Stmts Miss Branch BrPart Cover Missing ----------------------------------------------------------------- billing/views.py 8 0 2 0 100.00% py.test --cov-config=coverage_branch.ini \ --cov=billing.views \ billing/tests.py
98. mutpy [*] Start mutation process: - targets: billing.views - tests: billing.tests [*] Tests failed: - error in setUpClass (billing.tests.TestCase1) - django.core.exceptions.ImproperlyConfigured: Requested setting DATABASES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings. mut.py --target billing.views --unit-test billing.tests
99. mutpy class Command(BaseCommand): def handle(self, *args, **options): operators_set = operators.standard_operators if options['experimental_operators']: operators_set |= operators.experimental_operators controller = MutationController( target_loader=ModulesLoader(options['target'], None), test_loader=ModulesLoader(options['unit_test'], None), views=[TextView(colored_output=False, show_mutants=True)], mutant_generator=FirstOrderMutator(operators_set) ) controller.run() django_root/mutate_command/management/commands/mutate.py
100. mutpy [*] Mutation score [1.07321 s]: 0.0% - all: 22 - killed: 0 (0.0%) - survived: 22 (100.0%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%) python manage.py mutate \ --target billing.views --unit-test billing.tests
101. mutpy class RegexURLPattern(LocaleRegexProvider): def __init__(self, regex, callback, default_args=None, name=None): LocaleRegexProvider.__init__(self, regex) self.callback = callback # the view self.default_args = default_args or {} self.name = name django.urls.resolvers.RegexURLPattern
102. mutpy import importlib class Command(BaseCommand): def hack_django_for_mutate(self): def set_cb(self, value): self._cb = value def get_cb(self): module = importlib.import_module(self._cb.__module__) return module.__dict__.get(self._cb.__name__) import django.urls.resolvers as r r.RegexURLPattern.callback = property(callback, set_cb) def __init__(self, *args, **kwargs): self.hack_django_for_mutate() super().__init__(*args, **kwargs) def add_arguments(self, parser): # ...
103. mutpy [*] Mutation score [1.48715 s]: 100.0% - all: 22 - killed: 22 (100.0%) - survived: 0 (0.0%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%) python manage.py mutate \ --target billing.views --unit-test billing.tests
104. Спасибо за внимание! Вопросы? mi.0-0.im tsyganov-ivan.com
105. Links ✤ https://github.com/pytest-dev/pytest ✤ https://github.com/pytest-dev/pytest-flask ✤ https://github.com/pytest-dev/pytest-django ✤ https://bitbucket.org/ned/coveragepy ✤ https://github.com/pytest-dev/pytest-cov ✤ https://bitbucket.org/khalas/mutpy ✤ https://github.com/sixty-north/cosmic-ray