$30 off During Our Annual Pro Sale. View details »

PyCon Siberia 2016. Не доверяйте тестам!

PyCon Siberia 2016. Не доверяйте тестам!

Каждый программист рано или поздно начинает писать тесты на свой код. В какой-то момент он начинает задумываться о том, насколько его тесты хороши. В своем докладе я расскажу о том, какие инструменты для проверки качества тестов существуют, как они работают и почему они обманывают нас.

Ivan Tsyganov

October 03, 2016
Tweet

More Decks by Ivan Tsyganov

Other Decks in Programming

Transcript

  1. Цыганов Иван Positive Technologies Не доверяйте тестам!

  2. Обо мне ✤ Спикер PyCon Russia 2016, PiterPy#2 и PiterPy#3

    ✤ Люблю OpenSource ✤ Не умею frontend
  3. ✤ 15 лет практического опыта на рынке ИБ ✤ Более

    650 сотрудников в 9 странах ✤ Каждый год находим более 200 уязвимостей нулевого дня ✤ Проводим более 200 аудитов безопасности в крупнейших компаниях мира ежегодно
  4. None
  5. MaxPatrol ✤ Тестирование на проникновение (Pentest) ✤ Системные проверки (Audit)

    ✤ Соответствие стандартам (Compliance) ✤ Одна из крупнейших баз знаний в мире Система контроля защищенности и соответствия стандартам.
  6. ✤ Тестирование на проникновение (Pentest) ✤ Системные проверки (Audit) ✤

    Соответствие стандартам (Compliance) ✤ Одна из крупнейших баз знаний в мире Система контроля защищенности и соответствия стандартам. ✤ Системные проверки (Audit) MaxPatrol
  7. > 50 000 строк кода

  8. Зачем тестировать? ✤ Уверенность, что написанный код работает ✤ Ревью

    кода становится проще ✤ Гарантия, что ничего не сломалось при изменениях
  9. есть тесты != код протестирован

  10. Давайте писать тесты! def get_total_price(cart_prices): if len(cart_prices) == 0: return

    result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return result['TotalPrice'] - result.get('Discount')
  11. Плохой тест def get_total_price(cart_prices): if len(cart_prices) == 0: return result

    = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return result['TotalPrice'] - result.get('Discount') def test_get_total_price(): assert get_total_price([90, 10]) == 75
  12. Неожиданные данные >>> balance = 1000 >>> >>> goods =

    [] >>> >>> balance -= get_total_price(goods) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for -=: 'int' and 'NoneType' >>>
  13. есть тесты == есть тесты

  14. Как сделать тесты лучше? ✤ Проверить покрытие кода тестами ✤

    Попробовать мутационное тестирование
  15. coverage.py ✤ Позволяет проверить покрытие кода тестами ✤ Есть плагин

    для pytest
  16. coverage.py ✤ Позволяет проверить покрытие кода тестами ✤ Есть плагин

    для pytest ✤ В основном работает
  17. coverage.ini [report]
 show_missing = True
 precision = 2 py.test --cov-config=coverage.ini

    --cov=target test.py
  18. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount') def test_get_total_price(): assert get_total_price([90, 10]) == 75 Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 1 85.71% 2
  19. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount') def test_get_total_price(): assert get_total_price([90, 10]) == 75 Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 1 85.71% 2 2 if len(cart_prices) == 0:
  20. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount') def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 0 100.00%
  21. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount') def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 0 100.00%
  22. >>> get_total_price([90]) 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0:

    3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount')
  23. >>> get_total_price([90]) Traceback (most recent call last): File "<stdin>", line

    1, in <module> File "<stdin>", line 9, in get_total_price TypeError: unsupported operand type(s) for -: 'int' and 'NoneType' >>> 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount')
  24. None
  25. coverage.ini [report]
 show_missing = True
 precision = 2
 [run]
 branch

    = True py.test --cov-config=coverage.ini --cov=target test.py
  26. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 7 0 4 1 90.91% 6 ->9
  27. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 7 0 4 1 90.91% 6 ->9
  28. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 7 0 4 0 100.00%
  29. None
  30. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price) 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
  31. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price) def test_get_total_price(): assert get_total_price([90, 10]) == 75 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 6 1 4 1 80.00% 3, 2 ->3
  32. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 6 0 4 0 100.00%
  33. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 6 0 4 0 100.00% 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
  34. None
  35. Как считать coverage? Все строки Реально выполненные строки - Непокрытые

    строки =
  36. Все строки Source coverage.parser.PythonParser Statements

  37. coverage.parser.PythonParser ✤ Обходит все токены и отмечает «интересные» факты ✤

    Компилирует код. Обходит code-object и сохраняет номера строк
  38. Обход токенов ✤ Запоминает определения классов ✤ «Сворачивает» многострочные выражения

    ✤ Исключает комментарии
  39. Обход байткода ✤ Полностью повторяет метод dis.findlinestarts ✤ Анализирует code_obj.co_lnotab

    ✤ Генерирует пару (номер байткода, номер строки)
  40. Как считать coverage --branch? Все переходы Реально выполненные переходы -

    Непокрытые переходы =
  41. Все переходы Source coverage.parser.AstArcAnalyzer (from_line, to_line) coverage.parser.PythonParser

  42. coverage.parser.AstArcAnalyzer ✤ Обходит AST с корневой ноды ✤ Обрабатывает отдельно

    каждый тип нод отдельно
  43. Обработка ноды class While(stmt): _fields = ( 'test', 'body', 'orelse',

    ) while i<10: print(i) i += 1
  44. Обработка ноды class While(stmt): _fields = ( 'test', 'body', 'orelse',

    ) while i<10: print(i) i += 1 else: print('All done')
  45. Выполненные строки sys.settrace(tracefunc) Set the system’s trace function, which allows

    you to implement a Python source code debugger in Python. Trace functions should have three arguments: frame, event, and arg. frame is the current stack frame. event is a string: 'call', 'line', 'return', 'exception', 'c_call', 'c_return', or 'c_exception'. arg depends on the event type.
  46. PyTracer «call» event ✤ Сохраняем данные предыдущего контекста ✤ Начинаем

    собирать данные нового контекста ✤ Учитываем особенности генераторов
  47. PyTracer «line» event ✤ Запоминаем выполняемую строку ✤ Запоминаем переход

    между строками
  48. PyTracer «return» event ✤ Отмечаем выход из контекста ✤ Помним

    о том, что yield это тоже return
  49. Отчет ✤ Что выполнялось ✤ Что должно было выполниться ✤

    Ругаемся
  50. Зачем такие сложности? 1 for i in some_list: 2 if

    i == 'Hello': 3 print(i + ' World!') 4 elif i == 'Skip': 5 continue 6 else: 7 break 8 else: 9 print(r'¯\_(ツ)_/¯')
  51. Серебряная пуля?

  52. Не совсем…

  53. Что может пойти не так? 1 def make_dict(a,b,c): 2 return

    { 3 'a': a, 4 'b': b if a>1 else 0, 5 'c': [ 6 i for i in range(c) if i<(a*10) 7 ] 6 }
  54. None
  55. Мутационное тестирование ✤ Берем тестируемый код ✤ Мутируем ✤ Тестируем

    мутантов нашими тестами ✤ Тест не упал -> плохой тест
  56. Мутационное тестирование ✤ Берем тестируемый код ✤ Мутируем ✤ Тестируем

    мутантов нашими тестами ✤ Если тест не упал -> это плохой тест ✤ Тест не упал -> плохой тест
  57. Идея def mul(a, b): return a * b def test_mul():

    assert mul(2, 2) == 4
  58. Идея def mul(a, b): return a * b def test_mul():

    assert mul(2, 2) == 4 def mul(a, b): return a ** b
  59. Идея def mul(a, b): return a * b def test_mul():

    assert mul(2, 2) == 4 def mul(a, b): return a + b def mul(a, b): return a ** b
  60. Идея def mul(a, b): return a * b def test_mul():

    assert mul(2, 2) == 4 assert mul(2, 3) == 6 def mul(a, b): return a + b def mul(a, b): return a ** b
  61. Tools MutPy ✤ Проект заброшен cosmic-ray ✤ Активно развивается ✤

    Требует RabbitMQ
  62. Реализация Source NodeTransformer compile run test

  63. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3

    return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] / 0.25 8 …
  64. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3

    return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 9 return result['TotalPrice'] + result.get(‘Discount’, 0) …
  65. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3

    return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 2 if (not len(cart_prices) == 0): 3 return 0 …
  66. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3

    return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 2 if len(cart_prices) == 1: 3 return 0 …
  67. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3

    return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 2 if len(cart_prices) == 0: 3 return 1 …
  68. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3

    return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 5 result = {'': sum(cart_prices)} …
  69. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3

    return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) … 9 return result[‘some_key'] - result.get(‘Discount’, 0)
  70. Мутации 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3

    return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
  71. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(self): self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90) [*] Mutation score [0.50795 s]: 96.4% - all: 28 - killed: 27 (96.4%) - survived: 1 (3.6%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
  72. 1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return

    0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(self): self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90) [*] Mutation score [0.50795 s]: 96.4% - all: 28 - killed: 27 (96.4%) - survived: 1 (3.6%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%) - survived: 1 (3.6%)
  73. … ---------------------------------------------------------- 1: def get_total_price(cart_prices): 2: if len(cart_prices) == 0:

    ~3: pass 4: 5: result = {'TotalPrice': sum(cart_prices)} 6: if len(cart_prices) >= 2: 7: result['Discount'] = result['TotalPrice'] * 0.25 8: ---------------------------------------------------------- [0.00968 s] survived - [# 26] SDL target:5 : … [*] Mutation score [0.50795 s]: 96.4% - all: 28 - killed: 27 (96.4%) - survived: 1 (3.6%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
  74. 1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if

    len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(self): self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90) [*] Mutation score [0.44658 s]: 100.0% - all: 23 - killed: 23 (100.0%) - survived: 0 (0.0%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
  75. Идея имеет право на жизнь и работает! Но требует много

    ресурсов.
  76. 1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if

    len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90 Name Stmts Miss Cover Missing -------------------------------------------- target.py 5 0 100.00%
  77. 1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if

    len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 5 0 2 0 100.00%
  78. 1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if

    len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0) def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90 Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 5 0 2 0 100.00%
  79. Есть тесты != код протестирован

  80. Есть тесты != код протестирован Качество тестов важнее количества

  81. Есть тесты != код протестирован Качество тестов важнее количества 100%

    coverage - не повод расслабляться
  82. None
  83. None
  84. Simple app app = Flask(__name__) @app.route('/get_total_discount', methods=['POST']) def get_total_discount(): cart_prices

    = json.loads(request.form['cart_prices']) result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return jsonify(result['TotalPrice'] - result.get('Discount', 0)) flask_app.py
  85. pip install pytest-flask @pytest.fixture def app(): from flask_app import app

    return app def test_get_total_discount(client): get_total_discount = lambda prices: client.post( '/get_total_discount', data=dict(cart_prices=json.dumps(prices)) ).json assert get_total_discount([90, 10]) == 75 assert get_total_discount( []) == 0 assert get_total_discount([90]) == 90 test_flask_app.py
  86. pip install pytest-flask Name Stmts Miss Cover Missing ----------------------------------------------- flask_app.py

    9 0 100.00% py.test --cov-config=coverage.ini \ --cov=flask_app \ test_flask_app.py Name Stmts Miss Branch BrPart Cover Missing ------------------------------------------------------------- flask_app.py 9 0 2 0 100.00% py.test --cov-config=coverage_branch.ini \ --cov=flask_app \ test_flask_app.py
  87. mutpy class FlaskTestCase(unittest.TestCase): def setUp(self): self.app = flask_app.app.test_client() def post(self,

    path, data): return json.loads(self.app.post(path, data=data).data.decode('utf-8')) def test_get_total_discount(self): get_total_discount = lambda prices: self.post( '/get_total_discount', data=dict(cart_prices=json.dumps(prices)) ) self.assertEqual(get_total_discount([90, 10]), 75) unittest_flask_app.py
  88. mutpy [*] Mutation score [0.39122 s]: 100.0% - all: 27

    - killed: 1 (3.7%) - survived: 0 (0.0%) - incompetent: 26 (96.3%) - timeout: 0 (0.0%) mut.py --target flask_app --unit-test unittest_flask_app
  89. mutpy [*] Mutation score [0.39122 s]: 100.0% - all: 27

    - killed: 1 (3.7%) - survived: 0 (0.0%) - incompetent: 26 (96.3%) - timeout: 0 (0.0%) mut.py --target flask_app --unit-test unittest_flask_app
  90. mutpy def _matching_loader_thinks_module_is_package(loader, mod_name): #... raise AttributeError( ('%s.is_package() method is

    missing but is required by Flask of ' 'PEP 302 import hooks. If you do not use import hooks and ' 'you encounter this error please file a bug against Flask.') % loader.__class__.__name__)
  91. mutpy def _matching_loader_thinks_module_is_package(loader, mod_name): #... raise AttributeError( ('%s.is_package() method is

    missing but is required by Flask of ' 'PEP 302 import hooks. If you do not use import hooks and ' 'you encounter this error please file a bug against Flask.') % loader.__class__.__name__) class InjectImporter: def __init__(self, module): # ... def find_module(self, fullname, path=None): # ... def load_module(self, fullname): # ... def install(self): # ... def uninstall(cls): # ...
  92. mutpy class InjectImporter: def __init__(self, module): # ... def find_module(self,

    fullname, path=None): # ... def load_module(self, fullname): # ... def install(self): # ... def uninstall(cls): # … def is_package(self, fullname): # ...
  93. mutpy [*] Mutation score [1.14206 s]: 100.0% - all: 27

    - killed: 25 (92.6%) - survived: 0 (0.0%) - incompetent: 2 (7.4%) - timeout: 0 (0.0%) mut.py --target flask_app --unit-test unittest_flask_app
  94. None
  95. Simple app import json from django.http import HttpResponse def index(request):

    cart_prices = json.loads(request.POST['cart_prices']) result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return HttpResponse(result['TotalPrice'] - result.get('Discount', 0)) django_root/billing/views.py
  96. pip install pytest-django class TestCase1(TestCase): def test_get_total_price(self): get_total_price = lambda

    items: json.loads( self.client.post( '/billing/', data={'cart_prices': json.dumps(items)} ).content.decode('utf-8') ) self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90) django_root/billing/tests.py
  97. pip install pytest-django Name Stmts Miss Cover Missing --------------------------------------------------- billing/views.py

    8 0 100.00% py.test --cov-config=coverage.ini \ --cov=billing.views \ billing/tests.py Name Stmts Miss Branch BrPart Cover Missing ----------------------------------------------------------------- billing/views.py 8 0 2 0 100.00% py.test --cov-config=coverage_branch.ini \ --cov=billing.views \ billing/tests.py
  98. mutpy [*] Start mutation process: - targets: billing.views - tests:

    billing.tests [*] Tests failed: - error in setUpClass (billing.tests.TestCase1) - django.core.exceptions.ImproperlyConfigured: Requested setting DATABASES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings. mut.py --target billing.views --unit-test billing.tests
  99. mutpy class Command(BaseCommand): def handle(self, *args, **options): operators_set = operators.standard_operators

    if options['experimental_operators']: operators_set |= operators.experimental_operators controller = MutationController( target_loader=ModulesLoader(options['target'], None), test_loader=ModulesLoader(options['unit_test'], None), views=[TextView(colored_output=False, show_mutants=True)], mutant_generator=FirstOrderMutator(operators_set) ) controller.run() django_root/mutate_command/management/commands/mutate.py
  100. mutpy [*] Mutation score [1.07321 s]: 0.0% - all: 22

    - killed: 0 (0.0%) - survived: 22 (100.0%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%) python manage.py mutate \ --target billing.views --unit-test billing.tests
  101. mutpy class RegexURLPattern(LocaleRegexProvider): def __init__(self, regex, callback, default_args=None, name=None): LocaleRegexProvider.__init__(self,

    regex) self.callback = callback # the view self.default_args = default_args or {} self.name = name django.urls.resolvers.RegexURLPattern
  102. mutpy import importlib class Command(BaseCommand): def hack_django_for_mutate(self): def set_cb(self, value):

    self._cb = value def get_cb(self): module = importlib.import_module(self._cb.__module__) return module.__dict__.get(self._cb.__name__) import django.urls.resolvers as r r.RegexURLPattern.callback = property(callback, set_cb) def __init__(self, *args, **kwargs): self.hack_django_for_mutate() super().__init__(*args, **kwargs) def add_arguments(self, parser): # ...
  103. mutpy [*] Mutation score [1.48715 s]: 100.0% - all: 22

    - killed: 22 (100.0%) - survived: 0 (0.0%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%) python manage.py mutate \ --target billing.views --unit-test billing.tests
  104. Спасибо за внимание! Вопросы? mi.0-0.im tsyganov-ivan.com

  105. Links ✤ https://github.com/pytest-dev/pytest ✤ https://github.com/pytest-dev/pytest-flask ✤ https://github.com/pytest-dev/pytest-django ✤ https://bitbucket.org/ned/coveragepy ✤

    https://github.com/pytest-dev/pytest-cov ✤ https://bitbucket.org/khalas/mutpy ✤ https://github.com/sixty-north/cosmic-ray