Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PyCon Siberia 2016. Не доверяйте тестам!

PyCon Siberia 2016. Не доверяйте тестам!

Каждый программист рано или поздно начинает писать тесты на свой код. В какой-то момент он начинает задумываться о том, насколько его тесты хороши. В своем докладе я расскажу о том, какие инструменты для проверки качества тестов существуют, как они работают и почему они обманывают нас.

Ivan Tsyganov

October 03, 2016
Tweet

More Decks by Ivan Tsyganov

Other Decks in Programming

Transcript

  1. Цыганов Иван
    Positive Technologies
    Не доверяйте тестам!

    View full-size slide

  2. Обо мне
    ✤ Спикер PyCon Russia 2016,
    PiterPy#2 и PiterPy#3
    ✤ Люблю OpenSource
    ✤ Не умею frontend

    View full-size slide

  3. ✤ 15 лет практического опыта на рынке ИБ
    ✤ Более 650 сотрудников в 9 странах
    ✤ Каждый год находим более 200 уязвимостей
    нулевого дня
    ✤ Проводим более 200 аудитов безопасности в
    крупнейших компаниях мира ежегодно

    View full-size slide

  4. MaxPatrol
    ✤ Тестирование на проникновение (Pentest)
    ✤ Системные проверки (Audit)
    ✤ Соответствие стандартам (Compliance)
    ✤ Одна из крупнейших баз знаний в мире
    Система контроля защищенности и соответствия
    стандартам.

    View full-size slide

  5. ✤ Тестирование на проникновение (Pentest)
    ✤ Системные проверки (Audit)
    ✤ Соответствие стандартам (Compliance)
    ✤ Одна из крупнейших баз знаний в мире
    Система контроля защищенности и соответствия
    стандартам.
    ✤ Системные проверки (Audit)
    MaxPatrol

    View full-size slide

  6. > 50 000 строк кода

    View full-size slide

  7. Зачем тестировать?
    ✤ Уверенность, что написанный код работает
    ✤ Ревью кода становится проще
    ✤ Гарантия, что ничего не сломалось при
    изменениях

    View full-size slide

  8. есть тесты != код протестирован

    View full-size slide

  9. Давайте писать тесты!
    def get_total_price(cart_prices):
    if len(cart_prices) == 0:
    return
    result = {'TotalPrice': sum(cart_prices)}
    if len(cart_prices) >= 2:
    result['Discount'] = result['TotalPrice'] * 0.25
    return result['TotalPrice'] - result.get('Discount')

    View full-size slide

  10. Плохой тест
    def get_total_price(cart_prices):
    if len(cart_prices) == 0:
    return
    result = {'TotalPrice': sum(cart_prices)}
    if len(cart_prices) >= 2:
    result['Discount'] = result['TotalPrice'] * 0.25
    return result['TotalPrice'] - result.get('Discount')
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75

    View full-size slide

  11. Неожиданные данные
    >>> balance = 1000
    >>>
    >>> goods = []
    >>>
    >>> balance -= get_total_price(goods)
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: unsupported operand type(s) for -=: 'int' and 'NoneType'
    >>>

    View full-size slide

  12. есть тесты == есть тесты

    View full-size slide

  13. Как сделать тесты лучше?
    ✤ Проверить покрытие кода тестами
    ✤ Попробовать мутационное тестирование

    View full-size slide

  14. coverage.py
    ✤ Позволяет проверить покрытие кода тестами
    ✤ Есть плагин для pytest

    View full-size slide

  15. coverage.py
    ✤ Позволяет проверить покрытие кода тестами
    ✤ Есть плагин для pytest
    ✤ В основном работает

    View full-size slide

  16. coverage.ini
    [report]

    show_missing = True

    precision = 2
    py.test --cov-config=coverage.ini --cov=target test.py

    View full-size slide

  17. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    Name Stmts Miss Cover Missing
    --------------------------------------------
    target.py 7 1 85.71% 2

    View full-size slide

  18. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    Name Stmts Miss Cover Missing
    --------------------------------------------
    target.py 7 1 85.71% 2
    2 if len(cart_prices) == 0:

    View full-size slide

  19. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Cover Missing
    --------------------------------------------
    target.py 7 0 100.00%

    View full-size slide

  20. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Cover Missing
    --------------------------------------------
    target.py 7 0 100.00%

    View full-size slide

  21. >>> get_total_price([90])
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')

    View full-size slide

  22. >>> get_total_price([90])
    Traceback (most recent call last):
    File "", line 1, in
    File "", line 9, in get_total_price
    TypeError: unsupported operand type(s) for -: 'int' and
    'NoneType'
    >>>
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')

    View full-size slide

  23. coverage.ini
    [report]

    show_missing = True

    precision = 2

    [run]

    branch = True
    py.test --cov-config=coverage.ini --cov=target test.py

    View full-size slide

  24. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 7 0 4 1 90.91% 6 ->9

    View full-size slide

  25. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 7 0 4 1 90.91% 6 ->9

    View full-size slide

  26. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    assert get_total_price([90]) == 90
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 7 0 4 0 100.00%

    View full-size slide

  27. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 total_price = sum(cart_prices)
    6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
    7
    8 return total_price-get_discount(cart_prices, total_price)
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    View full-size slide

  28. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 total_price = sum(cart_prices)
    6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
    7
    8 return total_price-get_discount(cart_prices, total_price)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 6 1 4 1 80.00% 3, 2 ->3

    View full-size slide

  29. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 total_price = sum(cart_prices)
    6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
    7
    8 return total_price-get_discount(cart_prices, total_price)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 6 0 4 0 100.00%

    View full-size slide

  30. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 total_price = sum(cart_prices)
    6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
    7
    8 return total_price-get_discount(cart_prices, total_price)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 6 0 4 0 100.00%
    6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25

    View full-size slide

  31. Как считать coverage?
    Все строки
    Реально выполненные
    строки
    - Непокрытые
    строки
    =

    View full-size slide

  32. Все строки
    Source
    coverage.parser.PythonParser
    Statements

    View full-size slide

  33. coverage.parser.PythonParser
    ✤ Обходит все токены и отмечает «интересные»
    факты
    ✤ Компилирует код. Обходит code-object и
    сохраняет номера строк

    View full-size slide

  34. Обход токенов
    ✤ Запоминает определения классов
    ✤ «Сворачивает» многострочные выражения
    ✤ Исключает комментарии

    View full-size slide

  35. Обход байткода
    ✤ Полностью повторяет метод dis.findlinestarts
    ✤ Анализирует code_obj.co_lnotab
    ✤ Генерирует пару (номер байткода, номер строки)

    View full-size slide

  36. Как считать coverage --branch?
    Все переходы
    Реально выполненные
    переходы
    - Непокрытые
    переходы
    =

    View full-size slide

  37. Все переходы
    Source
    coverage.parser.AstArcAnalyzer
    (from_line, to_line)
    coverage.parser.PythonParser

    View full-size slide

  38. coverage.parser.AstArcAnalyzer
    ✤ Обходит AST с корневой ноды
    ✤ Обрабатывает отдельно каждый тип нод отдельно

    View full-size slide

  39. Обработка ноды
    class While(stmt):
    _fields = (
    'test',
    'body',
    'orelse',
    )
    while i<10:
    print(i)
    i += 1

    View full-size slide

  40. Обработка ноды
    class While(stmt):
    _fields = (
    'test',
    'body',
    'orelse',
    )
    while i<10:
    print(i)
    i += 1
    else:
    print('All done')

    View full-size slide

  41. Выполненные строки
    sys.settrace(tracefunc)
    Set the system’s trace function, which allows you to implement a
    Python source code debugger in Python.
    Trace functions should have three arguments: frame, event, and
    arg. frame is the current stack frame. event is a string: 'call',
    'line', 'return', 'exception', 'c_call', 'c_return', or
    'c_exception'. arg depends on the event type.

    View full-size slide

  42. PyTracer «call» event
    ✤ Сохраняем данные предыдущего контекста
    ✤ Начинаем собирать данные нового контекста
    ✤ Учитываем особенности генераторов

    View full-size slide

  43. PyTracer «line» event
    ✤ Запоминаем выполняемую строку
    ✤ Запоминаем переход между строками

    View full-size slide

  44. PyTracer «return» event
    ✤ Отмечаем выход из контекста
    ✤ Помним о том, что yield это тоже return

    View full-size slide

  45. Отчет
    ✤ Что выполнялось
    ✤ Что должно было выполниться
    ✤ Ругаемся

    View full-size slide

  46. Зачем такие сложности?
    1 for i in some_list:
    2 if i == 'Hello':
    3 print(i + ' World!')
    4 elif i == 'Skip':
    5 continue
    6 else:
    7 break
    8 else:
    9 print(r'¯\_(ツ)_/¯')

    View full-size slide

  47. Серебряная пуля?

    View full-size slide

  48. Не совсем…

    View full-size slide

  49. Что может пойти не так?
    1 def make_dict(a,b,c):
    2 return {
    3 'a': a,
    4 'b': b if a>1 else 0,
    5 'c': [
    6 i for i in range(c) if i<(a*10)
    7 ]
    6 }

    View full-size slide

  50. Мутационное тестирование
    ✤ Берем тестируемый код
    ✤ Мутируем
    ✤ Тестируем мутантов нашими тестами
    ✤ Тест не упал -> плохой тест

    View full-size slide

  51. Мутационное тестирование
    ✤ Берем тестируемый код
    ✤ Мутируем
    ✤ Тестируем мутантов нашими тестами
    ✤ Если тест не упал -> это плохой тест
    ✤ Тест не упал -> плохой тест

    View full-size slide

  52. Идея
    def mul(a, b):
    return a * b
    def test_mul():
    assert mul(2, 2) == 4

    View full-size slide

  53. Идея
    def mul(a, b):
    return a * b
    def test_mul():
    assert mul(2, 2) == 4
    def mul(a, b):
    return a ** b

    View full-size slide

  54. Идея
    def mul(a, b):
    return a * b
    def test_mul():
    assert mul(2, 2) == 4
    def mul(a, b):
    return a + b
    def mul(a, b):
    return a ** b

    View full-size slide

  55. Идея
    def mul(a, b):
    return a * b
    def test_mul():
    assert mul(2, 2) == 4
    assert mul(2, 3) == 6
    def mul(a, b):
    return a + b
    def mul(a, b):
    return a ** b

    View full-size slide

  56. Tools
    MutPy
    ✤ Проект заброшен
    cosmic-ray
    ✤ Активно развивается
    ✤ Требует RabbitMQ

    View full-size slide

  57. Реализация
    Source
    NodeTransformer
    compile
    run test

    View full-size slide

  58. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] / 0.25
    8

    View full-size slide

  59. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    9 return result['TotalPrice'] + result.get(‘Discount’, 0)

    View full-size slide

  60. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    2 if (not len(cart_prices) == 0):
    3 return 0

    View full-size slide

  61. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    2 if len(cart_prices) == 1:
    3 return 0

    View full-size slide

  62. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    2 if len(cart_prices) == 0:
    3 return 1

    View full-size slide

  63. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    5 result = {'': sum(cart_prices)}

    View full-size slide

  64. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    9 return result[‘some_key'] - result.get(‘Discount’, 0)

    View full-size slide

  65. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    View full-size slide

  66. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price(self):
    self.assertEqual(get_total_price([90, 10]), 75)
    self.assertEqual(get_total_price( []), 0)
    self.assertEqual(get_total_price([90]), 90)
    [*] Mutation score [0.50795 s]: 96.4%
    - all: 28
    - killed: 27 (96.4%)
    - survived: 1 (3.6%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)

    View full-size slide

  67. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price(self):
    self.assertEqual(get_total_price([90, 10]), 75)
    self.assertEqual(get_total_price( []), 0)
    self.assertEqual(get_total_price([90]), 90)
    [*] Mutation score [0.50795 s]: 96.4%
    - all: 28
    - killed: 27 (96.4%)
    - survived: 1 (3.6%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)
    - survived: 1 (3.6%)

    View full-size slide


  68. ----------------------------------------------------------
    1: def get_total_price(cart_prices):
    2: if len(cart_prices) == 0:
    ~3: pass
    4:
    5: result = {'TotalPrice': sum(cart_prices)}
    6: if len(cart_prices) >= 2:
    7: result['Discount'] = result['TotalPrice'] * 0.25
    8:
    ----------------------------------------------------------
    [0.00968 s] survived
    - [# 26] SDL target:5 :

    [*] Mutation score [0.50795 s]: 96.4%
    - all: 28
    - killed: 27 (96.4%)
    - survived: 1 (3.6%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)

    View full-size slide

  69. 1 def get_total_price(cart_prices):
    2 result = {'TotalPrice': sum(cart_prices)}
    3 if len(cart_prices) >= 2:
    4 result['Discount'] = result['TotalPrice'] * 0.25
    5
    6 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price(self):
    self.assertEqual(get_total_price([90, 10]), 75)
    self.assertEqual(get_total_price( []), 0)
    self.assertEqual(get_total_price([90]), 90)
    [*] Mutation score [0.44658 s]: 100.0%
    - all: 23
    - killed: 23 (100.0%)
    - survived: 0 (0.0%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)

    View full-size slide

  70. Идея имеет право на жизнь и работает!
    Но требует много ресурсов.

    View full-size slide

  71. 1 def get_total_price(cart_prices):
    2 result = {'TotalPrice': sum(cart_prices)}
    3 if len(cart_prices) >= 2:
    4 result['Discount'] = result['TotalPrice'] * 0.25
    5
    6 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    assert get_total_price([90]) == 90
    Name Stmts Miss Cover Missing
    --------------------------------------------
    target.py 5 0 100.00%

    View full-size slide

  72. 1 def get_total_price(cart_prices):
    2 result = {'TotalPrice': sum(cart_prices)}
    3 if len(cart_prices) >= 2:
    4 result['Discount'] = result['TotalPrice'] * 0.25
    5
    6 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    assert get_total_price([90]) == 90
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 5 0 2 0 100.00%

    View full-size slide

  73. 1 def get_total_price(cart_prices):
    2 result = {'TotalPrice': sum(cart_prices)}
    3 if len(cart_prices) >= 2:
    4 result['Discount'] = result['TotalPrice'] * 0.25
    5
    6 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    assert get_total_price([90]) == 90
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 5 0 2 0 100.00%

    View full-size slide

  74. Есть тесты != код протестирован

    View full-size slide

  75. Есть тесты != код протестирован
    Качество тестов важнее количества

    View full-size slide

  76. Есть тесты != код протестирован
    Качество тестов важнее количества
    100% coverage - не повод расслабляться

    View full-size slide

  77. Simple app
    app = Flask(__name__)
    @app.route('/get_total_discount', methods=['POST'])
    def get_total_discount():
    cart_prices = json.loads(request.form['cart_prices'])
    result = {'TotalPrice': sum(cart_prices)}
    if len(cart_prices) >= 2:
    result['Discount'] = result['TotalPrice'] * 0.25
    return jsonify(result['TotalPrice'] - result.get('Discount', 0))
    flask_app.py

    View full-size slide

  78. pip install pytest-flask
    @pytest.fixture
    def app():
    from flask_app import app
    return app
    def test_get_total_discount(client):
    get_total_discount = lambda prices: client.post(
    '/get_total_discount',
    data=dict(cart_prices=json.dumps(prices))
    ).json
    assert get_total_discount([90, 10]) == 75
    assert get_total_discount( []) == 0
    assert get_total_discount([90]) == 90
    test_flask_app.py

    View full-size slide

  79. pip install pytest-flask
    Name Stmts Miss Cover Missing
    -----------------------------------------------
    flask_app.py 9 0 100.00%
    py.test --cov-config=coverage.ini \
    --cov=flask_app \
    test_flask_app.py
    Name Stmts Miss Branch BrPart Cover Missing
    -------------------------------------------------------------
    flask_app.py 9 0 2 0 100.00%
    py.test --cov-config=coverage_branch.ini \
    --cov=flask_app \
    test_flask_app.py

    View full-size slide

  80. mutpy
    class FlaskTestCase(unittest.TestCase):
    def setUp(self):
    self.app = flask_app.app.test_client()
    def post(self, path, data):
    return json.loads(self.app.post(path, data=data).data.decode('utf-8'))
    def test_get_total_discount(self):
    get_total_discount = lambda prices: self.post(
    '/get_total_discount',
    data=dict(cart_prices=json.dumps(prices))
    )
    self.assertEqual(get_total_discount([90, 10]), 75)
    unittest_flask_app.py

    View full-size slide

  81. mutpy
    [*] Mutation score [0.39122 s]: 100.0%
    - all: 27
    - killed: 1 (3.7%)
    - survived: 0 (0.0%)
    - incompetent: 26 (96.3%)
    - timeout: 0 (0.0%)
    mut.py --target flask_app --unit-test unittest_flask_app

    View full-size slide

  82. mutpy
    [*] Mutation score [0.39122 s]: 100.0%
    - all: 27
    - killed: 1 (3.7%)
    - survived: 0 (0.0%)
    - incompetent: 26 (96.3%)
    - timeout: 0 (0.0%)
    mut.py --target flask_app --unit-test unittest_flask_app

    View full-size slide

  83. mutpy
    def _matching_loader_thinks_module_is_package(loader, mod_name):
    #...
    raise AttributeError(
    ('%s.is_package() method is missing but is required by Flask of '
    'PEP 302 import hooks. If you do not use import hooks and '
    'you encounter this error please file a bug against Flask.') %
    loader.__class__.__name__)

    View full-size slide

  84. mutpy
    def _matching_loader_thinks_module_is_package(loader, mod_name):
    #...
    raise AttributeError(
    ('%s.is_package() method is missing but is required by Flask of '
    'PEP 302 import hooks. If you do not use import hooks and '
    'you encounter this error please file a bug against Flask.') %
    loader.__class__.__name__)
    class InjectImporter:
    def __init__(self, module):
    # ...
    def find_module(self, fullname, path=None):
    # ...
    def load_module(self, fullname):
    # ...
    def install(self):
    # ...
    def uninstall(cls):
    # ...

    View full-size slide

  85. mutpy
    class InjectImporter:
    def __init__(self, module):
    # ...
    def find_module(self, fullname, path=None):
    # ...
    def load_module(self, fullname):
    # ...
    def install(self):
    # ...
    def uninstall(cls):
    # …
    def is_package(self, fullname):
    # ...

    View full-size slide

  86. mutpy
    [*] Mutation score [1.14206 s]: 100.0%
    - all: 27
    - killed: 25 (92.6%)
    - survived: 0 (0.0%)
    - incompetent: 2 (7.4%)
    - timeout: 0 (0.0%)
    mut.py --target flask_app --unit-test unittest_flask_app

    View full-size slide

  87. Simple app
    import json
    from django.http import HttpResponse
    def index(request):
    cart_prices = json.loads(request.POST['cart_prices'])
    result = {'TotalPrice': sum(cart_prices)}
    if len(cart_prices) >= 2:
    result['Discount'] = result['TotalPrice'] * 0.25
    return HttpResponse(result['TotalPrice'] - result.get('Discount', 0))
    django_root/billing/views.py

    View full-size slide

  88. pip install pytest-django
    class TestCase1(TestCase):
    def test_get_total_price(self):
    get_total_price = lambda items: json.loads(
    self.client.post(
    '/billing/', data={'cart_prices': json.dumps(items)}
    ).content.decode('utf-8')
    )
    self.assertEqual(get_total_price([90, 10]), 75)
    self.assertEqual(get_total_price( []), 0)
    self.assertEqual(get_total_price([90]), 90)
    django_root/billing/tests.py

    View full-size slide

  89. pip install pytest-django
    Name Stmts Miss Cover Missing
    ---------------------------------------------------
    billing/views.py 8 0 100.00%
    py.test --cov-config=coverage.ini \
    --cov=billing.views \
    billing/tests.py
    Name Stmts Miss Branch BrPart Cover Missing
    -----------------------------------------------------------------
    billing/views.py 8 0 2 0 100.00%
    py.test --cov-config=coverage_branch.ini \
    --cov=billing.views \
    billing/tests.py

    View full-size slide

  90. mutpy
    [*] Start mutation process:
    - targets: billing.views
    - tests: billing.tests
    [*] Tests failed:
    - error in setUpClass (billing.tests.TestCase1) -
    django.core.exceptions.ImproperlyConfigured: Requested setting DATABASES,
    but settings are not configured. You must either define the environment
    variable DJANGO_SETTINGS_MODULE or call settings.configure() before
    accessing settings.
    mut.py --target billing.views --unit-test billing.tests

    View full-size slide

  91. mutpy
    class Command(BaseCommand):
    def handle(self, *args, **options):
    operators_set = operators.standard_operators
    if options['experimental_operators']:
    operators_set |= operators.experimental_operators
    controller = MutationController(
    target_loader=ModulesLoader(options['target'], None),
    test_loader=ModulesLoader(options['unit_test'], None),
    views=[TextView(colored_output=False, show_mutants=True)],
    mutant_generator=FirstOrderMutator(operators_set)
    )
    controller.run()
    django_root/mutate_command/management/commands/mutate.py

    View full-size slide

  92. mutpy
    [*] Mutation score [1.07321 s]: 0.0%
    - all: 22
    - killed: 0 (0.0%)
    - survived: 22 (100.0%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)
    python manage.py mutate \
    --target billing.views
    --unit-test billing.tests

    View full-size slide

  93. mutpy
    class RegexURLPattern(LocaleRegexProvider):
    def __init__(self, regex, callback, default_args=None, name=None):
    LocaleRegexProvider.__init__(self, regex)
    self.callback = callback # the view
    self.default_args = default_args or {}
    self.name = name
    django.urls.resolvers.RegexURLPattern

    View full-size slide

  94. mutpy
    import importlib
    class Command(BaseCommand):
    def hack_django_for_mutate(self):
    def set_cb(self, value):
    self._cb = value
    def get_cb(self):
    module = importlib.import_module(self._cb.__module__)
    return module.__dict__.get(self._cb.__name__)
    import django.urls.resolvers as r
    r.RegexURLPattern.callback = property(callback, set_cb)
    def __init__(self, *args, **kwargs):
    self.hack_django_for_mutate()
    super().__init__(*args, **kwargs)
    def add_arguments(self, parser):
    # ...

    View full-size slide

  95. mutpy
    [*] Mutation score [1.48715 s]: 100.0%
    - all: 22
    - killed: 22 (100.0%)
    - survived: 0 (0.0%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)
    python manage.py mutate \
    --target billing.views
    --unit-test billing.tests

    View full-size slide

  96. Спасибо за внимание! Вопросы?
    mi.0-0.im
    tsyganov-ivan.com

    View full-size slide

  97. Links
    ✤ https://github.com/pytest-dev/pytest
    ✤ https://github.com/pytest-dev/pytest-flask
    ✤ https://github.com/pytest-dev/pytest-django
    ✤ https://bitbucket.org/ned/coveragepy
    ✤ https://github.com/pytest-dev/pytest-cov
    ✤ https://bitbucket.org/khalas/mutpy
    ✤ https://github.com/sixty-north/cosmic-ray

    View full-size slide