Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PyCon Siberia 2016. Не доверяйте тестам!

PyCon Siberia 2016. Не доверяйте тестам!

Каждый программист рано или поздно начинает писать тесты на свой код. В какой-то момент он начинает задумываться о том, насколько его тесты хороши. В своем докладе я расскажу о том, какие инструменты для проверки качества тестов существуют, как они работают и почему они обманывают нас.

Ivan Tsyganov

October 03, 2016
Tweet

More Decks by Ivan Tsyganov

Other Decks in Programming

Transcript

  1. Цыганов Иван
    Positive Technologies
    Не доверяйте тестам!

    View Slide

  2. Обо мне
    ✤ Спикер PyCon Russia 2016,
    PiterPy#2 и PiterPy#3
    ✤ Люблю OpenSource
    ✤ Не умею frontend

    View Slide

  3. ✤ 15 лет практического опыта на рынке ИБ
    ✤ Более 650 сотрудников в 9 странах
    ✤ Каждый год находим более 200 уязвимостей
    нулевого дня
    ✤ Проводим более 200 аудитов безопасности в
    крупнейших компаниях мира ежегодно

    View Slide

  4. View Slide

  5. MaxPatrol
    ✤ Тестирование на проникновение (Pentest)
    ✤ Системные проверки (Audit)
    ✤ Соответствие стандартам (Compliance)
    ✤ Одна из крупнейших баз знаний в мире
    Система контроля защищенности и соответствия
    стандартам.

    View Slide

  6. ✤ Тестирование на проникновение (Pentest)
    ✤ Системные проверки (Audit)
    ✤ Соответствие стандартам (Compliance)
    ✤ Одна из крупнейших баз знаний в мире
    Система контроля защищенности и соответствия
    стандартам.
    ✤ Системные проверки (Audit)
    MaxPatrol

    View Slide

  7. > 50 000 строк кода

    View Slide

  8. Зачем тестировать?
    ✤ Уверенность, что написанный код работает
    ✤ Ревью кода становится проще
    ✤ Гарантия, что ничего не сломалось при
    изменениях

    View Slide

  9. есть тесты != код протестирован

    View Slide

  10. Давайте писать тесты!
    def get_total_price(cart_prices):
    if len(cart_prices) == 0:
    return
    result = {'TotalPrice': sum(cart_prices)}
    if len(cart_prices) >= 2:
    result['Discount'] = result['TotalPrice'] * 0.25
    return result['TotalPrice'] - result.get('Discount')

    View Slide

  11. Плохой тест
    def get_total_price(cart_prices):
    if len(cart_prices) == 0:
    return
    result = {'TotalPrice': sum(cart_prices)}
    if len(cart_prices) >= 2:
    result['Discount'] = result['TotalPrice'] * 0.25
    return result['TotalPrice'] - result.get('Discount')
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75

    View Slide

  12. Неожиданные данные
    >>> balance = 1000
    >>>
    >>> goods = []
    >>>
    >>> balance -= get_total_price(goods)
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: unsupported operand type(s) for -=: 'int' and 'NoneType'
    >>>

    View Slide

  13. есть тесты == есть тесты

    View Slide

  14. Как сделать тесты лучше?
    ✤ Проверить покрытие кода тестами
    ✤ Попробовать мутационное тестирование

    View Slide

  15. coverage.py
    ✤ Позволяет проверить покрытие кода тестами
    ✤ Есть плагин для pytest

    View Slide

  16. coverage.py
    ✤ Позволяет проверить покрытие кода тестами
    ✤ Есть плагин для pytest
    ✤ В основном работает

    View Slide

  17. coverage.ini
    [report]

    show_missing = True

    precision = 2
    py.test --cov-config=coverage.ini --cov=target test.py

    View Slide

  18. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    Name Stmts Miss Cover Missing
    --------------------------------------------
    target.py 7 1 85.71% 2

    View Slide

  19. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    Name Stmts Miss Cover Missing
    --------------------------------------------
    target.py 7 1 85.71% 2
    2 if len(cart_prices) == 0:

    View Slide

  20. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Cover Missing
    --------------------------------------------
    target.py 7 0 100.00%

    View Slide

  21. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Cover Missing
    --------------------------------------------
    target.py 7 0 100.00%

    View Slide

  22. >>> get_total_price([90])
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')

    View Slide

  23. >>> get_total_price([90])
    Traceback (most recent call last):
    File "", line 1, in
    File "", line 9, in get_total_price
    TypeError: unsupported operand type(s) for -: 'int' and
    'NoneType'
    >>>
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get('Discount')

    View Slide

  24. View Slide

  25. coverage.ini
    [report]

    show_missing = True

    precision = 2

    [run]

    branch = True
    py.test --cov-config=coverage.ini --cov=target test.py

    View Slide

  26. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 7 0 4 1 90.91% 6 ->9

    View Slide

  27. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 7 0 4 1 90.91% 6 ->9

    View Slide

  28. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    assert get_total_price([90]) == 90
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 7 0 4 0 100.00%

    View Slide

  29. View Slide

  30. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 total_price = sum(cart_prices)
    6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
    7
    8 return total_price-get_discount(cart_prices, total_price)
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    View Slide

  31. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 total_price = sum(cart_prices)
    6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
    7
    8 return total_price-get_discount(cart_prices, total_price)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 6 1 4 1 80.00% 3, 2 ->3

    View Slide

  32. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 total_price = sum(cart_prices)
    6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
    7
    8 return total_price-get_discount(cart_prices, total_price)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 6 0 4 0 100.00%

    View Slide

  33. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 total_price = sum(cart_prices)
    6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
    7
    8 return total_price-get_discount(cart_prices, total_price)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 6 0 4 0 100.00%
    6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25

    View Slide

  34. View Slide

  35. Как считать coverage?
    Все строки
    Реально выполненные
    строки
    - Непокрытые
    строки
    =

    View Slide

  36. Все строки
    Source
    coverage.parser.PythonParser
    Statements

    View Slide

  37. coverage.parser.PythonParser
    ✤ Обходит все токены и отмечает «интересные»
    факты
    ✤ Компилирует код. Обходит code-object и
    сохраняет номера строк

    View Slide

  38. Обход токенов
    ✤ Запоминает определения классов
    ✤ «Сворачивает» многострочные выражения
    ✤ Исключает комментарии

    View Slide

  39. Обход байткода
    ✤ Полностью повторяет метод dis.findlinestarts
    ✤ Анализирует code_obj.co_lnotab
    ✤ Генерирует пару (номер байткода, номер строки)

    View Slide

  40. Как считать coverage --branch?
    Все переходы
    Реально выполненные
    переходы
    - Непокрытые
    переходы
    =

    View Slide

  41. Все переходы
    Source
    coverage.parser.AstArcAnalyzer
    (from_line, to_line)
    coverage.parser.PythonParser

    View Slide

  42. coverage.parser.AstArcAnalyzer
    ✤ Обходит AST с корневой ноды
    ✤ Обрабатывает отдельно каждый тип нод отдельно

    View Slide

  43. Обработка ноды
    class While(stmt):
    _fields = (
    'test',
    'body',
    'orelse',
    )
    while i<10:
    print(i)
    i += 1

    View Slide

  44. Обработка ноды
    class While(stmt):
    _fields = (
    'test',
    'body',
    'orelse',
    )
    while i<10:
    print(i)
    i += 1
    else:
    print('All done')

    View Slide

  45. Выполненные строки
    sys.settrace(tracefunc)
    Set the system’s trace function, which allows you to implement a
    Python source code debugger in Python.
    Trace functions should have three arguments: frame, event, and
    arg. frame is the current stack frame. event is a string: 'call',
    'line', 'return', 'exception', 'c_call', 'c_return', or
    'c_exception'. arg depends on the event type.

    View Slide

  46. PyTracer «call» event
    ✤ Сохраняем данные предыдущего контекста
    ✤ Начинаем собирать данные нового контекста
    ✤ Учитываем особенности генераторов

    View Slide

  47. PyTracer «line» event
    ✤ Запоминаем выполняемую строку
    ✤ Запоминаем переход между строками

    View Slide

  48. PyTracer «return» event
    ✤ Отмечаем выход из контекста
    ✤ Помним о том, что yield это тоже return

    View Slide

  49. Отчет
    ✤ Что выполнялось
    ✤ Что должно было выполниться
    ✤ Ругаемся

    View Slide

  50. Зачем такие сложности?
    1 for i in some_list:
    2 if i == 'Hello':
    3 print(i + ' World!')
    4 elif i == 'Skip':
    5 continue
    6 else:
    7 break
    8 else:
    9 print(r'¯\_(ツ)_/¯')

    View Slide

  51. Серебряная пуля?

    View Slide

  52. Не совсем…

    View Slide

  53. Что может пойти не так?
    1 def make_dict(a,b,c):
    2 return {
    3 'a': a,
    4 'b': b if a>1 else 0,
    5 'c': [
    6 i for i in range(c) if i7 ]
    6 }

    View Slide

  54. View Slide

  55. Мутационное тестирование
    ✤ Берем тестируемый код
    ✤ Мутируем
    ✤ Тестируем мутантов нашими тестами
    ✤ Тест не упал -> плохой тест

    View Slide

  56. Мутационное тестирование
    ✤ Берем тестируемый код
    ✤ Мутируем
    ✤ Тестируем мутантов нашими тестами
    ✤ Если тест не упал -> это плохой тест
    ✤ Тест не упал -> плохой тест

    View Slide

  57. Идея
    def mul(a, b):
    return a * b
    def test_mul():
    assert mul(2, 2) == 4

    View Slide

  58. Идея
    def mul(a, b):
    return a * b
    def test_mul():
    assert mul(2, 2) == 4
    def mul(a, b):
    return a ** b

    View Slide

  59. Идея
    def mul(a, b):
    return a * b
    def test_mul():
    assert mul(2, 2) == 4
    def mul(a, b):
    return a + b
    def mul(a, b):
    return a ** b

    View Slide

  60. Идея
    def mul(a, b):
    return a * b
    def test_mul():
    assert mul(2, 2) == 4
    assert mul(2, 3) == 6
    def mul(a, b):
    return a + b
    def mul(a, b):
    return a ** b

    View Slide

  61. Tools
    MutPy
    ✤ Проект заброшен
    cosmic-ray
    ✤ Активно развивается
    ✤ Требует RabbitMQ

    View Slide

  62. Реализация
    Source
    NodeTransformer
    compile
    run test

    View Slide

  63. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] / 0.25
    8

    View Slide

  64. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    9 return result['TotalPrice'] + result.get(‘Discount’, 0)

    View Slide

  65. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    2 if (not len(cart_prices) == 0):
    3 return 0

    View Slide

  66. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    2 if len(cart_prices) == 1:
    3 return 0

    View Slide

  67. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    2 if len(cart_prices) == 0:
    3 return 1

    View Slide

  68. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    5 result = {'': sum(cart_prices)}

    View Slide

  69. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    9 return result[‘some_key'] - result.get(‘Discount’, 0)

    View Slide

  70. Мутации
    1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)

    View Slide

  71. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price(self):
    self.assertEqual(get_total_price([90, 10]), 75)
    self.assertEqual(get_total_price( []), 0)
    self.assertEqual(get_total_price([90]), 90)
    [*] Mutation score [0.50795 s]: 96.4%
    - all: 28
    - killed: 27 (96.4%)
    - survived: 1 (3.6%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)

    View Slide

  72. 1 def get_total_price(cart_prices):
    2 if len(cart_prices) == 0:
    3 return 0
    4
    5 result = {'TotalPrice': sum(cart_prices)}
    6 if len(cart_prices) >= 2:
    7 result['Discount'] = result['TotalPrice'] * 0.25
    8
    9 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price(self):
    self.assertEqual(get_total_price([90, 10]), 75)
    self.assertEqual(get_total_price( []), 0)
    self.assertEqual(get_total_price([90]), 90)
    [*] Mutation score [0.50795 s]: 96.4%
    - all: 28
    - killed: 27 (96.4%)
    - survived: 1 (3.6%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)
    - survived: 1 (3.6%)

    View Slide


  73. ----------------------------------------------------------
    1: def get_total_price(cart_prices):
    2: if len(cart_prices) == 0:
    ~3: pass
    4:
    5: result = {'TotalPrice': sum(cart_prices)}
    6: if len(cart_prices) >= 2:
    7: result['Discount'] = result['TotalPrice'] * 0.25
    8:
    ----------------------------------------------------------
    [0.00968 s] survived
    - [# 26] SDL target:5 :

    [*] Mutation score [0.50795 s]: 96.4%
    - all: 28
    - killed: 27 (96.4%)
    - survived: 1 (3.6%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)

    View Slide

  74. 1 def get_total_price(cart_prices):
    2 result = {'TotalPrice': sum(cart_prices)}
    3 if len(cart_prices) >= 2:
    4 result['Discount'] = result['TotalPrice'] * 0.25
    5
    6 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price(self):
    self.assertEqual(get_total_price([90, 10]), 75)
    self.assertEqual(get_total_price( []), 0)
    self.assertEqual(get_total_price([90]), 90)
    [*] Mutation score [0.44658 s]: 100.0%
    - all: 23
    - killed: 23 (100.0%)
    - survived: 0 (0.0%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)

    View Slide

  75. Идея имеет право на жизнь и работает!
    Но требует много ресурсов.

    View Slide

  76. 1 def get_total_price(cart_prices):
    2 result = {'TotalPrice': sum(cart_prices)}
    3 if len(cart_prices) >= 2:
    4 result['Discount'] = result['TotalPrice'] * 0.25
    5
    6 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    assert get_total_price([90]) == 90
    Name Stmts Miss Cover Missing
    --------------------------------------------
    target.py 5 0 100.00%

    View Slide

  77. 1 def get_total_price(cart_prices):
    2 result = {'TotalPrice': sum(cart_prices)}
    3 if len(cart_prices) >= 2:
    4 result['Discount'] = result['TotalPrice'] * 0.25
    5
    6 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    assert get_total_price([90]) == 90
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 5 0 2 0 100.00%

    View Slide

  78. 1 def get_total_price(cart_prices):
    2 result = {'TotalPrice': sum(cart_prices)}
    3 if len(cart_prices) >= 2:
    4 result['Discount'] = result['TotalPrice'] * 0.25
    5
    6 return result['TotalPrice'] - result.get(‘Discount’, 0)
    def test_get_total_price():
    assert get_total_price([90, 10]) == 75
    assert get_total_price( []) == 0
    assert get_total_price([90]) == 90
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------
    target.py 5 0 2 0 100.00%

    View Slide

  79. Есть тесты != код протестирован

    View Slide

  80. Есть тесты != код протестирован
    Качество тестов важнее количества

    View Slide

  81. Есть тесты != код протестирован
    Качество тестов важнее количества
    100% coverage - не повод расслабляться

    View Slide

  82. View Slide

  83. View Slide

  84. Simple app
    app = Flask(__name__)
    @app.route('/get_total_discount', methods=['POST'])
    def get_total_discount():
    cart_prices = json.loads(request.form['cart_prices'])
    result = {'TotalPrice': sum(cart_prices)}
    if len(cart_prices) >= 2:
    result['Discount'] = result['TotalPrice'] * 0.25
    return jsonify(result['TotalPrice'] - result.get('Discount', 0))
    flask_app.py

    View Slide

  85. pip install pytest-flask
    @pytest.fixture
    def app():
    from flask_app import app
    return app
    def test_get_total_discount(client):
    get_total_discount = lambda prices: client.post(
    '/get_total_discount',
    data=dict(cart_prices=json.dumps(prices))
    ).json
    assert get_total_discount([90, 10]) == 75
    assert get_total_discount( []) == 0
    assert get_total_discount([90]) == 90
    test_flask_app.py

    View Slide

  86. pip install pytest-flask
    Name Stmts Miss Cover Missing
    -----------------------------------------------
    flask_app.py 9 0 100.00%
    py.test --cov-config=coverage.ini \
    --cov=flask_app \
    test_flask_app.py
    Name Stmts Miss Branch BrPart Cover Missing
    -------------------------------------------------------------
    flask_app.py 9 0 2 0 100.00%
    py.test --cov-config=coverage_branch.ini \
    --cov=flask_app \
    test_flask_app.py

    View Slide

  87. mutpy
    class FlaskTestCase(unittest.TestCase):
    def setUp(self):
    self.app = flask_app.app.test_client()
    def post(self, path, data):
    return json.loads(self.app.post(path, data=data).data.decode('utf-8'))
    def test_get_total_discount(self):
    get_total_discount = lambda prices: self.post(
    '/get_total_discount',
    data=dict(cart_prices=json.dumps(prices))
    )
    self.assertEqual(get_total_discount([90, 10]), 75)
    unittest_flask_app.py

    View Slide

  88. mutpy
    [*] Mutation score [0.39122 s]: 100.0%
    - all: 27
    - killed: 1 (3.7%)
    - survived: 0 (0.0%)
    - incompetent: 26 (96.3%)
    - timeout: 0 (0.0%)
    mut.py --target flask_app --unit-test unittest_flask_app

    View Slide

  89. mutpy
    [*] Mutation score [0.39122 s]: 100.0%
    - all: 27
    - killed: 1 (3.7%)
    - survived: 0 (0.0%)
    - incompetent: 26 (96.3%)
    - timeout: 0 (0.0%)
    mut.py --target flask_app --unit-test unittest_flask_app

    View Slide

  90. mutpy
    def _matching_loader_thinks_module_is_package(loader, mod_name):
    #...
    raise AttributeError(
    ('%s.is_package() method is missing but is required by Flask of '
    'PEP 302 import hooks. If you do not use import hooks and '
    'you encounter this error please file a bug against Flask.') %
    loader.__class__.__name__)

    View Slide

  91. mutpy
    def _matching_loader_thinks_module_is_package(loader, mod_name):
    #...
    raise AttributeError(
    ('%s.is_package() method is missing but is required by Flask of '
    'PEP 302 import hooks. If you do not use import hooks and '
    'you encounter this error please file a bug against Flask.') %
    loader.__class__.__name__)
    class InjectImporter:
    def __init__(self, module):
    # ...
    def find_module(self, fullname, path=None):
    # ...
    def load_module(self, fullname):
    # ...
    def install(self):
    # ...
    def uninstall(cls):
    # ...

    View Slide

  92. mutpy
    class InjectImporter:
    def __init__(self, module):
    # ...
    def find_module(self, fullname, path=None):
    # ...
    def load_module(self, fullname):
    # ...
    def install(self):
    # ...
    def uninstall(cls):
    # …
    def is_package(self, fullname):
    # ...

    View Slide

  93. mutpy
    [*] Mutation score [1.14206 s]: 100.0%
    - all: 27
    - killed: 25 (92.6%)
    - survived: 0 (0.0%)
    - incompetent: 2 (7.4%)
    - timeout: 0 (0.0%)
    mut.py --target flask_app --unit-test unittest_flask_app

    View Slide

  94. View Slide

  95. Simple app
    import json
    from django.http import HttpResponse
    def index(request):
    cart_prices = json.loads(request.POST['cart_prices'])
    result = {'TotalPrice': sum(cart_prices)}
    if len(cart_prices) >= 2:
    result['Discount'] = result['TotalPrice'] * 0.25
    return HttpResponse(result['TotalPrice'] - result.get('Discount', 0))
    django_root/billing/views.py

    View Slide

  96. pip install pytest-django
    class TestCase1(TestCase):
    def test_get_total_price(self):
    get_total_price = lambda items: json.loads(
    self.client.post(
    '/billing/', data={'cart_prices': json.dumps(items)}
    ).content.decode('utf-8')
    )
    self.assertEqual(get_total_price([90, 10]), 75)
    self.assertEqual(get_total_price( []), 0)
    self.assertEqual(get_total_price([90]), 90)
    django_root/billing/tests.py

    View Slide

  97. pip install pytest-django
    Name Stmts Miss Cover Missing
    ---------------------------------------------------
    billing/views.py 8 0 100.00%
    py.test --cov-config=coverage.ini \
    --cov=billing.views \
    billing/tests.py
    Name Stmts Miss Branch BrPart Cover Missing
    -----------------------------------------------------------------
    billing/views.py 8 0 2 0 100.00%
    py.test --cov-config=coverage_branch.ini \
    --cov=billing.views \
    billing/tests.py

    View Slide

  98. mutpy
    [*] Start mutation process:
    - targets: billing.views
    - tests: billing.tests
    [*] Tests failed:
    - error in setUpClass (billing.tests.TestCase1) -
    django.core.exceptions.ImproperlyConfigured: Requested setting DATABASES,
    but settings are not configured. You must either define the environment
    variable DJANGO_SETTINGS_MODULE or call settings.configure() before
    accessing settings.
    mut.py --target billing.views --unit-test billing.tests

    View Slide

  99. mutpy
    class Command(BaseCommand):
    def handle(self, *args, **options):
    operators_set = operators.standard_operators
    if options['experimental_operators']:
    operators_set |= operators.experimental_operators
    controller = MutationController(
    target_loader=ModulesLoader(options['target'], None),
    test_loader=ModulesLoader(options['unit_test'], None),
    views=[TextView(colored_output=False, show_mutants=True)],
    mutant_generator=FirstOrderMutator(operators_set)
    )
    controller.run()
    django_root/mutate_command/management/commands/mutate.py

    View Slide

  100. mutpy
    [*] Mutation score [1.07321 s]: 0.0%
    - all: 22
    - killed: 0 (0.0%)
    - survived: 22 (100.0%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)
    python manage.py mutate \
    --target billing.views
    --unit-test billing.tests

    View Slide

  101. mutpy
    class RegexURLPattern(LocaleRegexProvider):
    def __init__(self, regex, callback, default_args=None, name=None):
    LocaleRegexProvider.__init__(self, regex)
    self.callback = callback # the view
    self.default_args = default_args or {}
    self.name = name
    django.urls.resolvers.RegexURLPattern

    View Slide

  102. mutpy
    import importlib
    class Command(BaseCommand):
    def hack_django_for_mutate(self):
    def set_cb(self, value):
    self._cb = value
    def get_cb(self):
    module = importlib.import_module(self._cb.__module__)
    return module.__dict__.get(self._cb.__name__)
    import django.urls.resolvers as r
    r.RegexURLPattern.callback = property(callback, set_cb)
    def __init__(self, *args, **kwargs):
    self.hack_django_for_mutate()
    super().__init__(*args, **kwargs)
    def add_arguments(self, parser):
    # ...

    View Slide

  103. mutpy
    [*] Mutation score [1.48715 s]: 100.0%
    - all: 22
    - killed: 22 (100.0%)
    - survived: 0 (0.0%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)
    python manage.py mutate \
    --target billing.views
    --unit-test billing.tests

    View Slide

  104. Спасибо за внимание! Вопросы?
    mi.0-0.im
    tsyganov-ivan.com

    View Slide

  105. Links
    ✤ https://github.com/pytest-dev/pytest
    ✤ https://github.com/pytest-dev/pytest-flask
    ✤ https://github.com/pytest-dev/pytest-django
    ✤ https://bitbucket.org/ned/coveragepy
    ✤ https://github.com/pytest-dev/pytest-cov
    ✤ https://bitbucket.org/khalas/mutpy
    ✤ https://github.com/sixty-north/cosmic-ray

    View Slide