Testing to reduce support calls

For years we have been talking about testing to improve the quality of applications, but there are still many support calls coming in and bug reports submitted. Is it because our code is faulty and our tests are not covering the right components?

The answer lies in the perception of users and customers of the product which differs significantly from what developers and managers believe the user wants or needs. Welcome to user experience testing! In this talk I'll highlight some of the steps we took to address the ever growing user dissatisfaction of the software we thought was awesome and well tested.

Transcript

1. TESTING TO REDUCE SUPPORT CALLS
   Michelangelo van Dam
   

   View Slide

2. MICHELANGELO VAN DAM
   I'm a senior #php architect, co-founder and #ceo of @in2itvof, #community leader
   at @phpbenelux, coach at @CoderDojoBelgium, #MVP, #digitalnomad,
   likes #coffee.
   Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
   

   View Slide

3. I’M ACTIVELY INVOLVED IN THE COMMUNITY …
   DevOps

   Community
   Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
   

   View Slide

4. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
   

   View Slide

5. To be successful at testing you need to
   be able to look beyond the walls of the
   imaginary room!
   Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
   

   View Slide

6. WHY TESTING?
   ➤ Ensure business/regulatory requirements are met
   ➤ Prevent (serious) issues when used by customers
   ➤ Protect against loss of revenue or reputational damage
   ➤ Gain competitive advantage
   ➤ Receive trust and respect
   Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
   

   View Slide

7. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
   

   View Slide

8. HOW DO I START?
   Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
   

   View Slide

9. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
   

   View Slide

10. Even though F5 testing is easy to
    do, please adopt a more robust form of
    testing that can be shared with your
    team members
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

11. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

12. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

13. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

14. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

15. Be mindful that automated
    acceptance tests are sensitive for
    changes in layout of your application!
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

16. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

17. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

18. Feature: Meetup group has events
    In order to participate in an event
    As a participant
    A meetup group must have public events
    Scenario: Search for events of a user group
    Given the meetup group "GroningenPHP" exists
    When I search for meetup group "GroningenPHP"
    Then I should see at least "1" event listed
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

19. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

20. Feature: Meetup group has events
    In order to participate in an event
    As a participant
    A meetup group must have public events
    Scenario: Search for events of a user group
    Given the meetup group "GroningenFoo" exists
    When I search for meetup group "GroningenFoo"
    Then I should see at least "1" event listed
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

21. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

22. Behat tests that business
    requirements are met,
    not that correct logic is implemented!
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

23. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

24. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

25. LEGACY APPLICATIONS
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

26. If you measure code coverage,
    ensure that the core of your application
    is 100% covered with tests, even though
    it only is 1% of the whole code
    base!
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

27. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

28. USER EXPERIENCE TESTING
    Keep in touch with your users/customers
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

29. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

30. UNDERSTAND THE
    “BUSINESS VALUE”
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

31. WRITE OUT THE REQUIREMENTS
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

32. DRAW OR USE WIREFRAMES
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

33. START WITH A POC
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

34. CODE IN PROD FROM DAY 0
    Use feature toggles to perform A/B testing immediately
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

35. GIVE USERS A WAY TO PROVIDE
    FEEDBACK EASILY
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

36. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

37. KEEP DEVELOPMENT CYCLES
    SHORT AND AGILE
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

38. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    Agile Lean Kanban
    

    View Slide

39. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

40. PROTIP: KEEP TRACK OF…
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

41. OTHER FORMS OF
    TESTING
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

42. PERFORMANCE TESTING
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

43. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

44. SECURITY TESTING
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

45. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    Sqlmap®
    

    View Slide

46. RESILIENCE TESTING
    Chaos Engineering
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

47. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

48. MANUAL TESTING
    Not all things can be automated
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

49. EXPLORATORY TESTING
    Looking at the application from every angle to discover flaws in
    functionality, design or behaviour.
    A.K.A. BlackBox Testing
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

50. PENETRATION TESTING
    Finding and exploiting weaknesses, flaws or poor configurations
    in your application to gain privileged access or syphon data out
    of your business.
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

51. LEGAL TESTING & AUDITING
    In regulatory industries, legal testing and auditing is a must
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

52. Use the right tool for your testing
    requirements. Nothing worse than to
    waste time configuring a testing tool
    that provides no value!
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

53. AUTOMATED TESTING?
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

54. AUTOMATE WHAT’S POSSIBLE
    ➤ If it’s repetitive, automate it
    ➤ Prevents someone’s “bad day” to cause issues
    ➤ Helps when under stress or pressure
    ➤ Provides documentation and insights to everyone
    ➤ Can be fine-tuned over time
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

55. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

56. If I have to do the same thing more than
    once, I automate it!
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

57. WILL IT PREVENT FAILURE?
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

58. Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

59. WHEN FAILURE DOES HAPPEN…
    FAILURE
    RESOLUTION
    REVIEW
    REMEDIATION
    ANALYSIS
    BEST PRACTICES
    AUTOMATION
    ADOPTION
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    Credits: Josh Evans, Mastering Chaos at Netflix
    

    View Slide

60. WILL THIS REDUCE
    SUPPORT CALLS?
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

61. LET’S SUMMARISE
    ➤ We test functional requirements with Selenium
    ➤ We test business requirements with Behat
    ➤ We test code requirements with unit and mutation tests
    ➤ We let our target audience test our work early
    ➤ We test performance, security and resilience of our application
    ➤ We run all this in a continuous integration tool
    ➤ And we have regular manual exploratory and penetration tests
    ➤ We understand failures can still occur, but we address them as a
    team and adopt a culture of continuous learning & automation
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

62. Bring tests closer to the point
    where code is created and you
    already win time and money!
    SHIFT LEFT
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

63. WILL THIS REDUCE
    SUPPORT CALLS?
    YES
    Between 40% and 65% less
    support calls or bug reports*
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    (*) Review of in2it customers in 2018
    

    View Slide

64. RESOURCES 1/3
    ➤ PHP User Groups: php.ug
    ➤ Ministry of Testing:
    ministryoftesting.com
    ➤ Belgium DevOps Community:
    meetup.com/Devops-Belgium
    ➤ Belgium Cyber Security Coalition:
    cybersecuritycoalition.be
    ➤ Azure User Group Belgium: azug.be
    ➤ Global Open Source Initiative:
    opensource.org
    ➤ Selenium: seleniumhq.org
    ➤ Katalon Studio: katalon.com
    ➤ SauceLabs: saucelabs.com
    ➤ Behat: behat.org
    ➤ PHPUnit: phpunit.de
    ➤ Infection: infection.github.io
    ➤ Usabilla: usabilla.com
    ➤ Apache Bench: httpd.apache.org
    ➤ Apache JMeter: jmeter.apache.org
    ➤ Joe Dog Siege: joedog.org/siege-home
    ➤ MySQL Slap: dev.mysql.com
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

65. RESOURCES 2/3
    ➤ NeoLoad: neotys.com/neoload
    ➤ LoadNinja: loadninja.com
    ➤ OWASP Zed Attack Proxy: owasp.org
    ➤ Vega Vulnerability Scanner: subgraph.com/
    vega
    ➤ Burp Suite: portswigger.net/burp
    ➤ SonarQube: sonarqube.org
    ➤ RIPS Tech: ripstech.com
    ➤ SQLMap: sqlmap.org
    ➤ Arachni: arachni-scanner.com
    ➤ Netflix Chaos Monkey:
    techblog.netflix.com
    ➤ Jenkins: jenkins.io
    ➤ JetBrains TeamCity: jetbrains.com/
    teamcity
    ➤ Atlassian Bamboo: atlassian.com/
    software/bamboo
    ➤ GitHub: github.com
    ➤ GitLab: gitlab.com
    ➤ Atlassian BitBucket: bitbucket.org
    ➤ Phabricator: phacility.com/phabricator
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

66. RESOURCES 3/3
    ➤ Microsoft Azure: azure.microsoft.com
    ➤ Amazon Web Services:
    aws.amazon.com
    ➤ Google Cloud Platform:
    cloud.google.com
    ➤ Travis CI: travis-ci.org
    ➤ CodeShip: codeship.com
    ➤ Semaphore: semaphoreci.com
    ➤ Buddy: buddy.works
    ➤ Go CD: gocd.org
    ➤ NeverCode: nevercode.io
    ➤ Strider CD: strider-cd.github.io
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide

67. Improving quality one test at a $me
    Slides available at
    in2.se/reduce-support-calls
    Tes$ng to reduce support calls by Michelangelo van Dam (@DragonBe)
    

    View Slide