Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to Break Software: The art of continuous testing on Azure.

How to Break Software: The art of continuous testing on Azure.

Presented by Suhas Roa at the Global Azure Bootcamp 2019 in Sydney.

Are you curious to know why Microsoft merged its SDE and SDET positions into the software engineer title? As a developer how confident are you about writing code without any bugs? Join me for session on how Azure and DevOps is changing the way testing is done. Understand how quality unit tests to automated functional tests, all integrated with your CI/CD pipelines can help you ship your bug-free code faster!

Azure Sydney User Group

April 27, 2019
Tweet

More Decks by Azure Sydney User Group

Other Decks in Technology

Transcript

  1. Suhas Rao
    Cloud Solution Architect App Dev
    Microsoft
    @suhasaraos
    suhasaraos
    How to break software:
    The art of Continuous Testing
    Images are copyrighted to their respective creators

    View full-size slide

  2. Books that inspired me…

    View full-size slide

  3. Whose responsibility is software testing?
    A) Developers
    B) Business Analysts/Product Owners/Stakeholders
    C) Quality Analysts (QA)/Test Engineers
    D) End Users

    View full-size slide

  4. Pace of Business today
    “We are implementing features and products and
    using technology that were not invented 18 months
    ago. No longer can we afford these large monolithic
    programs that go on for two to three years. We know
    that what we set out to do at the beginning of that
    time is not what we will finish out doing. So, we are
    focusing on very rapid delivery cycles, asking
    ourselves: How do we mobilize a project very quickly?
    How do we use the right delivery techniques to work
    through it quickly? How do we get product into
    market or to customers or into the business?”
    - Bronwyn Clere, Executive Director for Capital
    Planning & Delivery, Telstra Corporation

    View full-size slide

  5. Every company is a software company
    “Every company is a software company.
    You have to start thinking and operating
    like a digital company. It’s no longer just
    about procuring one solution and
    deploying one. It’s not about one simple
    software solution. It’s really you yourself
    thinking of your own future as a digital
    company.”
    Satya Nadella, CEO of Microsoft
    Source: https://www.skyhighnetworks.com/cloud-security-blog/every-company-is-
    a-software-company-today/

    View full-size slide

  6. Complexity of Software
    • Let’s play a game.
    2 coins = {HH, HT, TH, TT}
    1 coins = {H, T}
    3 coins = {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT}
    How many coins for 1000 arrangements?
    • Software complexity grows in much the
    same way.
    • Making a small addition can be rather
    simple, but can result in an exponential
    increase in complexity on the entire
    program.
    Ex: Suppose you have a settings page
    with five settings, each setting having two
    possible states. That’s 32 possible
    configurations. Adding a sixth setting will
    result in 64 possible setting
    configurations.
    • Its imperative that each one of these
    configurations must be tested to
    ensure everything functions as
    intended.

    View full-size slide

  7. Applications are not in isolation anymore!
    Can you accurately determine the impact of your change –
    to all the applications beyond your own?

    View full-size slide

  8. https://www.tricentis.com/resources/software-fail-watch-5th-edition/

    View full-size slide

  9. Are we getting better at testing?

    View full-size slide

  10. Testing is indeed changing…
    • The responsibility for testing is
    now being shifted leftward,
    closer to the beginning of the
    software development lifecycle
    and then all the way along it.
    • Knocking down the wall that
    has traditionally separated
    developers and testers, making
    quality everyone’s
    responsibility.

    View full-size slide

  11. • 63% of software development delays occur in Test-QA practices
    across the lifecycle.
    • 56% of critical dependencies are unavailable
    • 50% of time is spent looking for test data
    • 64% of defects occur in the requirements phase
    • 70% of testing is still manual.
    Testing is indeed changing…

    View full-size slide

  12. What is continuous testing?
    • Continuous testing is the practice of testing across every activity in
    the SDLC to uncover and fix unexpected behaviors as soon as they are
    injected.
    • Continuous testing provides quality gates throughout the pipeline and
    increases confidence in code long before production.
    • Automated unit, integration, coded UI, and load tests are common in
    continuous testing.
    • Depth of testing often progresses as the environment gets closer to
    production.
    • In practice, continuous testing is the most difficult part of a
    continuous delivery pipeline to keep up to date.

    View full-size slide

  13. School of Continuous Testing
    • Twelve Disciplines
    • Examples based on Microsoft Technologies

    View full-size slide

  14. Requirements Engineering & Acceptance Criteria
    • Include all SDLC stakeholders into the
    continuous testing journey.
    • Must utilize better ways to
    communicate and collaborate on the
    requirements rather than just
    emails/word docs.
    • Take the acceptance criteria for each
    user story and create the tests to
    ensure each of those are met.
    • Ensure test steps to acceptance criteria
    traceability.
    • Define detailed acceptance criteria,
    which requires tests to also be more
    comprehensive.

    View full-size slide

  15. Unit Tests
    • The method of testing
    specific functions and
    areas (or units) of your
    code.
    • Select right tools:
    • Unit test Framework
    • Isolation Framework
    • Code coverage tools
    • Test Explorer

    View full-size slide

  16. On demand Environments
    • More Frequent Testing
    • Creation and teardown
    environments on demand
    • Co-ordinate testing through
    multiple environment owners that
    may not always have environments
    up and running (or required
    versions of it)
    • Use Cloud and Infra as Code to
    avoid bottlenecks.

    View full-size slide

  17. Test & Test Data Management
    • Right set of tests and data
    • Appropriate diversity of data for
    positive and negative scenarios
    • Synthetic data generation is the
    way to go to achieve continuous
    testing with the highest levels of
    confidence.

    View full-size slide

  18. API Testing
    • Enable alignment of the testing
    pyramid.
    • Strive to test as much at the Unit and
    API levels as possible and minimize
    reliance on UI testing
    Test Automation Pyramid

    View full-size slide

  19. Performance/Load Testing
    • Use the power and scale of
    Cloud to generate load.
    • Many free and commercial tools
    available (Like Apache JMeter)
    • Note :
    • Web performance and load test
    functionality in Visual Studio is
    deprecated.
    • Cloud load testing (CLT)
    functionality in Azure DevOps is
    deprecated.

    View full-size slide

  20. Security Testing
    • Application security area is always
    evolving, its important to stay
    updated.
    • Setup security alerts for vulnerable
    dependencies.
    • Setup Pipelines to run static
    analysis, dynamic analysis and
    software composition analysis
    • Security testing needs to happen at
    every step of the lifecycle.
    • Automate as much of Pen testing as
    you can in the release pipelines
    using tools like OWASP ZAP.

    View full-size slide

  21. Security Testing
    https://docs.microsoft.com/en-us/azure/devops/migrate/security-validation-cicd-pipeline?view=azure-devops

    View full-size slide

  22. Test Automation
    • When you think test early
    and test often by using
    automated testing, you
    essentially build a robust
    testing architecture that
    helps produce higher-
    quality software.
    • Follow the Pyramid model
    to Automate UI, Unit and
    API tests.
    • Tools like Selenium can be
    used for automating UI
    Tests.

    View full-size slide

  23. Automated Test Generation
    • Use tools like IntelliTest to explore your
    .NET code to generate test data and a
    suite of unit tests.
    • For every statement in the code, a test
    input is generated that will execute that
    statement.
    • A case analysis is performed for every
    conditional branch in the code.
    • Good starting point –must build on it.

    View full-size slide

  24. Pipeline Orchestration
    • It must be integrated with your
    automation suite.
    • Understand how it works, how to
    interpret results and how to make it
    scalable.
    • Must be transparent and everyone
    must have full visibility into what’s
    being run through the pipeline.

    View full-size slide

  25. Deployments
    • A/B Testing (Testing in Production)
    • A randomized experiment with two
    variants, A and B
    • Azure App Service supports Slots to
    run two or more variants of app
    simultaneously
    • Blue-Green Deployments
    • The staging slot represents “green”
    deployment and production slot
    represents “blue” deployment.
    • Canary Releases (Deployment Rings)
    • Reduces the risk of introducing a new
    software version in production by
    gradually rolling out the change to a
    small subgroup of users, before
    rolling it out to everyone

    View full-size slide

  26. Continuous Monitoring
    • Continuous monitoring across the entire SDLC, not just production,
    should be used as a compass to help you navigate your continuous
    testing transformation.
    • Dashboards in real time.
    Must be automatic and
    entire team must have
    access.
    • Use automated runners to
    identify issues that do not
    manifest on initial deploy &
    occur after certain time.

    View full-size slide

  27. 12 Key Disciplines of Continuous Testing (Recap)
    1. Requirements Engineering
    and Acceptance Criteria
    2. Unit tests
    3. On-Demand Environments
    4. Test and Test Data
    Management
    5. API Management
    6. Performance/Load Testing
    7. Security Testing
    8. Test Automation
    9. Automated Test Generation
    10. Pipeline Orchestration
    11. Deployments
    12. Continuous Monitoring

    View full-size slide

  28. Azure DevOps

    View full-size slide

  29. Azure DevOps
    Deliver value to your users faster
    using proven agile tools to plan,
    track, and discuss work across
    your teams.
    Build, test, and deploy with CI/CD that
    works with any language, platform,
    and cloud. Connect to GitHub or any
    other Git provider and deploy
    continuously.
    Get unlimited, cloud-hosted
    private Git repos and collaborate
    to build better code with pull
    requests and advanced file
    management.
    Test and ship with confidence
    using manual and exploratory
    testing tools.
    Create, host, and share packages with
    your team, and add artifacts to your
    CI/CD pipelines with a single click.
    Azure Boards Azure Repos
    Azure Pipelines
    Azure Test Plans Azure Artifacts
    https://azure.com/devops

    View full-size slide

  30. Get end-to-end traceability. Run tests and log
    defects from your browser. Track and assess quality
    throughout your testing lifecycle.
    Azure Test Plans
    Capture rich data
    Capture rich scenario data as you execute tests to
    make discovered defects actionable. Explore user
    stories without test cases or test steps. You can
    create test cases directly from your exploratory test
    sessions.
    Test across web and desktop
    Test your application where it lives. Complete
    scripted tests across desktop or web scenarios. Test
    on-premises application from the cloud and vice-
    versa.
    Get end-to-end traceability
    Leverage the same test tools across your engineers
    and user acceptance testing stakeholders. Pay for the
    tools only when you need them.
    https://azure.com/devops

    View full-size slide

  31. Self-Service Dev/Test Environments
    Azure Lab Services
    Simplify cloud environment
    management for developers
    and testers.
    Enforce policies and control
    costs with full visibility
    Use templates, custom
    images and formulas to
    reproduce environments.
    Orchestrate with Azure
    Pipelines or integrate using
    REST API

    View full-size slide

  32. Infrastructure and Configuration as Code
    Azure Resource Manager, Automation & 3rd Party Integrations
    Infrastructure as Code,
    built-in
    Support for 3rd party and OSS
    tooling such as Terraform,
    Ansible, Chef, Puppet &
    SaltStack
    Azure Config & Automation

    View full-size slide

  33. Gain full visibility and control of
    your cloud security state
    Continuous Security
    Azure Security Center
    Leverage ML to Proactively
    identify and mitigate risks to
    reduce exposure to attacks
    Quickly detect and respond
    to threats with advanced
    analytics

    View full-size slide

  34. Smarter Insights, Faster
    Azure Monitor, Application Insights & Log Analytics
    Pre-defined solutions with smart
    thresholds
    Visualize data in intuitive and
    customizable dashboards
    Separate the signal from
    the noise and accelerate
    root-cause analysis
    Integrate your existing
    processes & tools like
    Service Now

    View full-size slide

  35. Mix and match
    to create
    workflows with
    tools from
    Microsoft, open
    source or your
    favorite
    3rd party tools
    Azure DevOps: Choose the tools and clouds you love
    Azure
    DevOps lets
    developers
    choose the
    tools that
    are right for
    them
    Target any cloud, on-prem or both and deploy to the servers you need

    View full-size slide

  36. Summary
    • Learning the art of testing is like studying for a Ph.D.; the more you learn,
    the more you realize there is much more to know.
    • Testing must become integral to the development process (rather than a
    “hygiene” task tacked on when development is complete)
    • Don’t re-invent the wheel, rather build on it - what kind of patterns to
    adopt and where to look for faults/bugs.
    • Determine the right tests to execute at different stages of the delivery
    pipeline (smoke testing upon check-in, API/message layer testing after
    integration, end-to-end testing at the system level, …)
    • Each set of tests must execute fast enough that it does not create a
    bottleneck at the associated stage of the software delivery pipeline.

    View full-size slide

  37. © Microsoft Corporation
    Lets break the software before it
    breaks in production!

    View full-size slide

  38. © Microsoft Corporation
    Questions?
    Please provide Feedback
    Or
    aka.ms/suhasr2019

    View full-size slide