Cost of Quality How to Save Money

5206c19df417b8876825b5561344c1a0?s=47 Exactpro
November 12, 2011

Cost of Quality How to Save Money

EXTENT Conference October 2011 - Test Automation for Trading Systems

Presentation by Sibgatullina Taya
QA Solution Manager
HP Software CEE

5206c19df417b8876825b5561344c1a0?s=128

Exactpro

November 12, 2011
Tweet

Transcript

  1. Sibgatullina Taisiya Solution Consultant, CEE Cost of Quality How to

    Save money
  2. 2 HP Application Lifecycle Management & HP Quality Center 11.00

    Part 1: Quality assurance
  3. 3 HP Application Lifecycle Management & HP Quality Center 11.00

    3 Cost Of Quality
  4. 4 HP Application Lifecycle Management & HP Quality Center 11.00

    Application Lifecycle model
  5. 5 HP Application Lifecycle Management & HP Quality Center 11.00

    ALM 5
  6. HP (41.1%) IDC Worldwide Automated Software Quality Revenue by Vendor

    in „09 Market Leading Solutions IBM (Rational) 11.1% Microsoft 5.6% Compuware 5.3% Borland 2.9% Empirix 1.9% Others (Combined) 32.1% Source: Melinda Ballou IDC, “Worldwide Automated Software Quality 2009-2013 Forecast Update and 2008 Vendor Shares: Evolving Quality in a Challenging Market” (Doc # 219695E) October 2009 “As the market revenue leader, with a 41.1% share, HP is in a position to disproportionately influence the overall ASQ market.”
  7. ©2009 HP Confidential 7 New Gartner report just published in

    January 201 1 HP ALM #1 AGAIN
  8. Designed for the End-to-End Application Lifecycle HP ALM 1 1.00

    is a Unified Platform Requirements Management Requirements Management Requirements Definition Quality Management Functional Performance Security Development Management IDE, SCM, Build Integrations Defect Mgmt Secure Develop- ment Foundation Traceability Reporting Process Standardization (i.e. Waterfall, Agile) Extensibility (Open APIs) Integrations into Strategy and Planning (PPM, Gov) Integrations into Run (Deployment, APM, ITSM) Unified for the Core App Lifecycle, Integrated with the Complete App Lifecycle Application Lifecycle Management Project Planning & Tracking Enterprise Release Management
  9. ALM Platform Process Standardization (i.e. Waterfall, Agile) A single platform

    for managing the application lifecycle HP Application Lifecycle Management Dashboard Single Data Repository Reporting Open APIs (REST) HP Application Lifecycle Management Release Management Requirements Management Defect Management Project Planning & Tracking Business Models Test Lab Test Plan HP Unified Functional Testing Performance Optimization Application Security Testing Business Application Validation HP QuickTest Professional HP Service Test HP Performance Center HP LoadRunner HP QAInspect HP Business Process Testing Services Service Test Management Test Data Management Asset Sharing & Reuse
  10. 10 HP Application Lifecycle Management & HP Quality Center 11.00

    Project Planning & Tracking Track & measure project milestones and KPIs E2E Testing Readiness Assessment Testing Progress 92% Req Reviewed Req Approved 0 Defects High Defects Med Test Exec Tests Passed 15 98% 92% 10 Defects High Defects Med Test Auth Test Exec Tests Passed 25 98% 89% 45 Defects High Defects Med Test Exec Tests Passed 95 78% 90% 93% Test Authoring Integration E2E Testing User Acceptance Testing Requirements Authoring Development Test Execution Functional, Performance and Security 95% 85% Req Implemented Tests Reviewed 75% Development Progress Requirements Progress
  11. 11 Integrated solution for defining, managing and tracking requirements Requirements

    Management – Business analyst can collaborate on application requirements without fear of overriding other‟s work – Assess overall risk to the business through real- time visibility of test coverage and associated defects against business needs – Assess the impact of change before changing requirements – Easy adoption by Business Analysts – Define and track multiple requirement types – Establish templates to ensure higher quality requirements – Manage requirements dependencies – Bi-directional traceability from requirements to requirements, tests and defects – Manage change requests through formal approval process – Familiar rich test editor interface – Version Control and Baselining – Leverage existing assets in MS Word
  12. 12 Track changes made to entities in your project over

    time Baselining – Compare releases and cycles – A snapshot of the project state at a particular point in time – Enable rollback of assets to key points in the application lifecycle – Compare baselines to assess changes – Ability to use baselines in test execution – Create baselines – including traceability between assets – Compare baselines both at high level and drill down to specific changes including traceability – Report and print baselines for sign-off – Control who can create and modify baselines
  13. 13 Track and measure the progress and quality of releases

    and cycles Release & Cycle Management – Manage projects and assets against release and cycles – Track release progress – Gain visibility into release readiness and release quality – Compare releases and cycles for continuous process improvement – Manage requirements & tests according to application releases and dev/QA cycles – Compare planned vs. actual progress for each application release and cycle – Assess release quality across multiple cycles
  14. 14 Track and measure project milestones and KPIs Project Planning

    & Tracking – Visibility into application projects progress and status – Visibility into key project milestones – Identify delays or poor quality before it impacts the schedule – Proactively adjust plans when a project is not on track – Define and track project milestones – Define and measure key performance indicators across the application lifecycle – Automatically updates project status against milestones – Drill down on KPIs for more details
  15. 15 Define, manage and track all test scripts Test Plan

    – Create test cases to adequately test the requirements – Manage all test types (Functional, Performance, Security) in one place – Ensure requirements coverage through traceability to requirements – Manage change associated with requirements changes – Design, build and manage manual and automated test cases – Link test cases to requirements – Leverage test cases from MS Word and MS Excel – Version Control and Baselining
  16. 16 The next generation of manual testing HP Sprinter –

    Accelerate manual testing – Improved testing accuracy – Better communication of defects to development (accelerated defect remediation) – Unobtrusive, easy to use user interface – Supports exploratory testing – Reduce time and resources by testing multiple environments at once – Toolkit – Screen capture and annotations, measure objects position etc. – Record user actions to support exploratory testing – Automatic data injection – Mirror testing – Log defects with associated information directly into HP Quality Center/ALM
  17. 17 Track defects across the entire application lifecycle Defect Management

    – Manage, track and enforce defects across the application lifecycle – Create defects from manually or directly from the execution of manual and automated tests – Clearly communicate bugs to developers with context from requirements and tests – Integrated into developers IDE – Enforce defect lifecycle process through custom fields and workflow – Log defects directly from test runs with test details (test step, result, screenshots, attachments) to ensure the developers can reproduce the issue – Traceability to requirements, tests and other defects – Alert key stakeholders through automated emails
  18. 18 Clear communication and visibility into your application projects Reports,

    Graphs & Dashboard – Make informed business decisions – Clearly communicate to all stakeholders – Understand the real-time status of an application – Full trending analysis and insight into application projects – Understand release and cycle progress – Fully customizable project reports (export to MS Word) – Integrated Dashboard – Out of the box or customizable analysis reports and graphs – Report to MS Excel – Embed graphs and charts into web portals – Requirements traceability matrix
  19. ©2010 Hewlett-Packard Development Company, L.P. The information contained herein is

    subject to change without notice Development tools integration
  20. • Traceability between requirements, code changes and defects • Adjustable

    processes • Side-tools integrations Working with defects in various tools Quality Center Release & Requirements Test Plan Test Lab Defects Developer STRATEGY/ DEMAND Strategic demand • New applications • New services • Application integrations Operational demand Other tools JIRA BugZilla StarTeam CCE Tracker VersionOne
  21. Development tools integrations • Tasktop & CollabNet • Eclipse &

    Eclipse- based clients • Visual Studio • CollabNet • AccuRev • Perforce • CVS • ClearCase • CollabNet • Hudson • Electric Cloud • AnthillPro
  22. TFS HP Application Lifecycle Intelligence SVN Jenkins TFS CVS Electric

    Cloud SVN Hudson Waterfall RUP Scrum Hybrid Single Platform Test Build Req HP ALM
  23. TFS The Decision Support System for ALM… SVN Jenkins TFS

    CVS Electric Cloud SVN Hudson Waterfall RUP Scrum Hybrid Single Platform Test Build Req HP ALM
  24. Key ALI Features Code changes from Subversion Code changes from

    TFS New Module Drill down
  25. End-to-end app delivery Report Key ALI Features

  26. HP ALM 1 1 and TFS integration QC TFS TFS

    adapter for QC Synchronizer 1.4 Defects Bugs & Reqs Work Items Reqs
  27. 28 HP Application Lifecycle Management & HP Quality Center 11.00

    Part 2: Performance validation
  28. Solutions for management and automation of performance optimization HP Performance

    Center User/privilege management Infrastructure management Central repository Global access & collaboration Requirements & Defects Management Framework: Performance Center Resource pooling & scheduling HP Performance Center Family Performance Testing: LoadRunner / Performance Center VuGen Controller Load Generator Analysis Monitors Diagnostics J2EE .NET SOA SAP Oracle
  29. The market leading solution for performance validation HP LoadRunner Enables

    a consistent, repeatable process to: • Emulate production workload to test a broad range of applications • Identify performance bottlenecks, diagnose the root cause of issues • Optimize application performance before deployment Reduce risk of deploying systems that do not meet performance requirements Reduce cost by predicting scalability and capacity and eliminating over-deployment Reduce time by shortening test cycle time. Accelerate delivery of high quality applications
  30. – Sequential process takes time – Scripting expertise needed (especially

    for complex apps) – Protocol level scripts break with minor app changes Current scripting process The challenge: Scripting process Record a script Correlations Parameterize data Add transactions Add logic
  31. – Many technologies, frameworks and toolkits – Very customizable –

    almost no “vanilla” usage – Dynamic, asynchronous behavior – Client-side processing and logic – Most testing solutions do not support all the customizations New Web 2.0 technologies The challenge: Scripting for web 2.0 apps Services Web Server Web Services Web Server Web Server Web Server App Server Data Server Networks Data Server Cloud App Server EDW Client
  32. LoadRunner VuGen Record an application business process into an automated

    script Add end-user transactions, parameterize input data, add verification checkpoints Apply user behaviors such as network connectivity or pacing Application Scripts Business process VuGen Recorder Browser
  33. LoadRunner Controller  Design and execute load scenarios that mimic

    production conditions  Emulate hundreds or thousands of concurrent users  Gather end-to-end transaction response times  Collect application, network and system data across all architectural tiers and components Diagnostics Monitors Load balancer Web server App servers Database Network Controller Load generators VUsers US Europe
  34. The revolutionary way of scripting modern apps INTRODUCING: LoadRunner TruClient

    Scripting engine embedded within browser Transactions, and logic can be added while scripting Data can be parameterized while scripting Automatic suggestion of automated steps No correlations required! Interactive, dynamic scripting! Supports all Ajax applications
  35. Diagnose performance bottlenecks Key benefits • Monitor system performance during

    tests • Diagnose performance bottlenecks as they occur • Identify application code level problems • Resolve issues faster Key capabilities • Sitescope monitoring for system performance metrics • Diagnostics to drill down to the method level • Data available during run time as well as for post-test analysis Monitoring and Diagnostics  LoadRunner  Performance Center 36 6 November 2011
  36. Customize and extend the industry-standard platform Key benefits • Build

    niche/custom protocols • Export analysis data for custom reports • Include custom monitoring and diagnostics Key capabilities • Custom protocol development via a SDK • Analysis API to export data for custom reports • Extensible monitoring and diagnostics framework Extensible Platform HP  Partners --> •HP supports 46% of protocols customers need. •New Demand = 30% of current usage •25% of customers requested AJAX, RDP and VoIP •Smaller demand for multiple protocols sums up to big opportunity  LoadRunner  Performance Center 37 6 November 2011
  37. Performance Requirements & Defects, Coverage reports Performance Center 1 1.0

    Performance Requirements Requirement status Requirement coverage New PC Lifecycle User to access performance requirements and defects
  38. Next-gen platform: based on the ALM platform Performance Center 1

    1.0 Performance Center ALM Platform Performance Testing Performance Lab Mgmt (scheduling) Performance Requirements & Defects Performance Test Management  Admin Console  Users & license  Projects  CoE  Collaboration  Traceability  Reporting  Version Control  Authentication  Authorization  Workflow  APIs, Integrations
  39. Topology Performance Center 1 1.0 Drag & drop from list

    to build your topology diagram Define machine names and properties Right click to set up monitors
  40. – Timeslots • Redone in Outlook fashion • More information

    available regarding booked timeslots • Allows for maintenance windows Miscellaneous Performance Center 1 1.0 Shows when you can start a test based on your requirements
  41. Timeslots Performance Center 1 1.0 Gives information on used timeslots

  42. – Timeslots • Redone in Outlook fashion • More information

    available regarding booked timeslots • Allows for maintenance windows Miscellaneous Performance Center 1 1.0 Shows when you can start a test based on your requirements
  43. Timeslots Performance Center 1 1.0 Gives information on used timeslots

  44. Operational security management OPERATIONS Service desk Production monitoring Connect to

    production Go/ No Go Performance Center – fully integrated into the application lifecycle 45 6 November 2011 A repeatable quality management process mitigates risk 45 6 November 2011 Integrate with demand Functional requirements Business requirements Security requirements Performance requirements Other non- functional requirements REQUIREMENTS MANAGEMENT Align with management and stakeholders Collaborate with design and development teams Assess and Analyze risk Establish testing priorities Create test plans RISK-BASED TEST PLANNING TEST MANAGEMENT AND EXECUTION Execute security scans Identify and customize security policies DEFECT MANAGEMENT Execute functional tests Create manual test cases Automate regression test cases Execute tests, diagnose and resolve problems Create performance scripts and scenarios Connect to production STRATEGY/ DEMAND Strategic demand • New applications • New services • Application integrations Operational demand • Defects • Enhancements • Change requests Enterprise Architecture and Policies • SOA • Security Operational security management OPERATIONS Service desk Production monitoring Operational security management OPERATIONS Service desk Production monitoring STRATEGY/ DEMAND Strategic demand • New applications • New services • Application integrations Operational demand • Defects • Enhancements • Change requests Enterprise Architecture and Policies • SOA • Security Go/ No Go Execute tests, diagnose and resolve problems Create performance scripts and scenarios
  45. Part 3: HP UFT

  46. Part 3.1 Functional Testing and Service Test

  47. A Modern testing team has new needs People • Expand

    skills to test beyond the GUI • Extend testing to more members of the team • Increased collaboration with developers and business analysts Processes • Automate as early as possible • Accommodate faster change Technology • Complex integrated systems • Test GUIs, APIs, web services and vertical integration • Test RIA, Web 2.0
  48. Late stage testing is too expensive… Iteration 1 Iteration 2

    Iteration 3 Test When it was injected way back here? Why wait to find a bug here? Remember, the age old QA principle… Time between injection and identification Cost to Fix
  49. Test & automate as early as possible C1 C1 C2

    C1 C2 C3 G2 C1 C2 G1 G1 C3 C1 C2 G1 G3 G2 C3 C1 C2 G1 … … BP1 BP2 G3 G2 C3 C1 C2 G1 Regression test at the end of each iteration or as part of continuous build Automate tests when available components become stable C GUI-less component G GUI element BP business process stable component
  50. Modern applications have many layers Process Layer A new transaction

    Exchange Screen 5 (Java) Screen 6 (.NET) SAP Screen 3 (.NET) Screen 4 (Ajax) Web 2.0 Billing Screen 1 (Ajax) Screen 2 (Java) End to end business process Graphical user interface layer Headless Layer Interact with modern GUI interfaces, Web 2.0 Continue the scenario to the headless layer; test the integration between the layers Report on the quality of the entire end to end process Component 1 (.Net API) Component 2 (Java API) Component 3 (web service) Component 4 (JMS service) Component 5 (web service) Component 6 (REST service)
  51. HP Software solutions Process Layer A new transaction Exchange Screen

    5 (Java) Screen 6 (.NET) SAP Screen 3 (.NET) Screen 4 (Ajax) Web 2.0 Billing Screen 1 (Ajax) Screen 2 (Java) End to end business process Graphical user interface layer Headless Layer Component 1 (.Net API) Component 2 (Java API) Component 3 (web service) Component 4 (JMS service) Component 5 (web service) Component 6 (REST service) HP Unified Functional Testing HP Unified Functional Testing includes the ability to test the GUI layer + the Headless layer + the end to end process.
  52. Unified Functional Testing HP Service Test HP Functional Testing (HP

    QuickTest Pro + all add-ins) Call activities from one product to the other Transfer complex data structures Integration Unified report What‟s new with HP Unified Functional Testing 11.0?
  53. Report for a Unified Functional Testing scenario Shows activities from

    both HP Functional Testing & HP Service Test Multiple views of the flow Full traceability for easier debugging
  54. Testing Web 2.0 GUIs • Ajax toolkits including Dojo, GWT,

    YUI • RIA such as Flex, Silverlight v2.0, v3.0, (v4.0 soon), WPF extensibility • New support for Firefox OOB Web technology support • Wizard-like tool helps users extend support to more web 2.0 controls • Partners and customers can share code created to customize support for add‟l toolkits Community
  55. Easy, wizard-like interface Extends HP Functional Testing fast; uses Intellisense

    Automated deployment Free for anyone? Yes, FREE! Opening web extensibility to the online community: The Extensibility Accelerator
  56. HP Service Test 11.0 - new Visual Test Designer 11/6/2011

    Modern, intuitive test design interface Reusable modules improve maintainability Visual test designer extends testing to non-developers Flexible and extensible service processing
  57. BPT Positioning What is BPT? Business Process Testing is a

    framework for QA organizations to build and reuse structured testing components in both manual and automated testing. Target users Non-technical QA personnel Subject Matter Experts BPT is ideal to Build frameworks to maximize reuse of testing components Mitigate risks by aligning business requirements with end-to-end test scenarios
  58. Part 3.2 BPT

  59. BPT 1 1.0 Simpler. Faster. Better.

  60. What‟s new in BPT 11 Significantly improved performance Built an

    all new simple and intuitive user interface Provided the ability to leverage QC 1 1.0 Capabilities Enhanced Traceability via Component Criteria Rapid Manual Testing with Sprinter Integration
  61. BPT 11 Exponential Performance Boost More Components = Better Performance

    Number of components Performanc e
  62. Test Script view – New look & Feel Cleaner UI

    showing all BPT Resources and Dependencies Clean and Simple Iteration View •Tree structure instead of a flat list • Improved Iterations dialog (multi- iterations view & Single Iterations view) • new ‘Comments’ column for each Component instance in the test • Search option inside the Iterations dialog
  63. One-step Component Creation Process Simplified the component creation process from

    three steps to just one step
  64. Enhanced Drag and Drop Feature Proving the ability to drag

    and drop multiple test components into the test script area
  65. Summary of Key Enhancements and Features • All new BPT

    UI built using QC web toolkit for seamless look and feel • Ability to create a manual testing component in just one step • Ability to drag and drop multiple components into test script pane • Enhanced test iteration view to show different combinations • Simple and easy to understand all new test scrip view • Cleaner look and feel of QC resource dependencies in BPT • Application area is a standard test resource in BPT • BPT integration with QC Sprinter • Enhanced traceability to map requirements to component level • Major performance improvements with a redesigned execution engine
  66. Traceability & Metrics Single Platform Simplified Web Access Technology Agnostic

    Powerful Automation Quality at the Center Modular Integrated Heterogeneous Best-in-class HP Lifecycle Solutions & Operating Principles COMPLETE LIFECYCLE COVERAGE PLAN Project & Portfolio Management DELIVER RUN RETIRE HP ALM Quality Center Perform Center App. Security Center Business Availability Deployment Automation Service Management Records Management Archiving CORE LIFECYCLE EXCELLENCE HP APPLICATION LIFECYCLE MANAGEMENT Project Planning & Tracking Enterprise Release Management ALM Foundation Process standardization Reporting Extensibility Requirements Management Development Management Quality Management Definition Versioning & Change Control Link to IDE, SCM Defect Mgmt. Secure Code Function Perform Secure SOA Systinet
  67. Why HP? Integrated management & automation: instead of loosely coupled

    point tools, HP provides a cohesive platform for lifecycle management and automation. Technology & methodology agnostic: broadest support for heterogeneous environments (.NET, Java, SAP, Oracle, etc.) – covers more than 70 environments. Covers traditional and Agile methods in single solution. Project ready, enterprise scale: whether for a team of ten, or an enterprise of tens of thousands, HP solutions offer proven configurability and scalability. Complete application lifecycle: no other company offers the integrated product coverage of HP to support applications from beginning to end.