The Art of Self-Testing - Software Testing 101

Empowering Software Engineers for Quality and Efficiency.
Introductory Series by Freddy Munandar

The goal of the introductory series is to encourage you to build discipline and habits.

Agenda:
- Software Testing 101 ~8 min
- Understanding Self-Testing ~2 min
- Core Principles of Self-Testing ~3 min
- Advantages and Challenges of Self-Testing ~3 min
- Overcoming Challenges of Self-Testing ~5 min
- Self-Testing Guide ~5 min
- Q&A Part-1 ~5 min
- Static Testing 101 ~5 min
- Unit Testing 101 ⌨️ ~20 min
- Integration Testing 101 ⌨️ ~20 min
- Key Takeaways ~3 min
- Q&A ~10 min

Transcript

1. The Art of Self-Testing Empowering Software Engineers for Quality and Efficiency Introductory Series by Freddy Munandar Seri Pengantar
2. The goal of the introductory series is to encourage you to build discipline and habits.
3. Agenda • Software Testing 101 ~8 min • Understanding Self-Testing ~2 min • Core Principles of Self-Testing ~3 min • Advantages and Challenges of Self-Testing ~3 min • Overcoming Challenges of Self-Testing ~5 min • Self-Testing Guide ~5 min • Q&A Part-1 ~5 min • Static Testing 101 ~5 min • Unit Testing 101 ~20 min • Integration Testing 101 ~20 min • Web UI Testing 101 • App UI Testing 101 • Automated Test 101 • Key Takeaways ~3 min • Q&A ~10 min 3
4. Before we start please silence your smartphone, turn off notifications, or take a side anything that can distract you! 4
5. Software Testing 101 – Law • Testing can show the presence but not the absence of errors. –Djikstra ’70 • A developer is unsuited to test his or her code. –Weinberg ’71 • Approximately 80% of defects come from 20% of modules. –Pareto-Zipf ‘97 5
6. Software Testing 101 – Principles #1 Testing shows the presence of defects, not their absence. #2 Exhaustive testing is impossible. #3 Early testing saves time and money. #4 Defects cluster together. #5 Beware the pesticide paradox #6 Testing is context-dependent #7 Absence-of-errors is a fallacy 6
7. Software Testing 101 – Principles #1 Testing shows the presence of defects, not their absence. • Testing finds mistakes but can't guarantee perfection. • Testing exposes errors and their causes but can't ensure flawless software. Example: Testing may reveal a flaw in a payment processing system, such as incorrect calculation of transaction fees, but it cannot ensure that the system is immune to all errors. 7
8. Software Testing 101 – Principles #2 Exhaustive testing is impossible. • Testing everything is impossible. • It's impractical to test every possible scenario; focus on important ones. Example: Despite thorough testing of an e-commerce website, it's impossible to account for every possible combination of user behavior, such as unexpected interactions between features. 8
9. Software Testing 101 – Principles #3 Early testing saves time and money. • Start testing early to save time and money. • Testing at the beginning catches issues before they become costly to fix later. Example: Identifying a critical security vulnerability during the initial stages of software development saves significant resources compared to discovering it post-release. 9
10. Software Testing 101 – Principles #4 Defects cluster together. • Errors often gather in specific areas. • Most problems concentrate in a few parts, so prioritize testing there. Example: A thorough analysis of customer support tickets reveals that the majority of complaints stem from a specific feature, prompting focused testing efforts in that area. 10
11. Software Testing 101 – Principles #5 Beware the pesticide paradox • Tests need to evolve to remain effective. • Repeating the same tests without change won't catch new problems. Example: Despite passing all existing test cases, a software update introduces a new compatibility issue with a popular web browser, highlighting the need for ongoing test maintenance. 11
12. Software Testing 101 – Principles #6 Testing is context-dependent • Testing strategies must fit the context. • Testing approaches should adapt to the unique requirements and environment of each system. Example: Testing a medical device requires a different approach than testing a social media platform due to the stringent regulatory requirements and potential safety implications. 12
13. Software Testing 101 – Principles #7 Absence-of-errors is a fallacy • Perfect software doesn't exist. • Even if all requirements are met, software can still disappoint users or have usability issues. Example: Even if a mobile app meets all functional requirements, users may abandon it if the user interface is unintuitive or lacks accessibility features, impacting user satisfaction. 13
14. Software Testing 101 – Testing Levels 14 Testing Level Description Examples Component (Unit) Test Tests individual units or components to ensure they function correctly. Testing functions or methods with various inputs. Integration Test Verifies interactions between integrated components or modules. Testing interactions between component, partial frontend and backend services, or service to service. System Test Evaluates the entire system's behavior against requirements. Testing end-to-end user workflows in the software. Acceptance Test Validates software against acceptance criteria defined by stakeholders. Collaborating with end users to test usability and satisfaction.
15. Software Testing 101 – Test Types Functional Testing What: Checks if the software does what it's supposed to. How: Tests each part to see if it works correctly. Examples: Testing if clicking a button does what it should. Benefits: Makes sure it does its job right. Non-Functional Testing What: Looks at things like speed, security, and usability. How: Tests how fast it is, how secure, and how easy to use. Examples: Checking how many users it can handle at once, if it's safe from hackers. Benefits: Makes sure it's not just functional, but also fast, safe, and easy to use. 15
16. Aspect Functional Testing Non-Functional Testing Definition Verifies that the software functions correctly as per specified requirements. Evaluates attributes beyond functionality, such as performance, usability, and security. Focus Tests specific functions or features of the software. Tests quality attributes like performance, reliability, scalability, usability, and security. Requirements Requires knowledge of functional requirements and system behavior. Requires understanding of non-functional requirements and quality attributes. Testing Techniques Includes regression testing, smoke testing, sanity testing, and user acceptance testing (UAT). Includes performance testing, load testing, stress testing, security testing, usability testing, etc. Measure of Success Pass/fail based on whether functions work correctly as expected. Quantitative metrics (e.g., response time, throughput, error rates) or qualitative assessment (e.g., user satisfaction) based on quality attributes. Execution Approach Typically conducted using test cases derived from functional requirements. Uses specific test scenarios or metrics to evaluate quality attributes. Examples Testing login functionality, adding items to a shopping cart, etc. Testing response time under peak load, evaluating system security measures, assessing user interface responsiveness, etc. Functional Testing vs. Non-Functional 16
17. Software Testing 101 – Testing Pyramid 19
18. Software Testing 101 – Testing Pyramid 20
19. Software Testing 101 – Anti-Pattern 21
20. Software Testing 101 – Unit Testing 22
21. Software Testing 101 – Integration Testing 23
22. Software Testing 101 – System Testing 24
23. Software Testing 101 - Testing Techniques White-Box Testing What: Checks how the insides of the software work. How: Looks at the code to design tests. Examples: Checking every line of code, every path (e.g., Unit Testing with Testing Coverage Report). Benefits: Finds mistakes in how the code is written. Black-Box Testing What: Tests if the software does what it should. How: Tries different things without knowing how it's made. Examples: Trying different inputs and seeing what happens (e.g., API Testing). Benefits: Makes sure it works right from the outside. Grey-Box Testing What: Mixes white-box and black-box testing. How: Knows a bit about how it's made, but not everything. Examples: Testing with some inside knowledge, but also trying it from the outside (e.g., Integration Testing). Benefits: Gives a good balance of tests. 25
24. Software Testing 101 - Testing Techniques 26 PoC = Point of Control, PoO = Point of Observation Test object = A test is named according to the type of object to be tested (for example, “GUI test“ or “database test”)
25. Functional Testing vs. Black-Box Testing Aspect Functional Testing Black-Box Testing Definition Verifies that software functions as expected based on specified requirements. Tests the software's functionality without knowledge of its internal implementation. Focus Tests specific functions/features of the software. Tests the software's external behavior, inputs, and outputs. Knowledge Required Requires knowledge of functional requirements and system functionality. Does not require knowledge of internal code or system architecture. Approach Tests against documented functional requirements and specifications. Tests based on user expectations and system requirements, focusing on inputs and outputs. Techniques Includes regression testing, smoke testing, sanity testing, and user acceptance testing (UAT). Includes equivalence partitioning, boundary value analysis, decision tables, and state transition testing. Goal Ensures that the software performs its intended functions correctly. Validates that the software behaves as expected from a user's perspective. 27
26. Black-Box Testing - Equivalence Partitioning 28 It's like sorting your inputs into similar groups. We then test one input from each group to make sure we cover all possibilities
27. Black-Box Testing - Boundary Value Analysis 29 We pay extra attention to the edges of our input ranges. By testing values right at the edges and just beyond, we catch common mistakes.
28. Black-Box Testing - Decision Tables 30 Write down all possible conditions and what to do in each case. Helps make sure all situations are tested.
29. Black-Box Testing - Decision Tables 31 Write down all possible conditions and what to do in each case. Helps make sure all situations are tested. Age Is Member Eligibility < 0 - Invalid age < 18 - No 18-65 false No 18-65 true Yes > 65 - No
30. Black-Box Testing - State Transition 32 We're checking how our system behaves as it moves from one state to another. We want to make sure it behaves correctly during these transitions, like turning a light on or off.
31. Testing Strategies • Risk-Based Testing Strategy • Regression Testing Strategy • Test Automation Strategy • Continuous Testing Strategy • Exploratory Testing • User Acceptance Testing (UAT) Strategy • Localization and Internationalization Testing Strategy • Accessibility Testing Strategy • Performance Testing Strategy • Smoke Testing Strategy • Security Testing Strategy • Usability Testing Strategy • Compatibility Testing Strategy • Compliance Testing Strategy • Scalability Testing Strategy • Disaster Recovery Testing Strategy 33
32. Software Testing 101 – Dev vs. QA vs. SDET 34 QA Analyst SDET Dev / Software Engineer Manual Testing • • API Automated Testing • Unit Testing • Integration Testing Edge Case Testing • • Static Testing • Performance Testing** **SDET Core Team [X] @fredeatworld
33. Understanding Self-Testing Definition • Self-testing involves developers taking responsibility for testing their code before it goes through formal testing processes. • It includes creating and/or executing test cases to detect and address issues early in the development cycle. • What test cases mean here is writing unit testing / integration testing. 35
34. Understanding Self-Testing Why it’s important • Self-testing ensures early detection of defects, reducing cost and effort for fixing them later. • It fosters a culture of quality and accountability within development teams. • By catching issues early, it saves time and resources and accelerates time-to-market for software products. 36
35. Core Principles of Self-Testing 37 Core Principles Description Examples Early Detection Detect issues as early as possible in the development process to minimize their impact and reduce the cost of fixing them later. Running unit tests as code is being written to catch bugs before integration. Comprehensive Coverage Ensure that testing covers all critical functionalities, edge cases, and scenarios to verify the correctness and robustness of the software. Testing various input combinations in a login form (valid, invalid, edge cases). Automation Automate repetitive testing tasks to improve efficiency, repeatability, and reliability of tests, allowing faster feedback on code changes. Using a testing framework to automatically run regression tests after each build. Independence Test independently of external dependencies, environments, and other components to isolate issues and ensure accurate test results. Mocking external API responses in integration tests to test component in isolation. Continuous Improvement Continuously review and improve testing practices, test cases, and processes to adapt to changing requirements and enhance overall quality. Conducting retrospectives to identify areas for improvement in testing strategies.
36. # Statement 1 I will not produce harmful code. 2 The code that I produce will always be my best work. I will not knowingly allow code that is defective either in behavior or structure to accumulate. 3 I will produce, with each release, a quick, sure, and repeatable proof that every element of the code works as it should. 4 I will make frequent, small releases so that I do not impede the progress of others. 5 I will fearlessly and relentlessly improve my creations at every opportunity. I will never degrade them. 6 I will do all that I can to keep the productivity of myself and others as high as possible. I will do nothing that decreases that productivity. 7 I will continuously ensure that others can cover for me and that I can cover for them. 8 I will produce estimates that are honest in both magnitude and precision. I will not make promises without reasonable certainty. 9 I will respect my fellow programmers for their ethics, standards, disciplines, and skill. No other attribute or characteristic will be a factor in my regard for my fellow programmers. 10 I will never stop learning and improving my craft. The Oath 38 Part III: Ethics
37. Advantages and Challenges of Self-Testing Advantages • Early Bug Detection • Faster Feedback Loop • Improved Code Quality • Empowerment and Ownership • Integration with Development Workflow Challenges • Time and Resources • Expertise and Training • Bias and Blind Spots • Test Maintenance • Over-reliance on Automation 39
38. Challenges Solutions Time and Resources Prioritize Testing Activities • Allocate dedicated time for testing activities within the development schedule. • Use agile methodologies to incorporate testing into iterative development cycles. Expertise and Training Invest in Training and Skill Development • Provide developers with training in testing techniques, tools, and best practices. • Encourage knowledge sharing and collaboration among team members to enhance testing skills. Bias and Blind Spots Foster a Culture of Collaboration and Feedback • Encourage peer reviews and pair programming to promote collaboration and identify blind spots. • Implement code review processes that involve multiple team members to minimize bias. Test Maintenance Prioritize Test Refactoring and Cleanup • Regularly review and refactor existing tests to keep them relevant and maintainable. • Automate test maintenance tasks where possible to reduce manual effort and streamline the process. Over-reliance on Automation Balance Automated and Manual Testing • Strike a balance between automated and manual testing efforts based on the nature of the project and testing requirements. • Use manual testing for exploratory testing, usability testing, and edge cases that are challenging to automate. Overcoming Challenges of Self-Testing 40
39. Step Description Define Goals Clarify why you're testing: to ensure quality, find bugs early, and maintain reliability. Choose Techniques Select testing methods like unit testing, integration testing, regression testing, and static testing. Prepare Environment Set up testing environment with necessary tools and frameworks. Write Test Cases Develop clear and concise test cases covering different scenarios and functionalities. Automate Tests Use testing frameworks and tools to automate repetitive testing tasks where possible. Execute Regularly Integrate testing into your development workflow. Run tests regularly as code is written or updated. Analyze Results Review test results to identify issues, errors, and failures. Investigate root causes and prioritize fixes. Refactor Tests Continuously review and improve existing tests to keep them relevant and maintainable. Document Findings Document test cases, results, and insights to track testing efforts and share learnings with the team. Learn and Adapt Stay updated on testing practices, tools, and technologies. Continuously learn and adapt to new techniques. Self-Testing Guide - Steps 41
40. Step Example Define Goals Ensure that user authentication API endpoints perform securely and efficiently, providing a seamless user experience. Choose Techniques Unit Testing: Test individual functions using Jest. Integration Testing: Test API endpoints using Supertest. Static Analysis: Use ESLint to enforce coding standards. Prepare Environment Set up local development environment with Node.js, Express.js, and MongoDB for backend development. Write Test Cases Develop test cases for scenarios like successful login, invalid credentials, and user role-based access control. Automate Tests Automate API tests using Jest or Mocha with Supertest. Execute Regularly Integrate testing into CI/CD pipeline to run tests on every code push using Jenkins or GitLab CI. Analyze Results Review test reports from CI/CD pipeline to identify failing tests and prioritize fixes. Refactor Tests Refactor existing test cases to improve readability and maintainability. Document Findings Document test cases, results, and insights for knowledge sharing and collaboration. Learn and Adapt Stay updated on backend development techniques and testing frameworks through conferences and online courses. Self-Testing Guide - Examples 42
41. Q & A Part-1 ~5min Don't be a “fake plastic tree”, please ask. 43
42. Software Testing – Dev vs. QA vs. SDET 44 QA Analyst SDET Dev / Software Engineer Manual Testing • • API Automated Testing • Unit Testing • Integration Testing Edge Case Testing • • Static Testing • Performance Testing** **SDET Core Team [X] @fredeatworld
43. Static Testing 101 – Concept • The underlying concept of static testing is prevention. • Errors and other deviations from plan are identified before they can have a negative effect during further development. • Static tests identify faults directly in documents and other work products (e.g., Legacy code). • Reviews are static tests. 45
44. Static Testing 101 – Concept Typical defects found by reviews • Requirements defects • Software design defects • Coding defects • Deviations from standards • Faulty interface specifications • Security vulnerabilities 46
45. Static Testing 101 – Technique • Ad-hoc • No rules • Checklist-based • Pre-formulated questions support the review process • Scenarios and dry runs • Scenarios are run through • Role-based • Participants adopt specific roles for the review • Perspective-based • Different views on the review object 47
46. Static Testing 101 – Relevant Law • Requirement deficiences are the prime source of project failures. –Glass’ Law ’98 • The assessment: • Far too many • Unstable due to late changes • Ambiguous • Incomplete • Errors are most frequent during the requirements and design activities and are the more expensive the later they are removed. – Boehm’s Law ’75 48
47. Static Testing 101 – Relevant Law • The value of a model depends on the view taken, but none is best for all purposes. –Davis’ Law ’90 49
48. Static Testing 101 – Relevant Law • Good designs require deep application domain knowledge. –Curtis’ Law ’88,’90 50
49. Example Behavior Example Output Suggested Time Review UI Design • Identified inconsistent label or input naming. • Identified complex UI flow and suggested a simpler flow to reduce the development effort. • Identified won’t do feature development because overlapping with another team tasks. At least H-1 before design checkpoint session Review Product Brief Document • Identified business process is unclear. • Identified missing step on the business process. At least H-1 before design checkpoint or H+1 after design checkpoint Review Product Requirement Document (PRD) • Identified incomplete acceptance criteria on story. • Identified unclear story description. At least H-1 before Grooming Or before pre-grooming Review Test Strategy Document (TSD) • Identified some of scenario is missing. • Identified incomplete test scenario overview. At least H+1 after Grooming Review Technical Requirement Document (TRD) • Identified the ERD is incomplete, does not have column description, etc. • Identified missing important sequence diagram to provide clear communication between services. At least H+1 after Technical discussion Static Testing 101 – Example Behavior 51
50. Software Testing – Dev vs. QA vs. SDET 52 QA Analyst SDET Dev / Software Engineer Manual Testing • • API Automated Testing • Unit Testing • Integration Testing Edge Case Testing • • Static Testing • Performance Testing** **SDET Core Team [X] @fredeatworld
51. Unit Testing 101 – Goal 53 This phenomenon of quickly decreasing development speed is also known as software entropy. Entropy (the amount of disorder in a system) is a mathematical and scientific concept that can also apply to software systems. (If you’re interested in the math and science of entropy, look up the second law of thermodynamics.)
52. Unit Testing 101 – Goal 54 Remember, not all tests are created equal. Some of them are valuable and contribute a lot to overall software quality. Others don’t. They raise false alarms, don’t help you catch regression errors, and are slow and difficult to maintain.
53. Unit Testing 101 – Coverage 55 Code is a liability, not an asset. The more code you introduce, the more you extend the surface area for potential bugs in your software, and the higher the project’s upkeep cost. It’s always better to solve problems with as little code as possible.
54. Unit Testing 101 56
55. Unit Testing 101 – Definition • There are several interpretations about exactly what constitutes a unit. • A single procedure • A function • A body of code that implements a single function • Source code that fits on one page • A body of code that represents work done in 4 to 40 hours (as in a work breakdown structure) • The smallest body of code that can be compiled and executed by itself • 300 lines of code • In an object-oriented programming language, there is general agreement that a class is a unit. 57
56. Unit Testing 101 – Structure • AAA Pattern Given-When-Then Pattern • Given— Corresponds to the arrange section • When— Corresponds to the act section • Then— Corresponds to the assert section 58
57. Unit Testing 101 – Parameterized Test 59 Reduce the amount of test code.
58. Unit Testing 101 – Practices with GitHub Copilot File Structure Techniques: Mock Testing Coverage Report 60
59. Software Testing – Dev vs. QA vs. SDET 61 QA Analyst SDET Dev / Software Engineer Manual Testing • • API Automated Testing • Unit Testing • Integration Testing Edge Case Testing • • Static Testing • Performance Testing** **SDET Core Team [X] @fredeatworld
60. Integration Testing 101 62
61. Integration Testing 101 – Practices with GitHub Copilot File Structure Techniques: Mock Testing Coverage Report 63
62. Key Takeaways • Self-testing is essential, do not ship s**t. • Understanding Software Testing including Static Testing makes QA analysts, SDET, and Engineers live in harmony. • Unit Testing and Integration Testing make Software Engineers look awesome while keeping the high quality of their masterpiece. 65
63. Reading Material for Software Engineer *All books available at O`Reilly 67 1 2 3 4
64. Reading Material for Software Engineer 69 *All books available at O`Reilly
65. Reading Material for Software Engineer Use ChatGPT to Improve Code https://learning.oreilly.com/library/view/use-chatgpt- to/9781098163075/ch01.html Use ChatGPT for Test-First Development https://learning.oreilly.com/library/view/use-chatgpt- for/9781098165857/ch01.html Use ChatGPT to Write Unit Tests https://learning.oreilly.com/library/view/use-chatgpt- to/9781098167646/ch01.html 71 *All articles available at O`Reilly
66. Reading Material for QA Analyst / SDET This book is written for the test analyst who wants to achieve advanced skills in test analysis, design, and execution. With a hands-on, exercise- rich approach, this book teaches you how to define and carry out the tasks required to put a test strategy into action. https://learning.oreilly.com/library/view/advanced-software-testing/9781457189517/ https://learning.oreilly.com/library/view/advanced-software-testing/9781492016328/ This book teaches test managers what they need to know to achieve advanced skills in test estimation, test planning, test monitoring, and test control. Readers will learn how to define the overall testing goals and strategies for the systems being tested. 73 *All books available at O`Reilly
67. Reading Material for QA Analyst / SDET This book presents functional and technical aspects of testing as a coherent whole, which benefits test analyst/engineers and test managers. https://learning.oreilly.com/library/view/the-software-test/9781492014706/ https://learning.oreilly.com/library/view/advanced-software-testing/9781457189074/ This book is written for the technical test analyst who wants to achieve advanced skills in test analysis, design, and execution. 75 *All books available at O`Reilly
68. Q & A ~10min Don't be a “fake plastic tree”, please ask. 76
69. Don't stop here. Please read the book and do a discussion or sharing session more in-depth with your team. 77
70. Thank You See you on the next introductory series System analysis, system design, system optimization, etc. 78