The Benefits and Hidden Risks for QA and Test Engineering with Copilot

Transcript

1. Ichizoku.io Jay Revels, CEO
2. Who am I? 20+ Year Technology Career GTM Leader Last 10 years in Japan Today, I run a business here in Japan
3. We partner with global software companies and bring next generation SaaS solutions to Japanese companies. • Error and Performance monitoring • 800 Japan customers • Free to try!! • Japanese resources ◦ Implementation Guide ◦ Technical Blog • Extensive framework support • Local Japanese staff • LLM/ML monitoring sofware • First platform to Japan • Free to try!! ◦ Open source Version テスト用 ◦ Japanese content • Local Japan staff Monitoring Tools for Software Engineers and Ai Engineers
4. Mission Help software engineers release high quality code more quickly Help LLM/ML engineers release successful LLM and ML applications.
5. Leave Here Today With… • Why code quality tools are even more important with The Rise of Code Generation technology • A clear understanding of what Codecov does and does not do. • Who Codecov is for your engineering organization. • How Codecov can benefit Quality Assurance teams. CONFIDENTIAL
6. How does Codecov Help QA Teams? • Codecov enforces strong testing practices in engineering teams. • This ensures that less bugs make it to production, making it easier for QA teams to focus on testing more complex issues that are difficult to catch with automated testing. • Gives QA teams a shared language with engineering teams they can use to discuss code quality concerns. • Helps reduce the time lost via manual regression testing.
7. How does Codecov Help QA Teams? • Finally, no one wants to ship buggy software to end users. Codecov helps with that goal. • Lack of code coverage results in missing preventable bugs before shipping to production, which impacts user experience and developer productivity.
8. What is Code Coverage? • Code coverage is a metric / health score that can help you understand how much of your source code is tested. Lines of Code Tested Total Lines of Code = Code Coverage %
9. Why is Code Coverage Important? • Coverage is one of the most important health metrics companies rely on to determine code quality. • It helps developers surface areas of their codebase that are left untested before code is deployed and helps reduce the risk of shipping code that may cause errors downstream. • If AI is generating A LOT of code, teams must know if it is high quality code - Code Coverage Helps
10. Example of the Software Development Life Cycle Test / “CI” Release / “CD” Develop / “Git” Monitor Sentry’s got you on this one (java or kotlin) (swift or objective-C) (react-native) Unit + Integration + E2E E.g. app/play store
11. Example of the Software Development Life Cycle Test / “CI” Release / “CD” Develop / “Git” Monitor Sentry’s got you on this one (java or kotlin) (swift or objective-C) (react-native) Unit + Integration + E2E E.g. app/play store This is where Codecov lives
12. The Types of Tests (i.e., the testing pyramid) End-to-End Integration Unit Slower, more expensive Faster, cheaper subtract(4, 2) === 2 add (subtract(4, 1), 2) === 5
13. The Types of Tests (i.e., the testing pyramid) End-to-End Integration Unit Slower, more expensive Faster, cheaper • Codecov Can work with any testing strategy that produces a coverage report. • It can also work with testing strategies that employ some/all of these approaches at once. ✅ ✅ ✅
14. How does Codecov Work? • Codecov integrates with your SCM (Github, Bitbucket, Gitlab) and provides code coverage metrics on a Pull Request. • Developers upload code coverage reports to Codecov as part of their CI • Codecov then surfaces a code coverage percentage against your source code
15. How does Codecov Work? • Codecov provides automatic checks to ensure that coverage is high enough. ◦ These checks can be modified by the team. ◦ Patch: a check that ensures the contributed code is well tested ◦ Project: a check that ensures the coverage of the entire project is well-maintained.
16. What is a Pull Request? • A code change submitted to a project by a developer • Can be reviewed by another team member. Checks for: ◦ Code correctness ◦ Were tests written? ◦ Were enough tests written? ▪ This is where Codecov helps
17. AI Powered Code Review AI Powered Code Review - Open Beta • Integrated with Sentry Machine Learning ▪ Think of this as a PR first pass ▪ Bot instantly reviews your PR ▪ Get recommendations on low complexity problems ▪ Human reviewers can now focus on more complex problems of code change ▪ Minimizes time to first review, shortens review cycles
18. How does Codecov Work? • Advanced: Using Checks that apply to only portions of the codebase ◦ Ex: A large monorepo may want to use different checks for different projects in the monorepo. ◦ Codecov can do this using Components ◦ Consider two projects in a monorepo, Project A and Project B
19. How does Codecov Work? • Components, like all aspects of Codecov, are configurable as code in a codecov.yml file placed in the root of the repository. component_management: default_rules: statuses: - type: project target: auto individual_components: - component_id: project_a name: ProjectA paths: - "!conftest.py" - "!**/conftest.py" - "!**tests**/test_*.py" - "!database/tests/factories/**" - component_id: project_b name: ProjectB paths: - "!tasks/**" flag_regexes: - "unit"