Agile + Python Pyconph 2014

Transcript

1. Agile + Python Pycon PH 2014 Ronald Cheung Tuesday, February 25, 14
2. • Objective of this talk is a to give a beginners introduction to Agile to provide an Agile perspective for today’s workshops Tuesday, February 25, 14
3. • About me • Java / J2EE IBM / Oracle / Microsoft “Traditional” waterfall methodologies Tuesday, February 25, 14
4. • Joined a Startup company • Python + Django Postgresql Apache / Ngnix Linux Agile Tuesday, February 25, 14
5. • First learnt Python / Django in 2011 • Was able to be productive ( felt like kicking ass ) within a week • First project completed and released within 6 weeks Tuesday, February 25, 14
6. Agile • Agile is a Culture * set of beliefs and goals • Thesis: There is no “best” software development methodology, it depends on the situation Tuesday, February 25, 14
7. Agile Manifesto - Utah 2001 Values • Individuals and interaction over processes and tools • Working software over comprehensive documentation • Customer Collaboration over contract negotiation • Responding to change over following a plan Tuesday, February 25, 14
8. Agile Manifesto - Principles Our highest priority is to satisfy the customer through early and continuous delivery of valuable software Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage Scope should be fixed. Changes to scope should be challenged Everything should be defined upfront before software can be built Tuesday, February 25, 14
9. Agile Manifesto - Principles Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale Business people and developers must work together daily throughout the project Working software is delivered to the customer at the end Business people write documents for analysts who write documents for developers. Silos. Tuesday, February 25, 14
10. Agile Manifesto - Principles Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done The most efficient and effective method of conveying information to and within a development team is face-to-face conversation Following the process is more important than the individuals in the project Can you send me an email Tuesday, February 25, 14
11. Agile Manifesto - Principles Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely Project is progressing along great, look here are detailed gantt charts and business specifications Major deadline looming it’s crunch time. Everyone needs to work 20h days Tuesday, February 25, 14
12. Agile Manifesto - Principles Continuous attention to technical excellence and good design enhances agility Simplicity--the art of maximizing the amount of work not done--is essential Hacks, workarounds, technical debt is ok as long as the software functions The software must handle all possible eventualities Tuesday, February 25, 14
13. Agile Manifesto - Principles The best architectures, requirements, and designs emerge from self-organizing teams At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. The software architect knows designs everything. One individual knows it all. Onward we go to the next project! Tuesday, February 25, 14
14. Traditional “Waterfall” Conception Initiation Analysis Design Construction (Yay we finally get to code ) Testing Implementation Maintenance Keywords: Sequential Predictive Planning Time: Feb 22 What we think we need by May 2014 ... ... Real world stuff changes ( inconvenient! ) .. ... Project Released to Market May 2014 Tuesday, February 25, 14
15. Agile Brainstorm * Design * Build * QA * Release to market Brainstorm * Design * Build * QA * Release to market Brainstorm * Design * Build * QA * Release to market Brainstorm * Design * Build * QA * Release to market Time Feb 22 ... Mar 22 .. ... Apr 22 .. .. May 22 ... Keywords: Iterative Adaptive Discovery Discovery: Customers never use feature X Discovery: Major competitor just released amazing feature Y Tuesday, February 25, 14
16. Agile Practices • Product Backlogs / Sprint Backlogs • Cross functional teams • Continuous Integration • SCRUM • Burndown charts • Pair Programming • Planning Poker • Test Driven Development • Timeboxing • User stories • Story points • Velocity • Automated testing • Code coverage analysis Tuesday, February 25, 14
17. User Stories • Story is a unit of functionality that is valuable to a user • Includes the following elements: Who, What, Why • As a < stakeholder> I want to <objective> so that < reason > Tuesday, February 25, 14
18. User Stories • Independent • Negotiable • Valuable • Estimable • Small • Testable Tuesday, February 25, 14
19. • “As a blogger I want the length of the title field increased to 200 characters so I can write more descriptive titles” • “As a marketing manager I want to add a button to allow bloggers to share their content to twitter to give my content more visibility” • “As a blogger I want to be able to import my blogs from other blogging platforms so I can migrate my content to this platform” Tuesday, February 25, 14
20. Acceptance Criteria • Stories include exit criteria: What conditions need to be met in order for the story to be completed? • Contains 3 elements: 1. Actor 2. Verb to describe behavior 3. Observable and testable result • Example: 1. A blog reader visits the article 2. Clicks share to twitter 3. The link and description of the article is posted to the blog reader’s twitter account Tuesday, February 25, 14
21. User story Acceptance criteria Get title and URL from page Authenticate with Twitter API Post to Twitter stream API Show confirmation page • User stories are broken down into tasks by team members • Tasks need to be broken down into smaller tasks until they can be Estimated Tuesday, February 25, 14
22. Estimation • How much time / effort / resources is required to complete the User Story? • Traditional estimation often involves estimating a task in absolute terms for example number of hours / days. • Can be a difficult task for humans to do. Tuesday, February 25, 14
23. • How tall is the tallest building in meters? • Hard to determine Tuesday, February 25, 14
24. • What is the size of a building relative to another in story points? Tuesday, February 25, 14
25. • “As a blogger I want the length of the title field increased to 200 characters so I can write more descriptive titles” - 1 story point • “As a marketing manager I want to add a button to allow bloggers to share their content to twitter to give my content more visibility” - 3 story points • “As a blogger I want to be able to import my blogs from other blogging platforms so a can switch to this platform” - 9 story points Tuesday, February 25, 14
26. • How are story points helpful for estimating the size of a project if story points are relative? • As a development team works over many iterations or sprints, over time the number of story points can be measured and the number of story points completed per sprint can be computed. This number is the velocity. • By comparing the sum of the story points and velocity it can be estimated what stories can be completed within an iteration / sprint. Tuesday, February 25, 14
27. Test Driven Development ( TDD) • Iterative development often results in rapid changes to the codebase and frequent code refactoring is necessary • Automated testing allows major code refactoring to be done with confidence Tuesday, February 25, 14
28. Test Driven Development • “Move fast and break things” ( Facebook motto 2007 ) • Better: “Move fast and improve quality at the same time” Tuesday, February 25, 14
29. Test Driven Development ( TDD) • Automated tests are written before or at the same time as when production code is written • Tests are usually written to initially fail. Code passes unit testing once all tests are passed Tuesday, February 25, 14
30. Test Driven Development ( TDD) • Without automated tests: - In the future, pre-conditions may be changed - Developers other than the original author need to refactor the code • Errors introduced are not detected Tuesday, February 25, 14
31. TDD Cadence • Test -> Code -> Refactor Write Test Run Test Write Code Pass Code change Fail Run Test Fail Refactor Pass Tuesday, February 25, 14
32. Unit Testing User story Acceptance Criteria Get title and URL from page Authenticate with Twitter API Post to Twitter stream API Show confirmation page Test title property is returned Test URL property returned .... Tuesday, February 25, 14
33. Automated Testing Hierarchy of needs Unittest, nose, doctest, pytest, unittest2 Twill, webunit, Selenium GUI / Web Testing Unit Tests coverage.py, figleaf Code Coverage Continuous Integration Buildbot Automated testing is based on a foundation of lots and lots of unit tests “maslows hierarchy of needs” Tuesday, February 25, 14
34. Python Unittest module • Python’s integrated unit testing module • Concepts: - Test Fixture: Initialize pre-conditions for tests for example create data structures, populate databases - Test Case: Smallest unit of testing - Test Runner: Framework that executes all relevant unit test and collects the test results Tuesday, February 25, 14
35. Unit test case import unittest class myUnitTest(unittest.TestCase): def setUp(self): pass def test_numbers_3_4(self): self.assertEqual( multiply(3,4), 12) def test_strings_a_3(self): self.assertEqual( multiply('a',3), 'aaa') def tearDown(self): pass if __name__ == '__main__': unittest.main() • Fixtures: setUp() executes first to initialize preconditions tearDown() executes after test to cleanup • Unit test class inherits from unittest.TestCase • basic assert methods: assertEqual(a, b) assertNotEqual(a, b) assertTrue(x) assertFalse(x) Tuesday, February 25, 14
36. Running the unit test > python test_my_unittest.py .. --------------------------------------------------------------------- - Ran 2 tests in 0.000s OK > python test_my_unittest.py -v test_numbers_3_4 ... ok test_strings_a_3 ... ok --------------------------------------------------------------------- - Ran 2 tests in 0.000s OK • Any assert statement that returns false will be reported as a failure Tuesday, February 25, 14
37. Test Runner • Tedious to run each test individually • Nose combine test discovery with test running • Also integrates with code coverage tools Tuesday, February 25, 14
38. Nose • > pip install nose > ls -- myprojectdir/ ---- blog_app/ ------ blog_tests.py ---- email_app/ ------- email_tests.py > cd myprojectdir/ > nosetests > --------------------------------------------------------------------- - > blog_app.test_blog_title_1 ... ok > blog_app.test_blog_url_2 ... ok > blog_app.test_blog_share_3 ... ok > email_app.test_email_send_1 ... ok > email_app.test_email_receive_1 ... ok --------------------------------------------------------------------- - Ran 5 tests in 0.500s > nosetests --with-coverage > nosetests --with-figleaf • Nose will collect any class that is a subclass of unittest.testcase • naming conventions: Naming test_<app>.py or putting test classes within directories called “test” will help nose discover all tests. • Nose has a plugin system that integrates with other modules such as code coverage tools Tuesday, February 25, 14
39. GUI / Web Testing Unit Tests Code Coverage Continuous Integration Tuesday, February 25, 14
40. Code Coverage • Percentage of the source code tested by the test suite • Use code coverage tools to identify which areas of the code base require more unit tests. Tuesday, February 25, 14
41. Figleaf > pip install figleaf > figleaf myapp.py > figleaf2html • Minor Caveat: Figleaf will produce a metrics based on line coverage ( not branch coverage ). This means that even if 100% coverage is reported the code may not be 100% bug free • Running figleaf from command line will generate a coverage database in .figleaf • figleaf2html generates a HTML report from the .figleaf database Tuesday, February 25, 14
42. Line Coverage 1. if a.x or a.y: 2. func() • Minor Caveat: Figleaf will produce a metrics based on line coverage ( not branch coverage ). This means that even if 100% coverage is reported the code may not be 100% bug free • For example if a.x evaluates to True in all unit test, then a.y will never be evaluated. Although Line 1. will be marked as covered • Code Coverage tools do not measure the quality of unit test. Tuesday, February 25, 14
43. coverage.py > pip install coverage > coverage run myapp.py > coverage report $ coverage report -m Name Stmts Miss Cover Missing ------------------------------------------------------- my_program 20 4 80% 33-35, 39 my_other_module 56 6 89% 17-23 ------------------------------------------------------- TOTAL 76 10 87% > coverage html • Figleaf loosely based from coverage.py, both are very acceptable tools • For basic reporting on unit tests, coverage may be simpler to use • Figleaf may be more suitable when tests are executed over multiple runs and provides aggregated reporting. Tuesday, February 25, 14
44. GUI / Web Testing Unit Tests Code Coverage Continuous Integration Tuesday, February 25, 14
45. GUI / Web Testing • Testing from the perspective of the user / business • Can detect potential integration issues not covered by unit tests • Browser simulation: simulates browsers by implementing the HTTP request response protocol and parsing the response HTML Tuesday, February 25, 14
46. Twill go http://pycon.python.ph/ code 200 #assert http 200 title ‘PyCon Philippines 2014’ #test for title element find ‘The main goal of this conference is to provide a venue where the Python Programming language and surrounding technologies can be explored, discussed, and exercised’ • Python Scripting language to simulate browsing • Can simulate logins, form handling / submission, authentication • Has a Python API from twill import get_browser b = get_browser() from twill.commands import * go(‘http://pycon.python.ph/’) Tuesday, February 25, 14
47. GUI / Web Testing Unit Tests Code Coverage Continuous Integration Tuesday, February 25, 14
48. Continuous Integration • Putting everything together ( Agile Self- Actualization ) • Unit tests, code coverage reports, GUI tests are automatically run when a developer commits to a repo ( typically into master or baseline branch) Tuesday, February 25, 14
49. Continuous Integration Code committed to repo Automated Unit Tests Code Coverage GUI / Web Tests Broken Build Alert Code Coverage Report User Acceptance Report Automated Deployment Tuesday, February 25, 14
50. Continuous Integration • Broken Builds, bugs and other issues are detected as soon as code is committed • Commit small changes, Commit frequently ( At least once per day, preferably more ) • Code merge conflicts are constantly being resolved, rather than a large conflict resolution task at the last minute • The latest build is always available ( automated deployment ) • Metrics generated ( code coverage report, broken build / failed unit test reports ) provide constant feedback to the team to measure quality Tuesday, February 25, 14
51. Buildbot • Buildbot - Continuous Integration framework built on Python / Twisted • Buildbot includes components to: - Poll your repo of choice for changes - Builds the application on the target platform ( linux, windows, solaris ) - Runs automated unit tests, code coverage reports, web tests - Co-ordinates notification ( email, SMS, IRC ) Tuesday, February 25, 14
52. Buildbot Version Control ( Repo ) Buildmaster Windows build environment Linux build environment Solaris Build Environment Build Slaves SMS Email IRC Polls Version Control for updates Coordinates notification Initiates builds Tuesday, February 25, 14
53. Tuesday, February 25, 14
54. Summary Continuous Integration Buildbot GUI Web Testing Twill Code Coverage Figleaf, coverage.py Automated Unit Testing Unittest, nose Tuesday, February 25, 14
55. • Thesis: There is no “best” software development methodology, it depends on the situation • Agile is designed to manage the cost of change and is suitable for projects that may evolve rapidly Cost of Change Tuesday, February 25, 14
56. Thanks!! • ronaldlcheung@gmail.com • @ronaldlcheung • http://www.linkedin.com/in/ronaldlcheung Tuesday, February 25, 14
57. • Save22 is a company specializing in the development of Price Intelligence products • We use Python / Django as our primary development language / framework • We have a team currently based in Manila • We are always looking for talented and motivated engineers. Please drop me a line: ronald@save22.com Advertisement Ahead: About Save22 Tuesday, February 25, 14