PyConZA 2014: "Writing Python Code to Decide an Election" by Peter Lubell-Doughtie

Earlier this year Ona was given three weeks to write the software that will tally votes in the Libyan elections and decide who wins and who loses. This is not something we could get wrong. We combined agile development with best practices in testing and QA to build an open source tally system that was well tested, accurate, and easy to use. We will describe a success story of iterative behavior/test-driven-development under extreme conditions. Did the structure of the data change the day before the election? Yes. Did we have the tests to ensure that our implementation changes would not compromise the system's integrity? Yes, and they didn't.

This talk provides a narrative to both Software Engineers and Tech/Product Managers describing why best practices are essential for any organization and any project of any size. We will provide the audience with:

1. Real world examples they can implement in their own workflow and organizations,
2. Insight into what succeeded (quick iteration with prioritization) and what was challenging (nothing being static),
3. Anecdotes and coherent arguments they can take back to their organization to advocate for best practices.

Transcript

1. Writing Python Code to Decide an Election Peter Lubell-Doughtie
2. http://www.ly.undp.org/etc/designs/ UNDPIndiaDesign/img/home/logo- undp.png
6. the problem • tally many votes from many districts in many elections !
7. the problem • tally many votes from many districts in many elections • elections may span multiple district
8. the problem • tally many votes from many districts in many elections • elections may span multiple district • each ballot has a 7-stage verification process
9. Intake
10. Intake Data Entry
11. Intake Data Entry Corrections
12. Intake Data Entry Corrections Quality Control
13. Intake Data Entry Corrections Quality Control Archiving
14. …the other problem • we had 3 weeks
15. …the other problem • we had 3 weeks
16. only the essentials
17. only the essentials • what functionality must exist? !
18. only the essentials • what functionality must exist? • what tools already do some of that?
19. only the essentials • what functionality must exist? • what tools already do some of that? • what if we miss something?
20. a schema as a tool
21. a schema as a tool • decide how data is organized ! !
22. a schema as a tool • decide how data is organized • shared reference !
23. a schema as a tool • decide how data is organized • shared reference • scopes possible relationships
24. a schema as a tool • decide how data is organized • shared reference • scopes possible relationships • it’s updatable
25. issues and milestones
26. issues and milestones • convert discussion to issues !
27. issues and milestones • convert discussion to issues • decide prioritization through milestones
28. issues and milestones • convert discussion to issues • decide prioritization through milestones • communicate to what will get done when
29. do it
30. do it 1. assign an issue ! ! ! ! !
31. do it 1. assign an issue 2. write tests for the issue ! ! ! !
32. do it 1. assign an issue 2. write tests for the issue 3. write code for the tests ! ! !
33. do it 1. assign an issue 2. write tests for the issue 3. write code for the tests 4. issue a pull request ! !
34. do it 1. assign an issue 2. write tests for the issue 3. write code for the tests 4. issue a pull request 5. review the code !
35. do it 1. assign an issue 2. write tests for the issue 3. write code for the tests 4. issue a pull request 5. review the code 6. pass the CI
36. do it 1. assign an issue 2. write tests for the issue 3. write code for the tests 4. open a pull request 5. review the code 6. pass the CI 7. merge and go to step (1.)
37. automate speed
38. automate speed • anything that is done twice gets a script
39. automate speed • anything that is done twice gets a script • anything that breaks once gets a test
40. we used south
41. we used postgres
42. we used postgres
43. nose, flake8, and coverage
44. travis for ci
45. do not write code
46. do not write code • unless you really really have to ! ! !
47. guardian for permissions
48. enumfield state machines
49. enumfield state machines
50. reversion for data purity
51. fabric for remote execution
52. plan to handle what you have not planned for
53. plan to handle what you have not planned for • elections can span multiple districts
54. plan to handle what you have not planned for • elections can span multiple districts • did not know until the first day of live testing
55. plan to handle what you have not planned for
56. plan to handle what you have not planned for • we changed the data model => south ! !
57. plan to handle what you have not planned for • we changed the data model => south • we wrote tests => django-nose !
58. plan to handle what you have not planned for • we changed the data model => south • we wrote tests => django-nose • we changed the code => travis-ci
59. plan to handle what you have not planned for • we changed the data model => south • we wrote tests => django-nose • we changed the code => travis-ci • we deployed => fabric
60. sage advice • “simplicity is prerequisite for reliability” - Dijkstra
61. references • http://south.readthedocs.org/ • https://github.com/5monkeys/django-enumfield • http://www.fabfile.org/ • https://flake8.readthedocs.org/ • https://bitbucket.org/ned/coveragepy
62. thank you! • https://github.com/onaio/tally-ho • http://company.ona.io/ • http://peet.ldee.org/
63. peter@ona.io