Python, Government, and Contracts: providing a tool to investigate corruption

In the years after Hurricane Maria there has been a sharp increase in investigative journalism in Puerto Rico. Most of these investigations have been around questionable contracts awarded by different government entities and officials. We've used Python and Django to create a tool that consumes an unwieldy government site to make this data more accessible to journalist and citizens.

You can see the talk here: https://youtu.be/Kxu5zMh1bHA
What the speaker notes? Here you go: https://www.dropbox.com/s/x60msk60b8clyop/nbpy-python-governments-contracts-with-speaker-notes.pdf?dl=0

Transcript

1. Python, Government, and Contracts
2. A tool to investigate corruption
3. Hurricane María
5. September 26, 2017
6. 95% of the island had no cell phone service. Hector Retamal / AFP / Getty Images
7. Less than half of the population had tap water. El Vocero
8. 95% of the island was without power. Joe Raedle/Getty Images
9. October 26, 2017
10. 58% still had no cell phone service. Hector Retamal / AFP / Getty Images
11. 44% still had no water service. Ricardo Arduengo / AFP / Getty Images
12. 89% still had no power. AP Images
13. Three months later...
14. 6.5% still had no cell phone service. El Nuevo Día
15. 14% still had no water service. HECTOR RETAMAL (AFP)
16. 45% still had no power... US Government
17. ...that's over 1.5 million people. Angelina Ruiz-Lambides
18. Froilán Irizarry Rivera skfroi
19. Work Open source Community
20. Work Open source Community froi
21. Work Open source Community
22. José Padilla jpadilla_
23. The Maria Tech Brigade
25. statuspr.jpadilla.com
26. Why are we talking about what happened two years ago?
27. Suspicious contracts
28. The Puerto Rico Power Authority awarded a...
29. ...$300M contract to Whitefish Energy Holdings.
30. It stipulated that FEMA had approved it... Suspicious
31. FEMA denied any involvement. Suspicious
32. Puerto Rico could not audit the contract. Suspicious
34. The Puerto Rico Power Authority then awarded a...
35. ...$200M contract to Cobra Acquisitions...
36. ...followed by another $900M contract.
37. The company was founded during Q2 of 2017… Suspicious
38. ...making it younger than Whitefish. Suspicious
39. They had no other storm-related contracts. Suspicious
40. The deal had heavy input from FEMA. Suspicious
41. Cobra had no prior dealings with PREPA Suspicious
42. Suspicious
43. Office of the Comptroller of Puerto Rico
45. API
47. ContratosPR.com
51. Why Python + Django?
52. Batteries included
55. Ecosystem
58. All things data
61. Community ♥
63. Why have two projects?
64. contratospr-api
66. contratospr
68. Data Collection
69. Scraping HTML
71. Slow and steady
72. Celery for fun and profit
73. Collecting
74. 1.2 million contracts Collected
75. Normalizing
76. Storing
77. Downloading 195,582 documents
79. Extracting
80. Open source and community efforts
81. Community
82. Contributor Experience
83. Transparency
84. Financial sustainability
85. What's next?
86. Process all contract documents.
87. Create more insights and visualizations.
88. Registry of Corporations and Entities
89. Interviews
90. Engage with all of you.
91. AP / iCARLOS GIUSTI
92. ¡Gracias! José Padilla jpadilla_ lán Irizarry skfroi contratospr.com/about