Python, Government, and Contracts

Gave this presentation with Froilan Irizarry at PyGotham 2019

---

In the years after Hurricane Maria there has been a sharp increase in investigative journalism in Puerto Rico. Most of these investigations have been around questionable contracts awarded by different government entities and officials. We've used Python and Django to create a tool that consumes an unwieldy government site to make this data more accessible to journalist and citizens.

In the two years after Hurricane Maria struck Puerto Rico there has been a sharp increase in investigative journalism on the island. Most of these investigations have been around questionable contracts awarded by different government entities and officials in Puerto Rico. No-bid contracts like the ones award to White Fish and later to Cobra Acquisitions have generated national outrage. Lack of transparency, costly mismanagement, and abuse of power can be found throughout.

The Office of the Comptroller of Puerto Rico has built a tool that lets you search and download contract documents (after they’ve been redacted of sensitive information). While it serves its purpose of making this information available, the tool does not permit any mass data analysis or download. In less than a week we were able to release a very early version of our tool to improve this.

Our goal was to present a "better" way to search and use this data, create a sustainable project that could be easily used, while also keeping effort and cost to a minimum. For all these reasons we landed on using Python and Django. Django and the ecosystem around it presents an amazing opportunity to minimize effort and cost for a civic tech project while providing a robust way to manage the processes and data of a project like ContratosPR. Django's admin give use a central place to glance which contract documents need to be requested, OCR'ed, and provides the tools to execute the request process. All of this in combination with great docs and an amazing community made them a clear choice.

Transcript

1. Python, Government, and Contracts
2. A tool to investigate corruption
3. Hurricane María
5. September 26, 2017
6. 95% of the island had no cell phone service. Hector Retamal / AFP / Getty Images
7. Less than half of the population had tap water. El Vocero
8. 95% of the island was without power. Joe Raedle/Getty Images
9. October 26, 2017
10. 58% still had no cell phone service. Hector Retamal / AFP / Getty Images
11. 44% still had no water service. Ricardo Arduengo / AFP / Getty Images
12. 89% still had no power. AP Images
13. Three months later...
14. 6.5% still had no cell phone service. El Nuevo Día
15. 14% still had no water service. HECTOR RETAMAL (AFP)
16. 45% still had no power... US Government
17. ...that's over 1.5 million people. Angelina Ruiz-Lambides
18. José Padilla jpadilla_
19. Work Open source Community
20. Work Open source Community jpadilla
21. Work Open source Community
22. Froilán Irizarry skfroi
23. Work Open source Community
24. Work Open source Community froi
25. Work Open source Community
26. The Maria Tech Brigade
28. statuspr.jpadilla.com
29. Why are we talking about what happened two years ago?
30. Suspicious contracts
31. The Puerto Rico Power Authority awarded a...
32. ...$300M contract to Whitefish Energy Holdings.
33. It was a no-bid contract. Suspicious Should we add more context as to why this is suspicious.
34. It stipulated that FEMA had approved it... Suspicious
35. FEMA denied any involvement. Suspicious
36. Puerto Rico could not audit the contract. Suspicious
38. The Puerto Rico Power Authority then awarded a...
39. ...$200M contract to Cobra Acquisitions...
40. ...followed by another $900M contract.
41. The company was founded during Q2 of 2017… Suspicious
42. ...making it younger than Whitefish. Suspicious
43. They had no other storm-related contracts. Suspicious
44. The deal had heavy input from FEMA. Suspicious
45. Cobra had no prior dealings with PREPA Suspicious
46. Suspicious
47. Office of the Comptroller of Puerto Rico
49. API
51. ContratosPR.com
55. Why Python + Django?
56. Batteries included
59. Ecosystem
62. All things data
65. Community ♥
67. Why have two projects?
68. contratospr-api
70. contratospr
72. Data Collection
73. Scraping HTML $
75. Slow and steady
76. Celery for fun and profit
77. Collecting
78. Normalizing
79. Storing
80. Downloading
82. Extracting
83. Open source and community efforts
84. Community
85. Contributor Experience
86. Transparency
87. Financial sustainability
88. What's next?
89. Download all contract documents.
90. Create more insights and visualizations.
91. Registry of Corporations and Entities
92. Interviews
93. Engage with all of you.
94. AP / iCARLOS GIUSTI
95. ¡Gracias! José Padilla jpadilla_ Froilán Irizarry skfroi contratospr.com/about