Python, Government, and Contracts

Python, Government, and Contracts

Gave this presentation with Froilan Irizarry at PyGotham 2019

---

In the years after Hurricane Maria there has been a sharp increase in investigative journalism in Puerto Rico. Most of these investigations have been around questionable contracts awarded by different government entities and officials. We've used Python and Django to create a tool that consumes an unwieldy government site to make this data more accessible to journalist and citizens.

In the two years after Hurricane Maria struck Puerto Rico there has been a sharp increase in investigative journalism on the island. Most of these investigations have been around questionable contracts awarded by different government entities and officials in Puerto Rico. No-bid contracts like the ones award to White Fish and later to Cobra Acquisitions have generated national outrage. Lack of transparency, costly mismanagement, and abuse of power can be found throughout.

The Office of the Comptroller of Puerto Rico has built a tool that lets you search and download contract documents (after they’ve been redacted of sensitive information). While it serves its purpose of making this information available, the tool does not permit any mass data analysis or download. In less than a week we were able to release a very early version of our tool to improve this.

Our goal was to present a "better" way to search and use this data, create a sustainable project that could be easily used, while also keeping effort and cost to a minimum. For all these reasons we landed on using Python and Django. Django and the ecosystem around it presents an amazing opportunity to minimize effort and cost for a civic tech project while providing a robust way to manage the processes and data of a project like ContratosPR. Django's admin give use a central place to glance which contract documents need to be requested, OCR'ed, and provides the tools to execute the request process. All of this in combination with great docs and an amazing community made them a clear choice.

Becd166a81dc51c0009f602d175d0cc8?s=128

José Padilla

October 05, 2019
Tweet

Transcript

  1. Python, Government, and Contracts

  2. A tool to investigate corruption

  3. Hurricane María

  4. None
  5. September 26, 2017

  6. 95% of the island had no cell phone service. Hector

    Retamal / AFP / Getty Images
  7. Less than half of the population had tap water. El

    Vocero
  8. 95% of the island was without power. Joe Raedle/Getty Images

  9. October 26, 2017

  10. 58% still had no cell phone service. Hector Retamal /

    AFP / Getty Images
  11. 44% still had no water service. Ricardo Arduengo / AFP

    / Getty Images
  12. 89% still had no power. AP Images

  13. Three months later...

  14. 6.5% still had no cell phone service. El Nuevo Día

  15. 14% still had no water service. HECTOR RETAMAL (AFP)

  16. 45% still had no power... US Government

  17. ...that's over 1.5 million people. Angelina Ruiz-Lambides

  18. José Padilla jpadilla_

  19. Work Open source Community

  20. Work Open source Community jpadilla

  21. Work Open source Community

  22. Froilán Irizarry skfroi

  23. Work Open source Community

  24. Work Open source Community froi

  25. Work Open source Community

  26. The Maria Tech Brigade

  27. None
  28. statuspr.jpadilla.com

  29. Why are we talking about what happened two years ago?

  30. Suspicious contracts

  31. The Puerto Rico Power Authority awarded a...

  32. ...$300M contract to Whitefish Energy Holdings.

  33. It was a no-bid contract. Suspicious Should we add more

    context as to why this is suspicious.
  34. It stipulated that FEMA had approved it... Suspicious

  35. FEMA denied any involvement. Suspicious

  36. Puerto Rico could not audit the contract. Suspicious

  37. None
  38. The Puerto Rico Power Authority then awarded a...

  39. ...$200M contract to Cobra Acquisitions...

  40. ...followed by another $900M contract.

  41. The company was founded during Q2 of 2017… Suspicious

  42. ...making it younger than Whitefish. Suspicious

  43. They had no other storm-related contracts. Suspicious

  44. The deal had heavy input from FEMA. Suspicious

  45. Cobra had no prior dealings with PREPA Suspicious

  46. Suspicious

  47. Office of the Comptroller of Puerto Rico

  48. None
  49. API

  50. None
  51. ContratosPR.com

  52. None
  53. None
  54. None
  55. Why Python + Django?

  56. Batteries included

  57. None
  58. None
  59. Ecosystem

  60. None
  61. None
  62. All things data

  63. None
  64. None
  65. Community ♥

  66. None
  67. Why have two projects?

  68. contratospr-api

  69. None
  70. contratospr

  71. None
  72. Data Collection

  73. Scraping HTML $

  74. None
  75. Slow and steady

  76. Celery for fun and profit

  77. Collecting

  78. Normalizing

  79. Storing

  80. Downloading

  81. None
  82. Extracting

  83. Open source and community efforts

  84. Community

  85. Contributor Experience

  86. Transparency

  87. Financial sustainability

  88. What's next?

  89. Download all contract documents.

  90. Create more insights and visualizations.

  91. Registry of Corporations and Entities

  92. Interviews

  93. Engage with all of you.

  94. AP / iCARLOS GIUSTI

  95. ¡Gracias! José Padilla jpadilla_ Froilán Irizarry skfroi contratospr.com/about