$30 off During Our Annual Pro Sale. View Details »

Python, Government, and Contracts

José Padilla
October 05, 2019

Python, Government, and Contracts

Gave this presentation with Froilan Irizarry at PyGotham 2019

---

In the years after Hurricane Maria there has been a sharp increase in investigative journalism in Puerto Rico. Most of these investigations have been around questionable contracts awarded by different government entities and officials. We've used Python and Django to create a tool that consumes an unwieldy government site to make this data more accessible to journalist and citizens.

In the two years after Hurricane Maria struck Puerto Rico there has been a sharp increase in investigative journalism on the island. Most of these investigations have been around questionable contracts awarded by different government entities and officials in Puerto Rico. No-bid contracts like the ones award to White Fish and later to Cobra Acquisitions have generated national outrage. Lack of transparency, costly mismanagement, and abuse of power can be found throughout.

The Office of the Comptroller of Puerto Rico has built a tool that lets you search and download contract documents (after they’ve been redacted of sensitive information). While it serves its purpose of making this information available, the tool does not permit any mass data analysis or download. In less than a week we were able to release a very early version of our tool to improve this.

Our goal was to present a "better" way to search and use this data, create a sustainable project that could be easily used, while also keeping effort and cost to a minimum. For all these reasons we landed on using Python and Django. Django and the ecosystem around it presents an amazing opportunity to minimize effort and cost for a civic tech project while providing a robust way to manage the processes and data of a project like ContratosPR. Django's admin give use a central place to glance which contract documents need to be requested, OCR'ed, and provides the tools to execute the request process. All of this in combination with great docs and an amazing community made them a clear choice.

José Padilla

October 05, 2019
Tweet

More Decks by José Padilla

Other Decks in Technology

Transcript

  1. Python,
    Government,
    and Contracts

    View Slide

  2. A tool to
    investigate
    corruption

    View Slide

  3. Hurricane
    María

    View Slide

  4. View Slide

  5. September 26, 2017

    View Slide

  6. 95% of the island
    had no cell
    phone service.
    Hector Retamal / AFP / Getty Images

    View Slide

  7. Less than half of
    the population
    had tap water.
    El Vocero

    View Slide

  8. 95% of the island
    was without
    power.
    Joe Raedle/Getty Images

    View Slide

  9. October 26, 2017

    View Slide

  10. 58% still
    had no cell
    phone service.
    Hector Retamal / AFP / Getty Images

    View Slide

  11. 44% still
    had no
    water service.
    Ricardo Arduengo / AFP / Getty Images

    View Slide

  12. 89% still
    had no
    power.
    AP Images

    View Slide

  13. Three months later...

    View Slide

  14. 6.5% still
    had no cell
    phone service.
    El Nuevo Día

    View Slide

  15. 14% still
    had no
    water service.
    HECTOR RETAMAL (AFP)

    View Slide

  16. 45% still
    had no
    power...
    US Government

    View Slide

  17. ...that's over
    1.5 million
    people.
    Angelina Ruiz-Lambides

    View Slide

  18. José Padilla
    jpadilla_

    View Slide

  19. Work
    Open source
    Community

    View Slide

  20. Work
    Open source
    Community
    jpadilla

    View Slide

  21. Work
    Open source
    Community

    View Slide

  22. Froilán Irizarry
    skfroi

    View Slide

  23. Work
    Open source
    Community

    View Slide

  24. Work
    Open source
    Community
    froi

    View Slide

  25. Work
    Open source
    Community

    View Slide

  26. The Maria
    Tech Brigade

    View Slide

  27. View Slide

  28. statuspr.jpadilla.com

    View Slide

  29. Why are we
    talking about
    what happened
    two years ago?

    View Slide

  30. Suspicious
    contracts

    View Slide

  31. The Puerto Rico
    Power Authority
    awarded a...

    View Slide

  32. ...$300M
    contract to
    Whitefish Energy
    Holdings.

    View Slide

  33. It was a
    no-bid
    contract.
    Suspicious
    Should we add more context as to
    why this is suspicious.

    View Slide

  34. It stipulated
    that FEMA had
    approved it...
    Suspicious

    View Slide

  35. FEMA
    denied any
    involvement.
    Suspicious

    View Slide

  36. Puerto Rico
    could not audit
    the contract.
    Suspicious

    View Slide

  37. View Slide

  38. The Puerto Rico
    Power Authority
    then awarded a...

    View Slide

  39. ...$200M
    contract to Cobra
    Acquisitions...

    View Slide

  40. ...followed by
    another $900M
    contract.

    View Slide

  41. The company was
    founded during
    Q2 of 2017…
    Suspicious

    View Slide

  42. ...making it
    younger than
    Whitefish.
    Suspicious

    View Slide

  43. They had no other
    storm-related
    contracts.
    Suspicious

    View Slide

  44. The deal had
    heavy input from
    FEMA.
    Suspicious

    View Slide

  45. Cobra had no
    prior dealings
    with PREPA
    Suspicious

    View Slide

  46. Suspicious

    View Slide

  47. Office of the
    Comptroller of
    Puerto Rico

    View Slide

  48. View Slide

  49. API

    View Slide

  50. View Slide

  51. ContratosPR.com

    View Slide

  52. View Slide

  53. View Slide

  54. View Slide

  55. Why
    Python +
    Django?

    View Slide

  56. Batteries included

    View Slide

  57. View Slide

  58. View Slide

  59. Ecosystem

    View Slide

  60. View Slide

  61. View Slide

  62. All things data

    View Slide

  63. View Slide

  64. View Slide

  65. Community

    View Slide

  66. View Slide

  67. Why have
    two projects?

    View Slide

  68. contratospr-api

    View Slide

  69. View Slide

  70. contratospr

    View Slide

  71. View Slide

  72. Data Collection

    View Slide

  73. Scraping HTML
    $

    View Slide

  74. View Slide

  75. Slow and steady

    View Slide

  76. Celery for fun
    and profit

    View Slide

  77. Collecting

    View Slide

  78. Normalizing

    View Slide

  79. Storing

    View Slide

  80. Downloading

    View Slide

  81. View Slide

  82. Extracting

    View Slide

  83. Open source and
    community
    efforts

    View Slide

  84. Community

    View Slide

  85. Contributor
    Experience

    View Slide

  86. Transparency

    View Slide

  87. Financial
    sustainability

    View Slide

  88. What's next?

    View Slide

  89. Download
    all contract
    documents.

    View Slide

  90. Create more
    insights and
    visualizations.

    View Slide

  91. Registry of
    Corporations
    and Entities

    View Slide

  92. Interviews

    View Slide

  93. Engage with
    all of you.

    View Slide

  94. AP / iCARLOS GIUSTI

    View Slide

  95. ¡Gracias!
    José Padilla
    jpadilla_
    Froilán Irizarry
    skfroi
    contratospr.com/about

    View Slide