Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python, Government, and Contracts

Python, Government, and Contracts

Gave this presentation with Froilan Irizarry at PyCascades 2020

---

In the years after Hurricane Maria there has been a sharp increase in investigative journalism in Puerto Rico. Most of these investigations have been around questionable contracts awarded by different government entities and officials. We've used Python and Django to create a tool that consumes an unwieldy government site to make this data more accessible to journalist and citizens.

In the two years after Hurricane Maria struck Puerto Rico there has been a sharp increase in investigative journalism on the island. Most of these investigations have been around questionable contracts awarded by different government entities and officials in Puerto Rico. No-bid contracts like the ones award to White Fish and later to Cobra Acquisitions have generated national outrage. Lack of transparency, costly mismanagement, and abuse of power can be found throughout.

The Office of the Comptroller of Puerto Rico has built a tool that lets you search and download contract documents (after they’ve been redacted of sensitive information). While it serves its purpose of making this information available, the tool does not permit any mass data analysis or download. In less than a week we were able to release a very early version of our tool to improve this.

Our goal was to present a "better" way to search and use this data, create a sustainable project that could be easily used, while also keeping effort and cost to a minimum. For all these reasons we landed on using Python and Django. Django and the ecosystem around it presents an amazing opportunity to minimize effort and cost for a civic tech project while providing a robust way to manage the processes and data of a project like ContratosPR. Django's admin give use a central place to glance which contract documents need to be requested, OCR'ed, and provides the tools to execute the request process. All of this in combination with great docs and an amazing community made them a clear choice.

José Padilla

February 08, 2020
Tweet

More Decks by José Padilla

Other Decks in Technology

Transcript

  1. Python,
    Government,
    and Contracts

    View full-size slide

  2. A tool to
    investigate
    corruption

    View full-size slide

  3. Hurricane
    María

    View full-size slide

  4. September 26, 2017

    View full-size slide

  5. 95% of the island
    had no cell
    phone service.
    Hector Retamal / AFP / Getty Images

    View full-size slide

  6. Less than half of
    the population
    had tap water.
    El Vocero

    View full-size slide

  7. 95% of the island
    was without
    power.
    Joe Raedle/Getty Images

    View full-size slide

  8. October 26, 2017

    View full-size slide

  9. 58% still
    had no cell
    phone service.
    Hector Retamal / AFP / Getty Images

    View full-size slide

  10. 44% still
    had no
    water service.
    Ricardo Arduengo / AFP / Getty Images

    View full-size slide

  11. 89% still
    had no
    power.
    AP Images

    View full-size slide

  12. Three months later...

    View full-size slide

  13. 6.5% still
    had no cell
    phone service.
    El Nuevo Día

    View full-size slide

  14. 14% still
    had no
    water service.
    HECTOR RETAMAL (AFP)

    View full-size slide

  15. 45% still
    had no
    power...
    US Government

    View full-size slide

  16. ...that's over
    1.5 million
    people.
    Angelina Ruiz-Lambides

    View full-size slide

  17. José Padilla
    jpadilla_

    View full-size slide

  18. Work
    Open source
    Community

    View full-size slide

  19. Work
    Open source
    Community
    jpadilla

    View full-size slide

  20. Work
    Open source
    Community

    View full-size slide

  21. Froilán Irizarry
    froidotdev

    View full-size slide

  22. Work
    Open source
    Community

    View full-size slide

  23. Work
    Open source
    Community
    froi

    View full-size slide

  24. Work
    Open source
    Community

    View full-size slide

  25. The Maria
    Tech Brigade

    View full-size slide

  26. statuspr.jpadilla.com

    View full-size slide

  27. Why are we
    talking about
    what happened
    two years ago?

    View full-size slide

  28. Suspicious
    contracts

    View full-size slide

  29. The Puerto Rico
    Power Authority
    awarded a...

    View full-size slide

  30. ...$300M
    contract to
    Whitefish Energy
    Holdings.

    View full-size slide

  31. It was a
    no-bid
    contract.
    Suspicious

    View full-size slide

  32. It stipulated
    that FEMA had
    approved it...
    Suspicious

    View full-size slide

  33. FEMA
    denied any
    involvement.
    Suspicious

    View full-size slide

  34. Puerto Rico
    could not audit
    the contract.
    Suspicious

    View full-size slide

  35. The Puerto Rico
    Power Authority
    then awarded a...

    View full-size slide

  36. ...$200M
    contract to Cobra
    Acquisitions...

    View full-size slide

  37. ...followed by
    another $900M
    contract.

    View full-size slide

  38. The company was
    founded during
    Q2 of 2017…
    Suspicious

    View full-size slide

  39. ...making it
    younger than
    Whitefish.
    Suspicious

    View full-size slide

  40. They had no other
    storm-related
    contracts.
    Suspicious

    View full-size slide

  41. The deal had
    heavy input from
    FEMA.
    Suspicious

    View full-size slide

  42. Cobra had no
    prior dealings
    with PREPA
    Suspicious

    View full-size slide

  43. Office of the
    Comptroller of
    Puerto Rico

    View full-size slide

  44. ContratosPR.com

    View full-size slide

  45. Why
    Python +
    Django?

    View full-size slide

  46. Batteries included

    View full-size slide

  47. All things data

    View full-size slide

  48. Community

    View full-size slide

  49. Why have
    two projects?

    View full-size slide

  50. contratospr-api

    View full-size slide

  51. contratospr-web

    View full-size slide

  52. Data Collection

    View full-size slide

  53. Scraping HTML

    View full-size slide

  54. Slow and steady

    View full-size slide

  55. Celery for fun
    and profit

    View full-size slide

  56. Collecting
    976,833 contracts +
    270,156 amendments

    View full-size slide

  57. Downloading
    201,003 PDFs

    View full-size slide

  58. Open source and
    community
    efforts

    View full-size slide

  59. code4puertorico/contratospr
    code4puertorico/contratospr-web
    code4puertorico/contratospr-api

    View full-size slide

  60. Transparency

    View full-size slide

  61. http://bit.ly/
    contratospr-financials

    View full-size slide

  62. Nov. 2018 - Jan. 2020
    Grand Total
    $489.24 $254.67
    $743.91

    View full-size slide

  63. Date Description Amount Balance
    January 2020 Balance EOY 2019 -$10.50
    January 2020 Froi (Venmo) -$183.58 -$194.08
    January 2020 José reimbursement $183.58 -$10.50
    January 2020 Heroku hosting $21.64 $11.14
    January 2020 Compose database hosting $74.11 $85.25
    January 2020 José payment -$42.62 $42.63
    January 2020 Froi payment -$42.63 $0.00
    Total $0.00

    View full-size slide

  64. Financial
    sustainability

    View full-size slide

  65. What's next?

    View full-size slide

  66. Download
    all contract
    documents.

    View full-size slide

  67. Create more
    insights and
    visualizations.

    View full-size slide

  68. Registry of
    Corporations
    and Entities

    View full-size slide

  69. Engage with
    all of you.

    View full-size slide

  70. AP / iCARLOS GIUSTI
    Ricardo Arduengo/AFP/Getty Images
    Jose Jimenez / Getty Images Ricardo Arduengo/AFP/Getty Images
    Jose Jimenez/Getty Images
    Alejandro Granadillo/Anadolu Agency/Getty Images

    View full-size slide

  71. ¡Gracias!
    José Padilla
    jpadilla_
    Froilán Irizarry
    froidotdev
    contratospr.com/about

    View full-size slide