Handling Backend DB Model Design and Relationships, API Testing + Other Postman, Testing, Tools Used, Data Scraping/Sources, Dataviz, Mr. Bill Deployment Hosting, Deployment Process, GitLab CI integration Demo Live Demo, Questions? legislore.web.app legislore.me ★ ★ ★ ★ ★ ★
Legislation is historically difficult to read. LegisLore is designed to have an intuitive interface that supports quickly searching for relevant legislation information and navigating between tabs for congresspeople, bills, and committees. LegisLore can be accessed by anyone with an Internet connection, from historians who wish to gather third-party congressional data to civilians who are simply curious about the state of current US Congressional affairs.
customization ★ Pagination, integrating frontend with Backend API ★ Michelle ★ Phase III sorting/searching/filtering ★ Highlighting ★ Postman API Testing ★ Favicon Design ★ Mr. Bill ★ Great mascot ★ Patrick ★ Postman API Design ★ Debugging ★ Data Visualizations ★ Unit Testing ★ Isaac ★ API design + implementation + documentation ★ DB design + implementation + population ★ Unit tests ★ Data collection ★ Tim ★ Gitlab Issue Generator ★ Backend deployment ★ Ad-hoc bug-fixes legislore.web.app legislore.me
★ Site style using the Bulma CSS Framework, customized using Sass (example pictured) ★ Blue and red color scheme to fit the government look ★ Legal + Modern Look => Serif font for headings, Sans serif font for paragraph text $blue: #313b72; $red: #983628; $purple: #8a4d76; $brown: #493132; $beige-light: #d0d1cd; $beige-lighter: #efe9f4; // Update Bulma's global variables $family-sans-serif: "Lora", sans-serif; $primary: $blue; $grey-dark: $brown; $grey-light: $beige-light; $link: $red; $link-hover: $blue; legislore.web.app legislore.me
the built in JS fetch API ★ Requests are cached in the browser due to Cache Control headers // Try to make a request to the API. If it's up, let's use that! // If it's not, fall back to static json files. const getURLMaker = new Promise<Function>((resolve) => { fetch(API_URL).then(x => { if (x.status === 200) { // We have a server running! Let's use it! console.log("Using API!" ) resolve((path : string) => API_URL + path) } }).catch(error => { console.log(error) console.log("Error getting API data, falling back to static files..." ) resolve((path : string) => FALLBACK_URL + path.split("?")[0] + ".json") }) }) legislore.web.app legislore.me
(GCP Product) ★ Built and deployed using Gitlab CI Pipeline ★ Work on dev branch deploys to a staging site ★ Pushing to the main branch deploys the production site legislore.web.app legislore.me
URL query parameters, all of which are optional: • page [positive integer] The page of items to respond with. Defaults to 1. • items_per_page [positive integer] The number of items to respond with per page. Defaults to 10. If fewer than items_per_page items are available for the requested page, only that many items are returned. legislore.web.app legislore.me
order items. Defaults to null or the empty string, which corresponds to a undefined ordering of the items. Sortable fields are those where sortable is true, as returned by the /api/*_properties endpoints (more on this soon). sort_ascending [boolean] If the results are to be sorted, whether they are to be sorted in ascending order (true) or descending order (false). Defaults to true. Ascending order means: • for integers, from least to greatest, e.g. [0, 1, 2] • for booleans, from false to true, i.e. [false, true] • for strings, in lexicographical order, e.g. ["", "a", "b", "bb"] Descending, of course, means the opposite. legislore.web.app legislore.me
the valid sort_by values (and no other values) can be filtered by equality (eq) and not-equality (ne). A subset of these fields can also be filtered by greater than-ness (gt), less than-ness (lt), greater than or equal to-ness (ge), and less than or equal to-ness (le). Equality-comparable fields look like, e.g., chamber=eq_Senate or name.type=ne_HR. In general, they take the form field=op_val, where op is one of {eq, ne}, field is the left hand side of the comparison, and val is the right hand side of the comparison. Inequality-comparable fields look like, e.g., id=le_14 or established_date=gt_1950-01-01. In general, they take the form field=op_val, where op is one of {eq, ne, gt, lt, ge, le}, field is the left hand side of the comparison, and val is the right hand side of the comparison. The list of equality-comparable and inequality-comparable fields for a given model type are those where that model type's /api/*_properties endpoint (more on this soon) gives equality_filterable=true or inequality_filterable=true, respectively. legislore.web.app legislore.me
is parsed into a (potentially compound) predicate as follows: 1. The search string is tokenized on whitespace, as defined by Python's str.split(). 2. For each token, an attempt is made to find it contained within any searchable field. If it is found, the token is considered to be "matched". 3. Depending on the value of search_conjunction, either any or all of the tokens must be matched on a row for that row to pass the predicate corresponding to the search string. If there are no tokens, the predicate is equivalent to TRUE. Searchable fields are those whose type is string, as returned by the /api/*_properties endpoints (more on this soon). search_conjunction [string] If the results are to be filtered by a search query, whether all (search_conjunction=and) or any (search_conjunction=or) tokens need to be matched for the row to pass the filter. Defaults to and. legislore.web.app legislore.me
list endpoints /api/bills, /api/congresspeople, and /api/committees, packaging their responses in the form of: { "bill": {/*what /api/bills returns*/}, "congressperson": {/*what /api/congresspeople returns*/}, "committee": {/*what /api/committees returns*/} } Errors and pagination are localized. For example, items may exist on page p for bills but not committees. sort_by is ignored (defaults to sorting by ID). sort_ascending still works as expected. Valid filter values are those in the union of all valid filter values over all 3 model types. However, only values valid for a given model type will be used in the query for that model type, others being ignored. For Example, id=gt_50 will be applied to all 3 subqueries, but total_sponsors=le_3 will be applied only to the bill subquery. legislore.web.app legislore.me
endpoints and 1 endpoint for statistics on the GitLab repo: • / -- a welcome message, proof of online status • /api -- a list of all available endpoints • /api/gitlab_stats -- provides: ◦ the total number of commits, issues, and pipeline unit tests in the repo ◦ the per-member numbers of commits, issues, and pipeline unit tests in the repo ◦ links to the repo, the repo wiki, the repo issue tracker, and the API documentation legislore.web.app legislore.me
API design was centered around links between our 3 objects ★ Congressperson objects had fields for bills they voted for and committees they were on ★ Bill objects had fields for congresspeople who voted yea or nay and associated committees ★ Committee objects had fields for congresspeople and associated bills ★ The rest of our objects’ fields were derived based on what we felt made sense and what could be easily scraped from online sources (e.g. fields for bill descriptions, political parties congresspeople are affiliated with, etc.) legislore.web.app legislore.me
tests per each single instance endpoint (e.g. bill, congressperson, and committee), at least 1 test for every URL parameter for each list endpoint (e.g. items, items_per_page, sort_by, etc.) and at least 1 test for every important combination of URL parameters (e.g. items.length=x&items_per_page=y or sort_by=x&sort_ascending=y) for each list endpoint. legislore.web.app legislore.me
script to query api.congress.gov for data, but… ★ The API was rate limited to 1000 requests per hour, which is problematic because there are nearly 8000 bills for the 118th United States Congress ★ It was not feasible to query all the data we needed from the congress.gov API, so we turned to a bulk data source: https://www.govinfo.gov/bulkdata/BILLSTATUS/118/sres ★ Our database population script populated our MySQL database with information from the bulk data source legislore.web.app legislore.me
serialize() and preview() methods for our congressperson, bill, and committee objects ★ We wanted to make sure we were properly returning JSON representations of our objects ★ We wrote additional tests for our handle.py file and gitlab.py file ★ handle.py and gitlab.py handle API requests and deriving gitlab statistics for our website’s About page, respectively ★ Our website was deployed with React and styled with Bulma ★ We used the JavaScript Fetch API to query our backend API and React Suspense boundaries to display loading skeletons while our frontend waited on query results ★ Our backend is powered by Flask and our database is a MySQL database ★ Our frontend was deployed using Firebase Hosting, our backend services were deployed on the Google App Engine, and our MySQL database ran on Google’s Cloud SQL legislore.web.app legislore.me
mysql db on local machine 2. Run populate_db.py script 3. Confirm the contents (e.g. expected number of legislators) 4. Dump db to sql 5. Upload sql file to cloud storage 6. Import sql file into mysql instance legislore.web.app legislore.me
easy-to-navigate UI ★ Very fast website loading times ★ What could be improved: ★ Data was a bit limited, making it hard to extract interesting insights ★ Filter functionality could be expanded (more operators for certain fields) or those operands could be more carefully chosen (e.g. within ±x% rather than >= for latitude) ★ What we learned: ★ The review scores were heavily skewed toward 5-stars ★ Some cities had no 0 or 1-star reviews (Queens, San Diego) ★ Only one city had fewer than 50% 5-star reviews (The Bronx) ★ What puzzled us: ★ It wasn’t clear what all object fields conveyed ▪ There was a “price” rating for restaurants ▪ It wasn’t apparent if a higher price rating was a good thing (i.e. the restaurant is worth it for the cost of dining there) or if it was a bad thing (i.e. the higher price rating indicates that a restaurant is comparatively expensive) legislore.web.app legislore.me
easy-to-navigate UI ◦ Maintained control of project scope ◦ Time management, starting early ★ What could be improved: ◦ Very slow website loading times ◦ DB design: unify votes tables, break out some fields to their own tables to satisfy 3NF ◦ Extend beyond the 118th congress (some data was not available for prior congresses) ◦ Add committee logos to page, possibly editing them into a unified theme ★ What we learned: ◦ TypeScript, React, Bulma, Flask, SQLA, … ◦ How to add a suspense loader ◦ Careful and conscientious reading of rubrics ★ What puzzled us: ◦ Sometimes there would be discontiguities in the IDs of some models ◦ Why was our Google cloud account routinely downgrading from blaze to spark? legislore.web.app legislore.me
tmiket
rabernat
ryanjones
tammyeverts
reverentgeek
geshan
62gerente
mayunak
bryan
watany
nikkihalliwell
![Backend -API Sorting sort_by [string] The field by which to](https://files.speakerdeck.com/presentations/9ce60c268f1c4d6cb0a5add917c93de0/slide_25.jpg){kind=link}
![Backend -API Searching search [string] The search search string, which](https://files.speakerdeck.com/presentations/9ce60c268f1c4d6cb0a5add917c93de0/slide_27.jpg){kind=link}
