PYTHON IN CIVIL AND ENVIRONMENTAL ENGINEERING PAUL HOBSON GEOSYNTEC CONSULTANTS PDX PYTHON 2016-03-24 

PYTHON IN CIVIL ENGINEERING HEALTH AND SAFETY ▸ Nearest exit in case of emergency ▸ Ergonomics ▸ Right angles ▸ Head level ▸ Take breaks, look and move around 

PYTHON IN CIVIL ENGINEERING I’M AN ENGINEER (I STUDIED MUD) ▸ What is Civil Engineering? ▸ Structural ▸ Transportation ▸ Construction Management ▸ Geotechnical ▸ Environmental ▸ Water and Natural Resources Thesis Title: Rheologic and Flume Erosion Characteristics of Georgia Sediments at Bridge Piers 

PYTHON IN CIVIL ENGINEERING I WORK AS AN ENVIRONMENTAL CONSULTANT ▸ Help industry with compliance and remediation ▸ Litigation support for allocation of environmental damages ▸ Grant-funded research with NCHRP, FHWA, WERF 

PYTHON IN CIVIL ENGINEERING I WORK AS AN ENVIRONMENTAL CONSULTANT ▸ Build spatial databases of contaminant concentrations ▸ Scrape the web (brute force) to meteorologic and hydrologic data ▸ Statistical analysis of pollution mitigation and remediation systems ▸ Build inputs for, run, and analyze output from vetted numerical models ▸ Build tools to help my colleagues do the same 

PYTHON IN CIVIL ENGINEERING SUMMARY FOR THE REST OF THE TALK ▸ My take on the general state of the practice in engineering consulting ▸ Characteristic challenges of environmental consulting ▸ Tools I like to use ▸ Tools I have built over the course of my career 

PYTHON IN CIVIL ENGINEERING STATE OF THE PRACTICE - DATABASES ▸ Overwhelming everything is in MS Access ▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together ▸ Last Client’s DB ▸ 110 tables ▸ 200 saved insert/create/update ▸ 100 save select and pivot queries ▸ 25 queries that deleted stuff ▸ Everything connected with 40 VBA forms, 15 utility modules (9000 LOC) 

PYTHON IN CIVIL ENGINEERING 

PYTHON IN CIVIL ENGINEERING STATE OF THE PRACTICE - ANALYSIS ▸ Excel is the behemoth that cannot be stopped ▸ Most people manage their data with it ▸ No one follows best practices (www.datacarpentry.org/ spreadsheet-ecology-lesson) ▸ Will be pried from cold dead hands, despite well known statistical errors (recently resolved, IIRC) 

PYTHON IN CIVIL ENGINEERING MANNING’S EQUATION 

PYTHON IN CIVIL ENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS ▸ GUI or cmd-line based numerical models maintained by federal agencies ▸ SWMM ▸ 1-D urban hydrology ▸ Written in very legible C ▸ Open source ▸ Maintained by EPA, transitioning to UT Austin ▸ Compiles on Linux and Windows ▸ HEC-RAS ▸ 2-D River hydraulics ▸ Completely closed source ▸ Guarded over by US Army Corps of Engineers ▸ EFDC ▸ Spaghetti FORTRAN ▸ Compiles on Linux but results are garbage ▸ 3-D river hydraulics, sediment and pollutant fate and transport ▸ Very optimized: 1 yr simulation takes ~ 1 day 

PYTHON IN CIVIL ENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS 

PYTHON IN CIVIL ENGINEERING 

PYTHON IN CIVIL ENGINEERING STATE OF THE PRACTICE - DOMAIN SPECIFIC TOOLS ▸ All (numerically) solve the St. Venant Equations ▸ Differential equations in time and space ▸ Very difficult to even work with symbolically ▸ Large code bases ▸ Represent a significant intellectual investment from civil/ environmental community 

PYTHON IN CIVIL ENGINEERING WHERE DOES PYTHON FIT IN ALL OF THIS? ▸ Scrape model inputs from web ▸ Hack input files for batch processing ▸ Move data around between formats ▸ General data and model results analysis ▸ (Rarely) Wrap C/Fortran libraries with ctypes/cython/ numpy 

PYTHON IN CIVIL ENGINEERING CIVIL/ENVIRONMENTAL CHALLENGES ▸ Arcane input file formats ▸ Left-censored (non-detect) data ▸ Non-parametric statistics ▸ Regression-on-order statistics ▸ Kaplan-Meier ▸ Project management inertia ▸ Disconnect between libraries and project needs ▸ ~5 coworkers (out of ~600) are comfortable with python 

PYTHON IN CIVIL ENGINEERING TOOLS I USE ▸ numpy/scipy ▸ Fast numeric arrays implemented in C ▸ Specialized scientific functions ▸ Matplotlib/seaborn: generic and statistical 2-D visualizations ▸ pandas/statsmodels: table-like data structures and statistical models ▸ Jupyter: interactive computing via notebooks ▸ conda: sane installation of python packages on Windows and other operating systems 

PYTHON IN CIVIL ENGINEERING 

PYTHON IN CIVIL ENGINEERING SCRAPING WEATHER DATA ▸ conda install --channel=phobson cloudside ▸ Python 3 fork of python-metar by Tom Pollard (github.com/tomp/python- metar.git) ▸ ~1000 lines of regex ▸ Cloudside uses python-metar to pull 5-min and hourly data from the FAA ▸ ~1000 lines of regex ▸ 24229KPDX PDX20120101000009801/01/12 00:00:31 5-MIN KPDX 010800Z 31005KT 10SM OVC085 04/02 A3026 -280 85 -1500 290/05 RMK A02 P0005 

PYTHON IN CIVIL ENGINEERING WEATHER DATA ▸ Simple commands for bulk data download and parsing ▸ Creates a directory structure for each station, source, and stage of processing ▸ searches for existing files before attempting to download or parse ▸ High-level plotting functions 

PYTHON IN CIVIL ENGINEERING 

PYTHON IN CIVIL ENGINEERING 

PYTHON IN CIVIL ENGINEERING WQIO - WATER QUALITY, INFLOW/OUTFLOW ▸ conda install --channel=phobson wqio ▸ Centered around examining the efficacy of so-called BMPs ▸ Handles left-censored data with ROS ▸ No such things as having “zero” pollution ▸ Can only say that things are “less than” the instrument’s precision ▸ Bias-corrected, accelerated non-parametric bootstrapping to estimate confidence intervals around statistics ▸ High-level plotting interface built on seaborn ▸ Basis for client-specific libraries that I distribute to them through conda 

PYTHON IN CIVIL ENGINEERING PARAMNORMAL - CONVENTIONALLY PARAMETERIZED STATISTICAL DISTRIBUTIONS ▸ http://phobson.github.io/paramnormal/index.html ▸ conda install --channel=phobson paramnormal ▸ Problem: scipy distributions are incredibly flexible, but also perhaps a bit over generalized ▸ Wanted to create an API that let statisticians of all caliber specify and fit distributions using the parameters they read about in text books. ▸ The activity module provides a simple interface for creating, fitting, and plotting statistical distributions 

PYTHON IN CIVIL ENGINEERING PARANORMAL: BRIEF EXAMPLES ▸ (super fast demo) 

PYTHON IN CIVIL ENGINEERING PARAMNORMAL: ENDORSEMENTS 

PYTHON IN CIVIL ENGINEERING MPL-PROBSCALE ▸ Legit probability scales for matplotlib ▸ Similar to a quantile plot, but expressed as a probability instead of a z-score ▸ Simply import probscale and you’re set import probscale ax1.set_xscale('prob') ax1.set_xlim(left=2, right=98) ax1.set_xlabel('Normal probability scale') ax2.set_xscale('prob', dist=paramnormal.beta(α=3, β=2)) ax2.set_xlim(left=2, right=98) ax2.set_xlabel('Beta probability scale (α=3, β=2)') 

PYTHON IN CIVIL ENGINEERING MPL-PROBSCALE ▸ Can fit linear regression in probability space ▸ Top-level functions for easy plotting 

PYTHON IN CIVIL ENGINEERING GENERATING MODEL GRIDS ▸ Multi-corner, curvilinear-orthogonal grids are fairly difficult to compute — lots of very expensive proprietary stuff out there. ▸ Dr. Pavel Sakov (Australian Bureau of Meteorology) wrote gridgen-c ▸ Built a types interface with help from: ▸ Dr. Robert Hetland (Texas A&M) ▸ Dr. Richard Signell (USGS) ▸ Mac and Linux binaries available through conda ▸ Docs at: phobson.github.io/pygridgen 

PYTHON IN CIVIL ENGINEERING GRID EXAMPLE 

PYTHON IN CIVIL ENGINEERING GRID EXAMPLE 

PYTHON IN CIVIL ENGINEERING HIGHER LEVEL GRID OPERATIONS ▸ conda install --channel=conda-forge pygridtools ▸ geosyntec.github.io/pygridtools ▸ Interactivity with grid generation parameters ▸ Simple, general manipulations methods (merge, split, refine, transform) ▸ Took inspiration from pandas/xarray/seaborn ▸ File IO with general and GIS data formats via fiona 

PYTHON IN CIVIL ENGINEERING GRID EXAMPLE - MANIPULATION grid1, grid2, grid3 = make_test_grids() fig = ( grid2.merge(grid3, how='vert', where='-', shift=2) .merge(grid1, how='horiz', where='-', shift=11) .refine(10, axis=1, n_points=4) .refine(13, axis=0, n_points=2) .transform(lambda x: x*5 + 2) .update_cell_mask() .plotCells(ax=ax, cell_kws=dict(facecolor=palette[2], alpha=0.75)) ) 

PYTHON IN CIVIL ENGINEERING GRIDS IN THE REAL WORLD: SULLIVAN CREEK, NE WASHINGTON 

PYTHON IN CIVIL ENGINEERING PROJECTING MODEL RESULTS ON THE GRID 

PYTHON IN CIVIL ENGINEERING PROJECTING MODEL RESULTS ON THE GRID 

PYTHON IN CIVIL ENGINEERING RESOURCES FOR GETTING STARTED AS A MEAT-SPACE ENGINEER ▸ I was lucky enough to get started with numpy and matplotlib while: ▸ I was still very familiar with MATLAB ▸ Both projects were directly trying to take market share away from MATLAB ▸ APIs and efforts have changed since then. ▸ More recently, a new employee started with MATLAB experience from college. We gave them a copy of Python For Data Analysis by Wes McKinney (author of pandas), and were pretty productive with python in a couple of week. 

PYTHON IN CIVIL ENGINEERING LINK DUMP ▸ Me: ▸ https://twitter.com/pmhobson ▸ https://github.com/phobson ▸ My (documented) Projects ▸ http://geosyntec.github.io/pygridtools/ ▸ http://phobson.github.io/paramnormal/ ▸ http://phobson.github.io/mpl-probscale ▸ https://github.com/phobson/cloudside (docs forthcoming) 

DEMOS & QUESTIONS. Thank you