Footer Text Date PYTHON AND ARCGIS ONLINE Working with the ArcGIS API for Python 25 October 2018 

Footer Text Date Agenda 2 Journey to the Python API Need for automation Specs/ web content cycle Python script for automating the overwrite process of web feature services 1 1 2 2 3 3 4 4 Cleaning up feature services – Live Demo 4 4 

Footer Text Date Task Boss: “Find a way to dynamically update the copyright date for the new year “ 3 Me: “Yeah sure no problem” Also Me: 

Footer Text Date 4 

Footer Text Date Problem  TxDOT manages 60+ authoritative online services – Difficult to see a comprehensive view of all the metadata on ArcGIS Online – Updating metadata/settings individually can take time – Multiple people managing services and metadata 5 

Footer Text Date Need for Automation – Quarterly Exports  Quarterly exports of TxDOT’s Geospatial Roadway Inventory Database (GRID) that need to be updated on ArcGIS Online  Datasets need to be updated but symbology/metadata stays the same  Maintain consistency through all of the metadata/settings/etc… for authoritative services on ArcGIS Online 6 

Footer Text Date 7 

Footer Text Date Solution: ArcGIS API for Python  Automate the overwrite process of services  Organize metadata and settings  Maintain consistency  Very easy to use 8 Image Cred: https://developers.arcgis.com/python/ 

Footer Text Date What is the ArcGIS API for Python  “The ArcGIS API for Python is a powerful, modern and easy to use Pythonic library to perform GIS visualization and analysis, spatial data management and GIS system administration tasks that can run both interactively, and using scripts.”  “Implemented on top of the REST APIs of the Web GIS platform, but you use Python to connect and interact with the platform.” Representational State Transfer (REST) – exposes constraints to work with the services Application program interface (API) – provides interaction with the exposed services 9 

Footer Text Date Specs/Quick Notes  ArcGIS Pro v2.1 and later – Python 3 – conda and the arcgis package pre-installed  ArcGIS API for Python v1.5  To run Arcpy and Python API set IDE interpreter to: – C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe  Using service definitions to update/publish services 10 

Footer Text Date Content Cycle 11 Editing ETL’d into GRID Overwrite Feature Services Extracted from GRID to SDE Online Feature Services 

Footer Text Date Setting up ArcPro 12 

Footer Text Date Overwrite Quarterly Exports  Workflow – Export updated feature layers from SDE – Create connection to AGOL – Create local sddraft files – Create sd files – Query AGOL and overwrite service – Clean up metadata 13 

Footer Text Date Export updated feature layers from SDE import arcpy, os from arcgis.gis import GIS import sys # Set environment settings arcpy.env.workspace = r'C:\\..sde' arcpy.env.overwriteOutput = True # Set local variables in_features=['TPP_GIS.APP_TPP_GIS_ADMIN.GRID_Concurrencies','TPP_GIS.APP_TPP_GIS_ADMIN.GRID_Contr ol_Section','TPP_GIS.APP_TPP_GIS_ADMIN.GRID_Functional_System','TPP_GIS.APP_TPP_GIS_ADMIN.GRID_Num ber_of_Through_Lanes','TPP_GIS.APP_TPP_GIS_ADMIN.GRID_Reference_Marker','TPP_GIS.APP_TPP_GIS_ADMI N.GRID_Roadbed_Base','TPP_GIS.APP_TPP_GIS_ADMIN.GRID_Roadbed_Surface','TPP_GIS.APP_TPP_GIS_ADMI N.GRID_Roadbed_Width','TPP_GIS.APP_TPP_GIS_ADMIN.GRID_Roadway_Status'] out_location = r'C:\\..gdb' # Execute FeatureClassToGeodatabase arcpy.FeatureClassToGeodatabase_conversion(in_features, out_location) 14 

Footer Text Date Sign into ArcGIS Online/ Set workspace # Sign in to portal user = “Username" password = “Password" portal = "http://www.arcgis.com" gis = GIS(portal, user, password) tag = "Quarterly" # set workspace gdb arcpy.env.workspace = r"C:\\...gdb“ # Set output file names outdir = r"C:\\...“ # Reference map to publish aprx = arcpy.mp.ArcGISProject(r"C:\\...aprx") m = aprx.listMaps()[0] #Grab list of feature classes in map fcList = arcpy.ListFeatureClasses() fcList.sort() 15 

Footer Text Date Export sddraft files #Get list of filenames.sddraft sddraftList =[] i = 0 for fc in fcList: sddraft_filename = fcList[i] + ".sddraft" sddraft_output_filename = os.path.join(outdir, sddraft_filename) sddraftList.append(sddraft_output_filename) i+=1 sddraftList.sort() #Loop through and create sddraft files for each layer i = 0 for lyrs in m.listLayers(): sharing_draft = m.getWebLayerSharingDraft("HOSTING_SERVER", "FEATURE", lyrs, lyrs) sharing_draft.credits = "TxDOT – TPP – Data Management" sharing_draft.useLimitations = "Copyright 2018…" sharing_draft.exportToSDDraft(sddraftList[i]) # Create Service Definition Draft file i+=1 16 

Footer Text Date Query AGOL and Overwrite  #Loop through and upload service definition file i = 0 for fc in fcList: layer = fcList[i] sd_filename = layer + ".sd" sd_output_filename = os.path.join(outdir, sd_filename) os.remove(sd_output_filename) try: sdItem = gis.content.search("{} AND owner:{} AND tags:{}".format(layer, user, tag), item_type="Service Definition", sort_field="title", sort_order="asc", max_items=100)[0] weblayer = sdItem.title if layer == weblayer: try: arcpy.StageService_server(sddraftList[i], sd_output_filename) sdItem.update(data=sd_output_filename) sdItem.publish(overwrite=True) arcpy.AddMessage("Successfully published " + layer) i+=1 except IndexError: arcpy.AddMessage("Failure to publish " + layer) continue 17 

Footer Text Date Content 18 

Footer Text Date Live Demo – Updating Metadata  Find inconsistencies in metadata – Different spatial references – Unneeded tags – Inconsistent credits – Description updates – Various Settings – Unneeded column names  Update Item details, set settings,… in bulk 19 

Footer Text Date Questions? Stephen Ross [email protected] Github: RephenSoss 20