Governmental data Import and OpenStreetMap. Case report of Japan

Talk at State of the Map Taiwan 2017.

Transcript

1. Governmental data Import and OpenStreetMap. Case report of Japan Satoshi IIDA Twitter @nyampire nyampire@gmail.com
2. About me • OpenStreetMap: Mapper since 2010 • OpenStreetMap Foundation Japan: Vice President since 2014 • Main activity: Community support • OpenStreetMap Foundation: member since 2013 • Daily Work on JIPDEC • OpenData & Geospatial consultant
3. More simply…
4. This guy on SNS
5. Today’s Agenda 1. General information about Import. 2. Past Imports & Current relationship on Japan region. 3. Import methodology (Based on my experience).
6. General Information about Import
7. Abstract • Importing (also known as Bulk Importing) is the process of uploading external data to OSM. • [snip] • Unlike editing the map or uploading GPX traces, importing combines an existing dataset with the OSM dataset, and thus typically includes a complex merging process. ——Quote from OSM wiki - import
8. Import? It’s easy thing. Install “opendata" JOSM plugin and put shapefiles onto it, then UPLOAD!!
10. 3 perspectives about import 
    5FDIOJDBM 
    0QFSBUJPOBM 
    -FHBM 'JMF
    GPSNBU
    DPOWFSU %FEJDBUFE
    BDDPVOU -JDFOTF
    DPNQBUJCJMJUZ 5BHHJOH
    TDIFNF 0SHBOJ[F
    B
    UBTL
    UFBN 4QFDJBM
    QFSNJTTJPO :PV
    IBWF
    UP
    NFFU
    UIPTF
    SFRVJSFNFOUT
11. Oh WHY? I have a data and want to import to improve the map!
13. Some tragedy in past…
14. Unnecessary tags From “Worst of OSM”
15. Row quality From “Worst of OSM”
16. Misalignment From “Worst of OSM”
17. Duplicate entry From “Worst of OSM”
18. So OpenStreetMap has import guidelines
20. Note: “Community” mentioned above include both local community & international community. So it is mandatory to be reviewed from international members on “Imports" Mailing List Procedure of import • Step 1 - Prerequisites • Step 2 - Community Buy-in • Step 3 - License approval • Step 4 - Documentation • Step 5 - Import Review • Step 6 - Uploading From “OSM wiki import/Guidelines”
21. History of imports on Japan Region
22. 2008 2010 2012 2014 2017 MLIT: Kokudo Suuchi Joho (国⼟土数値情報, KSJ) data import, 2008-2011 Yahoo Japan!/ALPS: Road data import, 2011-2014 MLIT: KSJ Municipality boundary re-import, 2014-2015 Municipality : City planning OpenData, 2013-2016 • Urayasu City building footprint • Sabae City building footprint • Sizuoka City building footprint • Tondabayashi City road and building footprint Japan National OpenData Strategy start Import Guideline is discussed and improved
23. Current relationship from OSM community • OSM community has major relationship with • Open Data advocates • Researchers, Civic tech activists, Developers • Civic, NPO activities • Town walk activity • Government Officers • Geospatial Information Distribution (Japanese geo data platform) • Local government officers • Central government officers (especially MLIT, GSI)
24. Major Data from GSI • Traceable license of their GSI MAP (地理理院地図) tiles. • Base map, disaster response • GSI map has special advantage for accurate positioning (due to it is created under Survey law quality assurance). • And most important, it is distributed XYZ tile format!!
25. GSI data which we could not use • OSM could NOT import Raw data from GSI. • Survey Data (測量量成果) like Fundamental Geospatial Data (基盤地図情報) is protected under Survey law. • This data is super useful, but very hard to tackle to…
26. Current relationship with Local governments • Some local governments are viewing OpenStreetMap as… • use case of their Open Data. • Free (sometimes “as beer”, “less cost”) map data platform • Touchpoint for Civic activity (esp. through Mapping Party) • Each municipality has different view on OpenStreetMap. • It should be acceptable. • At first stage, OpenStreetMap have to make our data useful to civic, includes those governments.
27. Challenge for next step Officers at local government = “Local knowledge specialist”
28. Import methodology (Based on my experience)
29. 3 perspectives about import 
    5FDIOJDBM 
    0QFSBUJPOBM 
    -FHBM 'JMF
    GPSNBU
    DPOWFSU %FEJDBUFE
    BDDPVOU -JDFOTF
    DPNQBUJCJMJUZ 5BHHJOH
    TDIFNF 0SHBOJ[F
    B
    UBTL
    UFBN 4QFDJBM
    QFSNJTTJPO
30. Technical
31. File format convert • Projection convert: GDAL (ogr2ogr) • File Format & encoding convert: ogr2osm • Especially shapefile -> osm xml • Shapefile attribution to OSM tag convert • I recommend Paul Norman’s version • Tell your FOSS4G friends for more details :) Technical
32. Tag Scheme • Investigate original data column to determine which column to be used. • Need discussion on talk-* ML and imports ML. • Most of original column is USELESS. Technical
33. Tag Scheme sample Technical See OSM wiki 
 “Import/Catalogue” for more details.
34. ogr2osm script sample Technical
35. Operational
36. Dedicated account • Make a import dedicated account to separate your work from your own account. • [sample] • Urayasu_bld_import • helicoysfh_tondabayashi_import • KSJ2_adm_bnd_imprt • It is very easy to revert if some trouble happens. Operational
37. Organize your Team • Collaborative import is becoming popular. • Tasking Manager is a great tool to manage collaboration of team members. • e.g. imports by HOT team, Mapbox team, local communities and so on… • This could be reduce tasks for 1 person. • And it helps training other members. Operational
38. In the other words…
39. Assemble your Sentai!! Operational CC BY 2.0 Kyoryuger Roll Call GIF, flickr
40. Or you have to be a Lonely Hero… CC BY, Kevin Dooley “batman” on flikr
41. Operational
42. TIPS at work • Merging with existing data is the most hardest work. • Divide tasks for small amounts. • Especially if it is about building or address import. • OSM US team has published some tools regarding address conflation during NYC and SF import. • Validation is mandatory. Operational
43. Legal
44. License compatibility • ODbL has very limited license compatibility Legal
45. Legal perspective • CCBY is not compatible with ODbL. • Taiwan Open Government Licence is not compatible as well. (Japanese one is also) • So you have to ask government if we try to import the data into OpenStreetMap. Legal
46. Special Permission • OpenStreetMap Foundation has published a template for asking permission. • Based on CC BY 4.0 publication (also 3.0, 2.0). • 2 items of incompatibility • Attribution on “Contributor” Page (Attribution is not signed to OSM data itself) • Technical restriction on Downstream (e.g. DRM. This is obviously different) Legal
48. ⿇麻煩？ bothersome?
49. NON! • Think different :) • Application to officers = Making a touchpoint. • A chance to introduce, discuss and include them. • A chance to walk together!
50. ٯڥ 逆境ナイン 3巻表紙より
51. Import is hard. But it has very effective to improve data coverage. And also effective to improve collaboration.
52. \(･ω･\) Work together! (/･ω･)/
53. Questions and feedback welcomed! Satoshi IIDA Twitter @nyampire nyampire@gmail.com