Governmental data Import and OpenStreetMap. Case report of Japan

C6a14c5446474588d8a9120e08b9880a?s=47 Satoshi IIDA
September 10, 2017

Governmental data Import and OpenStreetMap. Case report of Japan

Talk at State of the Map Taiwan 2017.


Satoshi IIDA

September 10, 2017


  1. Governmental data Import and OpenStreetMap. Case report of Japan Satoshi

    IIDA Twitter @nyampire
  2. About me • OpenStreetMap: Mapper since 2010 • OpenStreetMap Foundation

    Japan: Vice President since 2014 • Main activity: Community support • OpenStreetMap Foundation: member since 2013 • Daily Work on JIPDEC • OpenData & Geospatial consultant
  3. More simply…

  4. This guy on SNS

  5. Today’s Agenda 1. General information about Import. 2. Past Imports

    & Current relationship on Japan region. 3. Import methodology (Based on my experience).
  6. General Information about Import

  7. Abstract • Importing (also known as Bulk Importing) is the

    process of uploading external data to OSM. • [snip] • Unlike editing the map or uploading GPX traces, importing combines an existing dataset with the OSM dataset, and thus typically includes a complex merging process. ——Quote from OSM wiki - import
  8. Import? It’s easy thing. Install “opendata" JOSM plugin and put

    shapefiles onto it, then UPLOAD!!
  9. None

  11. Oh WHY? I have a data and want to import

    to improve the map!
  12. None
  13. Some tragedy in past…

  14. Unnecessary tags From “Worst of OSM”

  15. Row quality From “Worst of OSM”

  16. Misalignment From “Worst of OSM”

  17. Duplicate entry From “Worst of OSM”

  18. So OpenStreetMap has import guidelines

  19. None
  20. Note: “Community” mentioned above include both local community & international

    community. So it is mandatory to be reviewed from international members on “Imports" Mailing List Procedure of import • Step 1 - Prerequisites • Step 2 - Community Buy-in • Step 3 - License approval • Step 4 - Documentation • Step 5 - Import Review • Step 6 - Uploading From “OSM wiki import/Guidelines”
  21. History of imports on Japan Region

  22. 2008 2010 2012 2014 2017 MLIT: Kokudo Suuchi Joho (国⼟土数値情報,

    KSJ) data import, 2008-2011 Yahoo Japan!/ALPS: Road data import, 2011-2014 MLIT: KSJ Municipality boundary re-import, 2014-2015 Municipality : City planning OpenData, 2013-2016 • Urayasu City building footprint • Sabae City building footprint • Sizuoka City building footprint • Tondabayashi City road and building footprint Japan National OpenData Strategy start Import Guideline is discussed and improved
  23. Current relationship from OSM community • OSM community has major

    relationship with • Open Data advocates • Researchers, Civic tech activists, Developers • Civic, NPO activities • Town walk activity • Government Officers • Geospatial Information Distribution (Japanese geo data platform) • Local government officers • Central government officers (especially MLIT, GSI)
  24. Major Data from GSI • Traceable license of their GSI

    MAP (地理理院地図) tiles. • Base map, disaster response • GSI map has special advantage for accurate positioning (due to it is created under Survey law quality assurance). • And most important, it is distributed XYZ tile format!!
  25. GSI data which we could not use • OSM could

    NOT import Raw data from GSI. • Survey Data (測量量成果) like Fundamental Geospatial Data (基盤地図情報) is protected under Survey law. • This data is super useful, but very hard to tackle to…
  26. Current relationship with Local governments • Some local governments are

    viewing OpenStreetMap as… • use case of their Open Data. • Free (sometimes “as beer”, “less cost”) map data platform • Touchpoint for Civic activity (esp. through Mapping Party) • Each municipality has different view on OpenStreetMap. • It should be acceptable. • At first stage, OpenStreetMap have to make our data useful to civic, includes those governments.
  27. Challenge for next step Officers at local government = “Local

    knowledge specialist”
  28. Import methodology (Based on my experience)


  30. Technical

  31. File format convert • Projection convert: GDAL (ogr2ogr) • File

    Format & encoding convert: ogr2osm • Especially shapefile -> osm xml • Shapefile attribution to OSM tag convert • I recommend Paul Norman’s version • Tell your FOSS4G friends for more details :) Technical
  32. Tag Scheme • Investigate original data column to determine which

    column to be used. • Need discussion on talk-* ML and imports ML. • Most of original column is USELESS. Technical
  33. Tag Scheme sample Technical See OSM wiki 
 “Import/Catalogue” for

    more details.
  34. ogr2osm script sample Technical

  35. Operational

  36. Dedicated account • Make a import dedicated account to separate

    your work from your own account. • [sample] • Urayasu_bld_import • helicoysfh_tondabayashi_import • KSJ2_adm_bnd_imprt • It is very easy to revert if some trouble happens. Operational
  37. Organize your Team • Collaborative import is becoming popular. •

    Tasking Manager is a great tool to manage collaboration of team members. • e.g. imports by HOT team, Mapbox team, local communities and so on… • This could be reduce tasks for 1 person. • And it helps training other members. Operational
  38. In the other words…

  39. Assemble your Sentai!! Operational CC BY 2.0 Kyoryuger Roll Call

    GIF, flickr
  40. Or you have to be a Lonely Hero… CC BY,

    Kevin Dooley “batman” on flikr
  41. Operational

  42. TIPS at work • Merging with existing data is the

    most hardest work. • Divide tasks for small amounts. • Especially if it is about building or address import. • OSM US team has published some tools regarding address conflation during NYC and SF import. • Validation is mandatory. Operational
  43. Legal

  44. License compatibility • ODbL has very limited license compatibility Legal

  45. Legal perspective • CCBY is not compatible with ODbL. •

    Taiwan Open Government Licence is not compatible as well. (Japanese one is also) • So you have to ask government if we try to import the data into OpenStreetMap. Legal
  46. Special Permission • OpenStreetMap Foundation has published a template for

    asking permission. • Based on CC BY 4.0 publication (also 3.0, 2.0). • 2 items of incompatibility • Attribution on “Contributor” Page (Attribution is not signed to OSM data itself) • Technical restriction on Downstream (e.g. DRM. This is obviously different) Legal
  47. None
  48. ⿇麻煩? bothersome?

  49. NON! • Think different :) • Application to officers =

    Making a touchpoint. • A chance to introduce, discuss and include them. • A chance to walk together!
  50. ٯڥ 逆境ナイン 3巻表紙より

  51. Import is hard. But it has very effective to improve

    data coverage. And also effective to improve collaboration.
  52. \(・ω・\) Work together! (/・ω・)/

  53. Questions and feedback welcomed! Satoshi IIDA Twitter @nyampire