Importing into OpenStreetMap with ogr2osm

Transcript

1. Hello Name Introduce self Here to talk about ogr2osm 1
2. I’ll talk aboutwhat ogr2osm can do for you How it works Present a case study But first, why care? 2
3. This is why you should care I wish I could say problems like this are uncommon but it didn’t take long for my database to find ways with over 300 tags from bad imports Imports like this are useless with no osm tags and even if they include some osm tags the number of confusing tags puts new mappers off 3
4. I did not start ogr2osm but I have taken over maintenance of it 4
5. Ogr2osm has some features that make it superior to other general-purpose shapefile and geodata converters Because it uses ogr it can read virtually any file format. About the only format it doesn’t support is a postgis db but that can be easily added if someone wants it It reprojects to WGS84, saving a step with ogr2ogr with most sources It will work with multi-layer sources, correctly combining shared nodes across layers with an error tolerance And most importantly, you can use python functions to convert datasource tags to OSM tags Lastly, it comes with some documentation 5
6. Installing ogr2osm is fairly easy. The only dependencies are python and gdal with python bindings. Git is required to install it and the translations Should mention Translations are their own git sub-module. This makes it possible to use your own repo for the translations without touching ogr2osm itself 6
7. The easiest way for me to explain how ogr2osm works is to briefly run through the code execution 7
8. First the file is read in By default it reads with OGR it and it makes an in-memory copy of it 8
9. Then it works on each layer Often there is just one layer but it will handle multiple ones 9
10. A sequence of steps happens for each layer 10
11. The first step for processing the layer is passing the layer through the filterLayer translation function This is a function that you can write There are two common uses The first is to drop layers that you don’t want to convert to osm for importing The second is to add new fields to the layer and to add fields to objects indicating what layer they’re from for later on 11
12. The next step is to reproject the layer to EPSG:4326 that OSM uses Ogr does all the work here so not much to say --- Next is filterFeature() Another user defined function Most common use is to drop features that you aren’t interested --- Each feature is turned from an ogr object into osm nodes ways and MPs This is where the bulk of the code is Ogr2osm will only create multipolygons if absolutely necessary. The old SVN version did so whenever there were overlapping ways but the new one does not do so. Intend to add this as an option It will handle stuff like multilinestrings without problems If you’re only using and not developing the details don’t matter too much 12
13. --- Next is the filterTags() function This is the most important user defined function. It turns source attributions into OSM tags and it’s up to you to write it. The use of a function allows complicated logic and stuff like name expansion --- filterFeaturePost is another not commonly used function. UVM uses these for buildings in a fairly complicated way. --- You then repeat for each 12
14. Next is filterFeature() Another user defined function Most common use is to drop features that you aren’t interested 13
15. In addition to each layer the features also need to be turned into EPSG:4326. not much to say here either 14
16. Each feature is turned from an ogr object into osm nodes ways and MPs This is where the bulk of the code is Ogr2osm will only create multipolygons if absolutely necessary. The old SVN version did so whenever there were overlapping ways but the new one does not do so. Intend to add this as an option It will handle stuff like multilinestrings without problems If you’re only using and not developing the details don’t matter too much 15
17. Next is the filterTags() function This is the most important user defined function. It turns source attributions into OSM tags and it’s up to you to write it. The use of a function allows complicated logic and stuff like name expansion As it’s one argument it is passed a dictionary which is the fields on the feature and it returns another dictionary which is the OSM tags 16
18. filterFeaturePost is another not commonly used function. UVM uses these for buildings in a fairly complicated way. --- You then repeat for each 17
19. You then repeat for each 18
23. As an example of writing translations, I have a case study of data from the City of Surrey that I’ve been working on This is a large and extremely detailed data set I should mention that converting the data is only half the problem. Once it’s converted you have to figure out how to best use the data, but that’s another talk in itself Although I am looking at one specific dataset most government datasets are fairly similar in what they tag and how their taggings don’t exactly correspond to OSM taggings What do I mean by a lot of data? 22
24. This is about 10x5 city blocks, about 40 hectares. As you can see, there’s a lot of data 23
25. This is one corner showing how much data there is to sort through, but I won’t go over all of it here 24
26. The first step is generally to clip the data to a smaller area ogr2osm will deal with gigabyte .osm files but normally you want to load these up into JOSM which doesn’t work so well Ogr2ogr can trim the files down to an area, but you need to be aware of a couple gotchas If your source is .mdb or something that supports long field names converting to a shapefile will truncate them You also have to deal with coordinate systems. I have a one line command in my speaker’s notes that will be up later that does this I have a one-line ogr2ogr command in my notes that handles ogr2ogr -progress -spat `echo "-122.89 49.18" | gdaltransform -s_srs EPSG:4326 - t_srs EPSG:26910 | cut -d \ -f 1,2` `echo "-122.85 49.22" | gdaltransform -s_srs EPSG:4326 -t_srs EPSG:26910 | cut -d \ -f 1,2` SurreyData_clip SurreyData 25
27. The first step is to see what layers you can drop The easiest way to do this is use the layer translation and open the resulting .osm file I’m have scripting for the surrey example before it makes it to ogr2osm that removes the undesired layers so this example is actually from a NHD translation It removes some layers which indicate NHD subbasin divisions which don’t correspond to anything physical It also adds a __LAYER field which is needed to identify which layer a feature comes from 26
28. With a complex shapefile there might be 50 fields to use when converting to OSM tags, so it really helps to have a system for writing your filterTags function This is the system I use, with an example This function first exits if it’s an untagged object like a node It then deletes some items from attrs and converts one to a ref tag It then adds any unknown fields as tags For an automatically run script you’d want to make it error if it came across unknown fields since this would indicate the fields had changed 27
29. The first stop to writing a good filtertags is to get rid of most of the fields Depending on how the data was generated you’ll have a lot of useless fields. The surrey data is basically a dump out of their database so it includes every possible field 28
30. The first step is to identify the main field for the layer. Most layers will have some kind of main field. In this case there are a total of 44 fields. 17 can be dropped right away and it’s helpful to do this just to clear up what you’re looking at 29
31. In this case the main field is RC_TYPE and RC_TYPE2 Generally the main field will give you the main tag In this case it tells you that some are highways and others are railways but doesn’t always tell you what type of highway The important point is, and I cannot emph. this enough, is to always verify the fields. You cannot assume that the descriptions mean what you think they do For a big dataset like NHD you can’t even assume that it’s consistent across the country Because road is ambiguous we need to look at it in more detail 30
32. Once you’ve identified a value, in this case RC_TYPE2=Road you want to then look at that in detail. As you can see, we still don’t have enough detail to reliably tag the class of road for RD_CLASS arterial, so we need to look for more detail 31