MESH and OpenStreetMap - State of the Map Scotland 2015

MESH (Mapping Edinburgh’s Social History) is a path-breaking AHRC funded project that uses data based on addresses and areas to represent historical information. It provides a spatial dimension that enriches and enhances an understanding of the past.

This particular talk shows:
- why the MESH team chose OpenStreetMap (OSM) among different cartographic solutions,
- how the team improved OSM in Edinburgh using historical maps (out-of-copyright town plans and Ordnance Survey maps provided by the National Library of Scotland) and intensive ground surveying,
- the quality of the mapping including a comparison with current Ordnance Survey data,
- how the team and historians start to use it (tools, results),
- how the work presented here can be reused by everyone including academics, public institutions and private companies.

More info about the project: http://www.mesh.ed.ac.uk/

Transcript

1. MESH Mapping Edinburgh's
   Social History
   Eric Grosso, Richard Rodger, Sophie McCallum
   

   View Slide

2. MESH Mapping Edinburgh's
   Social History
   Co-editors:
   Richard Rodger, Bob Morris, Charlie Withers, Michael Lynch, Simon Stronach
   Partners:
   Team:
   Eric Grosso, Sophie McCallum, Eisa Esfanjari, Leila Amely, Marc di Tommasi,
   Michael Brown, Wilson Smith
   

   View Slide

3. 3
   Disclaimer
   What you'll see here is:
   - open data
   - open source
   - open data
   - open source
   - open data
   - open source
   

   View Slide

4. 4
   Disclaimer
   

   View Slide

5. 5
   Outline
   MESH
   MESH and OpenStreetMap
   Talk about history (Richard)
   Demo
   

   View Slide

6. 6
   History and Maps
   Maps and History
   About MESH
   

   View Slide

7. 7
   MESH – main objectives
   Atlas
   Data
   Maps
   Tools
   Networks
   

   View Slide

8. 8
   MESH – an atlas around 10 themes
   1. Living and dying
   2. Making, selling and earning
   3. Teaching and learning
   4. Worshipping
   5. Feeding and drinking
   6. Moving and communicating
   7. Socialising
   8. Managing and administering
   

   View Slide

9. 9
   MESH – constraints
   Atlas
   Data
   Maps
   Tools
   Networks
   

   View Slide

10. 10
    MESH – constraints... after 3 years...
    Atlas
    Data
    Maps
    Tools
    Networks
    

    View Slide

11. 11
    MESH – constraints... after 3 years...
    Atlas
    Data
    Maps
    Tools
    Networks
    Robustness (HGIS)
    Open source tools
    Open data
    Sustainability (5-10 years)
    

    View Slide

12. 12
    MESH – sustainability and legacy
    Atlas
    Data
    Maps
    Tools
    Networks
    

    View Slide

13. 13
    Constraints vs Questions / Solutions
    Open data and tools, Open licence, OpenStreetMap
    Georeferencing process
    Data integration
    Geocoding
    Robustness (HGIS)
    Open source tools
    Open data
    Sustainability (5-10 years)
    

    View Slide

14. 14
    Mapping
    Edinburgh
    (MESH and OSM)
    

    View Slide

15. 15
    MESH – how to build the HGIS?
    Historical needs, historical questions
    Choice of a period, delimitation of a space (international, national,
    regional, local), choice of a scale / level of detail
    Need for resources / data: which ones? Need to create them?
    Different resources: written documents, historical maps, surveys
    (archaeological, historical)
    Problem to know the content of these resources and to access these
    resources
    Problem solved more and more thanks to digitisation (parallel
    between digitisation and the emergence of HGIS)
    

    View Slide

16. 16
    MESH – how to build the HGIS?
    A solution could be...
    Current Google Maps
    +
    Advocates 1861, using the Google Maps API
    Overlay
    

    View Slide

17. 17
    Historical Geographic Information System
    Other resources
    Optical character recognition (OCR)
    Geocoding
    Old maps (OS / NLS)
    Georeferencing (GCP,
    transformation, evaluation, etc.)
    Vectorisation?
    MESH – how to build the HGIS?
    

    View Slide

18. 18
    MESH – which map or/and database to use?
    Google Maps (API) Ordnance Survey OpenStreetMap
    

    View Slide

19. 19
    MESH – which map or/and database to use?
    Google Maps
    

    View Slide

20. 20
    MESH – which map or/and database to use?
    Google Maps
    

    View Slide

21. 21
    MESH – which map or/and database to use?
    Google Maps
    

    View Slide

22. 22
    MESH – which map or/and database to use?
    Ordnance Survey
    (open data)
    

    View Slide

23. 23
    MESH – which map or/and database to use?
    Ordnance Survey (open data)
    

    View Slide

24. 24
    MESH – which map or/and database to use?
    OpenStreetMap, Edinburgh, September 2013
    

    View Slide

25. 25
    MESH – which map or/and database to use?
    OpenStreetMap, Edinburgh, September 2013
    

    View Slide

26. 26
    MESH – which map or/and database to use?
    OpenStreetMap, Edinburgh, September 2013
    

    View Slide

27. 27
    MESH – which map or/and database to use?
    OpenStreetMap, Edinburgh, September 2013
    

    View Slide

28. 28
    MESH – which map or/and database to use?
    GOOGLE
    ORDNANCE
    SURVEY
    MAPPING FROM
    SCRATCH
    OSM
    TYPE OF DATA Images only
    Raster and
    Vector
    Vector Vector
    COPYRIGHT Yes Yes Our choice No
    OPEN LICENCE No
    Only OpenData
    (poor quality)
    Our choice Yes (ODbL)
    DATA
    REDISTRIBUTION No No Our choice Yes (ODbL)
    QUALITY
    Relatively good
    (depends where)
    Very good (1m) Very poor
    Good (1-3m)
    – generally < 5m –
    DATABASE
    QUERY
    Only through
    Google APIs
    Yes but tools
    need to be
    developed
    Yes but tools
    need to be
    developed
    Yes with some
    already existing
    tools
    

    View Slide

29. 29
    OpenStreetMap, Edinburgh: resources
    John Wood map, 1831
    Bartholomew plan, 1912
    OS town plan, 1876-1877 OS town plan, 1893-1894
    Bartholomew plan, 1919 Bartholomew Post Office map,
    1939-1940
    Historical maps from the National Library of Scotland
    

    View Slide

30. 30
    Old maps: georeferencing
    Principle: ground control points and transformation(s)
    

    View Slide

31. 31
    OpenStreetMap, Edinburgh: resources
    OS town plan, 1849-1851 OS town plan, 1876-1877
    OS town plan, 1893-1894 OS town plan, 1940s
    

    View Slide

32. 32
    Old maps: georeferencing
    What extent?
    

    View Slide

33. 33
    OpenStreetMap, Edinburgh: resources
    OS town plan, 1893-1894 OS town plan, 1940s
    

    View Slide

34. 34
    OSM vs Old map: vectorisation
    NLS, OS town plan, 1940s
    

    View Slide

35. 35
    OSM vs Old map: vectorisation
    JOSM in action
    

    View Slide

36. 36
    OSM vs Old map: vectorisation
    JOSM in action
    

    View Slide

37. 37
    OSM vs Old map: vectorisation
    JOSM in action
    

    View Slide

38. 38
    OSM vs Old map: vectorisation
    JOSM in action
    

    View Slide

39. 39
    OSM vs Old map: vectorisation
    JOSM in action
    

    View Slide

40. 40
    OSM vs Old map: vectorisation
    JOSM in action
    

    View Slide

41. 41
    OSM vs Old map: vectorisation
    The MESH method
    Improve the roads → skeleton
    Footprint rather than roofprint
    Using lower elements (walls, roads) to manage the offset
    Tracing the walls and the gardens/plots help to understand the
    differences between the old map and the reality (Bing/survey)
    Using plots to create new buildings not even in Bing
    Adding tags
    Full survey
    

    View Slide

42. 42
    OSM vs Old map: vectorisation
    The MESH method (data model for addresses)
    And if there is a shop?
    1 address =
    addr:country
    addr:city
    addr:street
    addr:housenumber
    source
    (level)
    (addr:postcode)
    (addr:housename)
    1 address (amenity)=
    addr:street
    addr:housenumber
    source
    (level)
    

    View Slide

43. 43
    OSM: survey
    It's just a small part of our surveys
    All our surveys will be released
    to show that we walked each street
    

    View Slide

44. 44
    Old maps: vectorisation/data integration
    

    View Slide

45. 45
    Old maps: vectorisation/data integration
    

    View Slide

46. 46
    Old maps: vectorisation/data integration
    

    View Slide

47. 47
    OSM survey
    

    View Slide

48. 48
    OpenStreetMap, Edinburgh
    

    View Slide

49. 49
    OpenStreetMap, Edinburgh: MESH contributions
    

    View Slide

50. 50
    OpenStreetMap, Edinburgh: MESH contributions
    

    View Slide

51. 51
    OpenStreetMap, Edinburgh: MESH contributions
    

    View Slide

52. 52
    OpenStreetMap, Edinburgh: MESH contributions
    September 2013 Now
    

    View Slide

53. 53
    OpenStreetMap, Edinburgh: MESH contributions
    September 2013 Now
    

    View Slide

54. 54
    OpenStreetMap, Edinburgh: MESH contributions
    September 2013 Now
    

    View Slide

55. 55
    Towards a Digital Elevation Model
    NLS, OS town plan, 1940s
    

    View Slide

56. 56
    Towards a Digital Elevation Model
    Benchmark (B.M.)
    

    View Slide

57. 57
    Towards a Digital Elevation Model
    

    View Slide

58. 58
    Towards a Digital Elevation Model
    

    View Slide

59. 59
    OpenStreetMap, Edinburgh: few numbers
    Edinburgh is the 14th most populated city in UK
    3-4 man-years required to map it fully
    Accuracy: 1-3 meters
    Main enhancements and additions:
    - 15 000 road segments (25 000 in total)
    - 52 000 buildings (60 000 in total)
    - 50 000 addresses (including house numbers, shops, amenities)
    (52 000 addresses in total)
    - 11 000 POIs (12 500 POIs in total)
    

    View Slide

60. 60
    OpenStreetMap, Edinburgh: few numbers
    Other features of interest
    - railways, transport networks, waterways,
    administrative boundaries, green spaces, leisure facilities
    - gardens, walls, fences, etc.
    Ah! numbers... but what does really represent the survey?
    - 47,000 house numbers located at the door
    - total length of the roads: 530km / 330 miles
    = Edinburgh-London walk (as the crow flies)
    - total area of the buildigns: 8.3 km² / 3.2 sq miles
    = 200 Murrayfield stadiums
    200
    

    View Slide

61. 61
    Is mapping Edinburgh in OSM a big challenge?
    

    View Slide

62. 62
    Certainly a big challenge but we try using...
    fdecomite / Brand new bricks (licence: CC BY 2.0) – https://www.flickr.com/photos/fdecomite/2710132377/
    

    View Slide

63. 63
    to build....
    Rob Young / Legoland Windsor - Edinburgh Castle (licence: CC BY 2.0) – https://www.flickr.com/photos/rob-young/2835889828
    

    View Slide

64. 64
    and to avoid....
    Eirik Newth / Lego City: Collapse (licence: CC BY 2.0) – https://www.flickr.com/photos/eiriknewth/238681391
    

    View Slide

65. 65
    How to provide a proof?
    vs
    

    View Slide

66. 66
    How to provide a proof?
    © Crown Copyright and database right. All rights reserved. Ordnance
    Survey License number 100023420.
    

    View Slide

67. 67
    How to provide a proof?
    

    View Slide

68. 68
    How to provide a proof?
    

    View Slide

69. 69
    How to provide a proof?
    

    View Slide

70. 70
    OpenStreetMap, Edinburgh
    And now, so what?
    200
    

    View Slide

71. 71
    OpenStreetMap, Edinburgh
    200
    It proves something...
    our method can be applied
    everywhere in Scotland,
    in UK!
    

    View Slide

72. 72
    OpenStreetMap, Edinburgh
    200
    

    View Slide

73. 73
    OpenStreetMap, Edinburgh
    200
    

    View Slide

74. 74
    OpenStreetMap, Edinburgh
    200
    

    View Slide

75. 75
    OpenStreetMap, Edinburgh
    200
    

    View Slide

76. 76
    OpenStreetMap, Edinburgh
    200
    It also allows to do
    silly things...
    

    View Slide

77. 77
    OpenStreetMap, Edinburgh
    200
    

    View Slide

78. 78
    OpenStreetMap, Edinburgh
    200
    

    View Slide

79. 79
    OpenStreetMap, Edinburgh
    200
    

    View Slide

80. 80
    OpenStreetMap, Edinburgh
    200
    

    View Slide

81. 81
    OpenStreetMap, Edinburgh
    200
    

    View Slide

82. 82
    OpenStreetMap, Edinburgh
    200
    It can be reused by others
    

    View Slide

83. 83
    OpenStreetMap, Edinburgh
    200
    Academics
    

    View Slide

84. 84
    OpenStreetMap, Edinburgh
    200
    Public institutions
    

    View Slide

85. 85
    Geocoding – CEC data (tables and chairs permits)
    

    View Slide

86. 86
    Improvement of existing databases (RCAHMS)
    

    View Slide

87. 87
    Improvement of existing databases (RCAHMS)
    

    View Slide

88. 88
    Improvement of existing databases (RCAHMS)
    

    View Slide

89. 89
    Improvement of existing databases (RCAHMS)
    

    View Slide

90. 90
    OpenStreetMap, Edinburgh
    200
    What is the next step for us?
    (OSM)
    

    View Slide

91. 91
    OpenStreetMap, Edinburgh: road map
    “Freeze” OpenStreetMap
    Go back in time
    

    View Slide

92. 92
    Historical GIS: data integration
    John Wood map, 1831
    Bartholomew plan, 1912
    Superimposition...
    

    View Slide

93. 93
    OpenStreetMap, Edinburgh
    200
    What is the next step for us?
    (geocoder)
    

    View Slide

94. 94
    Geocoder and historical geocoders
    Geocoder
    - first version based on Nominatim
    

    View Slide

95. 95
    Geocoder and historical geocoders
    Geocoder
    - first version based on Nominatim
    - second version based on addok
    Performance: at least 140-150 addresses per second
    = Maps in seconds
    addok include process to correct misspellings
    Rather easy to to keep good performances with different databases
    of addresses (historical addresses)
    Possibility to add “fake” interpolated addresses
    CSV import/export
    

    View Slide

96. 96
    House numbers in Edinburgh
    Geocoding at a house number level, until 1800-1820
    Robert Kirkwood map (1817)
    Before 1800, geocoding at a street level
    

    View Slide

97. 97
    OSM survey
    

    View Slide

98. 98
    OSM survey
    

    View Slide

99. 99
    OSM survey
    

    View Slide

100. 100
     Routing
     Which routes to go to the coal mine? (Marc Di Tommasi)
     

     View Slide

101. 101
     Historical data
     Boundaries
     Abercrombie report
     Data from the old maps
     Data from historical sources
     Scottish Post Office Directories
     

     View Slide

102. 102
     Boundaries
     Registration districts (1865-1902)
     

     View Slide

103. 103
     Boundaries
     Municipal wards (1852-1902)
     

     View Slide

104. 104
     Abercrombie report
     Survey of dwelling sizes / family sizes / dwelling ownerships
     

     View Slide

105. 105
     Scottish Post Office Directories
     Content:
     - name/occupation/address(es)
     Licence:
     - CC BY-NC-SA
     - incompatible with OSM
     Post Office Directories (NLS)
     Optical Character Recognition (OCR)
     # old fonts / old letters / columns
     # one entry = several lines
     and
     Geocoding (Addresses)
     aka historical yellow pages (nearly 150 volumes)
     

     View Slide

106. 106
     Scottish Post Office Directories
     

     View Slide

107. 107
     Scottish Post Office Directories
     

     View Slide

108. 108
     Scottish Post Office Directories
     

     View Slide

109. 109
     Talk about history
     

     View Slide

110. Bakers
     1910-11
     

     View Slide

111. 111
     Edinburgh Retail Business Structure 1910-11
     Butchers Bakers Spirit
     Merchants
     % % %
     women (Mrs, Miss) 1 3 10
     family (& son, brother) 4 11 1
     company (& Co.) 1 2 6
     limited company (Ltd.) 3 3 0
     have 2 premises 13 32 8
     have 3 3 16 1
     have 4 1 8 0
     have 5 1 5
     have 6 1 4
     have 7+
     ( Co-op) 15 7
     have 8-12 premises 6
     have 13+ (Co-op) 10
     single premises 87 67 92
     

     View Slide

112. 112
     Spirit Dealers 1910-11 Central Area
     © MESH, NLS
     and OSM
     community
     

     View Slide

113. 113
     Data from the old maps: pubs
     Current pubs (South Edinburgh): exist (blue) or not (red) in 1940s
     

     View Slide

114. 114
     Data from the old maps: tram lines
     Tram lines: current (blue) and 1893 (red)
     

     View Slide

115. 115
     Data from the old maps: tram lines
     Tram lines: current (blue) and 1944 (orange)
     

     View Slide

116. 116
     Data from the old maps: comparison
     Current transport networks (OSM data)
     

     View Slide

117. 117
     Demo
     

     View Slide

118. 118
     want to contact us?
     www.mesh.ed.ac.uk
     @mesh_ed
     mesh@ed.ac.uk
     thank you
     

     View Slide