Designing better tools for OSM Quality Control

How can we ensure high quality contributions in a crowd-sourced geo-spatial project? This talks walks through common changes to OpenStreetMap project that need validation, and how we designed OSMCha (OSM Changeset Analyzer) to allow community to find and validate these changesets.

This was presented at State of the Map US, 2017 at Boulder, CO.
https://2017.stateofthemap.us/program/osm-quality-control.html
Watch the talk video here: https://www.youtube.com/watch?v=ufkt0lLtVek

Transcript

1. OSM Quality Control Rasagy Sharma Designer, Mapbox @rasagy / @mapbox
2. OSM Quality Control 4,278,791 members 4,465 active yesterday 52,998,958 changesets 31,981 changesets yesterday Biggest geo-spatial crowd-sourced project Stats from OSM Stats http://osmstats.neis-one.org/
3. OSM Quality Control “How do we ensure high quality data in a crowd-sourced project?” Validating OSM OSM Validation Watching for Vandalism Validating the Map Arun, SOTM 2017 Poornima & Chethan, SOTM Asia 2017 Sajjad, SOTM 2016 Sanjay, SOTM US 2016 4,278,791 members 4,465 active yesterday 52,998,958 changesets 31,981 changesets yesterday Biggest geo-spatial crowd-sourced project
4. OSM Quality Control Designing better tools for Patterns in contribution that need validation How we redesigned OSMCha What’s next?
5. Thinking OSM is a private map How do we welcome new users and make sure their first few edits are good?
6. Unaware of exact tagging schema How can we examine specific tag changes to features in a changeset?
7. Intentional, harmful changes What are the common types of changesets that are harmful to OSM?
8. Intentional, harmful changes What are the common types of changesets that are harmful to OSM?
9. 2016: How we (Mapbox Data team) used to validate OSMCha — Built by Wille Marcel in 2015, supported by Mapbox Basic set of filters, search results as a list, details in a single page.
10. Mapbox Data Team: Internal feedback • Favorite commonly used filters • Get notifications when specific filters have new changesests • Add context with changeset discussions • Give feedback on the automated flags • Track status & action • Speed up the changeset map loading time
11. Community User Interviews • Focused on their own local region • Work within small local communities • Use mailing lists to discuss suspicious changes • Focus on welcoming new users to the community • Use usernames, changeset comments, object count etc. to decide if they want to examine more details
12. Sketching + iterations
13. Sketching + iterations
14. Typical workflow: Filter, Select, View, Review
15. Filters UI: Grouped with more descriptions
16. Flags: Review requested in Colorado
17. More flags: Set of automated flags (detectors)
18. Saved Filters: Revisit, Share, RSS for notification
19. Changeset list: Key metadata of a changeset
20. Redesigned basemap & feature visualization
21. Details view: Interactivity for feature tags
22. Panels: Focusing on each aspect of a changeset
23. Redesigning: Manual tags added by community
24. Redesigning: Connecting other tools by community • OSM • JOSM & iD • Achavi • OSM-HV • Level 0 • HDYC • Deep History
25. Response: Internal feedback • Made the team more efficient when parsing through changesets • Could distribute validation tasks based on flagged reasons • Could keep track of severity and response on specific changesets
26. Response: External feedback
27. What’s next? • Validation isn’t a solved problem — how do we scale our efforts? • How do we speed up identifying critical changesets? • What can we learn from the changeset reviews (!/") — Machine Learning! The only way forward: Engaging OSM community to not just map, but also validate changes.
28. Thank you! • Meet us tomorrow for the Validation Jam (9AM | Ketchum 1B60) • Tweet to @rasagy or @mapbox • Share feedback on UI https://github.com/mapbox/osmcha-frontend/ • (Also check out the OSMCha API & OSM Compare) • Big thanks to Wille Marcel & other contributors (issues + PR!)