Are Safer Looking Neighborhoods More Lively? A multimodal Investigation into Urban Life

Presentation for the Responsive Environments: City eMotion course of Harvard University 2017.
Study from Google street view images in order to understand safety perception in urban enviroment with Deep Learning techniques. Original paper: https://arxiv.org/abs/1608.00462

Transcript

1. Marco De Nadai Are Safer Looking Neighborhoods More Lively? A multimodal Investigation into Urban Life
2. Who am I? 2 • Ph.D. student in Computer science • Reality: I’m in Computational Social Science • University of Trento (Italy) - FBK
3. Outline 3 • Motivation to study cities • How we do it • Security perception • Methods • Results • Next
4. Cities, very difficult to explain 4 COMPLEX Not only agglomeration of residents, factories, shops • Millions of individuals • Continuously evolving A small change generates a cascading throughout
5. The role of data mining and machine learning 5 DATA MINING • Inexpensive way to understand mechanisms; • New stimulus to social research; MACHINE LEARNING • New tools to expand the notion of what is predictable; Shmueli, Galit. "To explain or to predict?." Statistical science 25, no. 3 (2010): 289-310.
6. Predict deprivation and per-capita income solely relying on mobility diversity and social diversity 6 Pappalardo, L., Vanhoof, M., Gabrielli, L., Smoreda, Z., Pedreschi, D., & Giannotti, F. (2016). An analytical framework to nowcast well-being using mobile phone data. International Journal of Data Science and Analytics
7. Predict poverty from satellite imagery (75% variation of economic outcomes) 7 Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S. (2016). Combining satellite imagery and machine learning to predict poverty. Science
8. Predict crime rates from POIs, mobility, demographics 8 Wang, H., Kifer, D., Graif, C., & Li, Z. (2016). Crime rate inference with big data. In Proceedings of the 22nd ACM SIGKDD.
9. Understand underlying mechanisms of a city 9 Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
10. Understand underlying mechanisms of a city 10 Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016. Describe
11. Understand underlying mechanisms of a city 11 Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016. Describe Predict
12. Understand underlying mechanisms of a city 12 Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016. Describe Predict Generate
13. Understand underlying mechanisms of a city 13 Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016. Describe Predict Generate
14. Multi-modal understanding 14
15. A multitude of dimensions and aspects! 15 DESCRIBE & PREDICT PREDICT & GENERATE Crime 3 Urban vitality 1 Security perception 2 Structural design 4
16. Security perception SUBJECTIVE CHARACTERISTICS Q: how can new sources of data and deep learning models help to link urban visual perception and the behavior of people?
17. Broken windows theory • City mismanagement • Dirty places • Poor infrastructure Lead to misbehavior => Crime 17 Wilson, James Q., and George L. Kelling. "Broken windows." Critical issues in policing: Contemporary readings (1982): 395- 407. 2 1 3 SECURITY PERCEPTION 4
18. Jane Jacobs and Newman Two of the most influential books in city planning • Lit streets • Street-facing windows • Physical demarcation private-public 18 Klemek, C. (2011) ‘Dead or Alive at Fifty? Reading Jane Jacobs on her Golden Anniversary’ Dissent, Vol. 58, No. 2, 75–79. 2 1 3 URBAN VITALITY 4
19. Are people avoiding places where they feel unsafe? QUESTION
20. Answer by surveys! • High costs • Not scalable • # people • # cities • How should we design the survey? 20
21. Visual perception and liveliness AUTOMATICALLY LINKED
22. The multi-modal approach 22 SAFETY PERCEPTION LIVELINESS 1 2 MULTI- MODAL APPROACH
23. Safety perception: MIT Place Pulse 23 1 2 MULTI- MODAL APPROACH
24. Place Pulse 2.0 24 Salesses, P., Schechtner, K., & Hidalgo, C. A. (2013). The collaborative image of the city: mapping the inequality of urban perception. PloS one 1 2 MULTI- MODAL APPROACH
25. Safety perception: MIT Place Pulse 25 … 1 10 1 2 MULTI- MODAL APPROACH Place Pulse • New York • Boston • Linz • Salzburg Place Pulse 2 • Rome • Milan PROBLEM: • Just thousands of examples • Sparse in space and # votes
26. Safety perception: fix sparse images • From 6.7 to 100 images per km^2 • 360 degrees of images • Learning safety perception, predict in Rome and Milan • Aggregation per district 26 1 2 MULTI- MODAL APPROACH
27. Deep Learning Networks (DNNs) • How do you extract features? • Handcraft features • Implicitly (Kernels) • Self-taught learning (learn representation) • Black boxes • Hard to train 27 1 2 MULTI- MODAL APPROACH
28. DNNs: Representation 28 1 2 MULTI- MODAL APPROACH * From Paolo Frasconi slides – Deep Learning course – University of Trento
29. DNNs: Representation 29 1 2 MULTI- MODAL APPROACH
30. DNNs: Representation 30 1 2 MULTI- MODAL APPROACH
31. DNNs: pre-trained Imagenet/Places 31 1 2 MULTI- MODAL APPROACH • Trained on 1.2 millions of images Hard to train?
32. Deep Neural Networks 32 1 2 MULTI- MODAL APPROACH x1 x2 x3 Layer 1 Layer 2 Layer 4 Layer 3 x4 Score
33. Safety perception: fix sparse votes • Learning safety perception, predict in Rome and Milan • Standard architecture AlexNet CNN 33 * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using Places Database.” NIPS, 2014. 1 2 MULTI- MODAL APPROACH
34. Safety perception: fix sparse votes • Learning safety perception, predict in Rome and Milan • Standard architecture AlexNet CNN 34 * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using Places Database.” NIPS, 2014. Model type NY - NY NY - Boston Boston - Boston Boston - NY 1 2 MULTI- MODAL APPROACH
35. Safety perception: fix sparse votes • Learning safety perception, predict in Rome and Milan • Standard architecture AlexNet CNN 35 * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using Places Database.” NIPS, 2014. ** Ordonez, Vicente, and Tamara L. Berg. "Learning high-level judgments of urban perception.” ECCV, 2014. Model type State of art [**] Our model NY - NY 0.687 0.718 NY - Boston 0.701 0.734 Boston - Boston 0.718 0.744 Boston - NY 0.636 0.693 1 2 MULTI- MODAL APPROACH
36. Safety perception: aggregation 36 1 2 MULTI- MODAL APPROACH 3.5 3.7 3.9 4.2 4.4 4.6 4.8 5.0 Safety score DUOMO SAN SIRO QUARTO OGGIARO CITTA' STUDI BICOCCA TRASTEVERE TIBURTINO OSTIENSE PRIMAVALLE LESS SAFE SAFER ROME MILAN
37. Liveliness
38. 1 2 Liveliness: mobile phone data • Anonymized mobile phone calls activity as a proxy for urban liveliness • Broken down by gender, age • 3 months time span (in 2015) • Rome and Milan 38 MULTI- MODAL APPROACH
39. Liveliness: metrics • People around in district |',)*+ | |' | • Fraction females |',)*+ | |',)*+ | • People below 30 (and above 50) |(< 30)',)*+ | |',)*+ | 39 0.2 0.0 0.2 0.4 0.6 0.8 1.0 Land Use Mix 0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 31.9 Activity density ⇥ 103 ROME MILAN 0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 31.9 Activity density 103 MILAN MULTI- MODAL APPROACH 1 2
40. 1 Link: regression 40 4.2 4.4 4.6 4.8 5.0 afety score DUOMO SAN SIRO QUARTO OGGIARO CITTA' STUDI BICOCCA MULTI- MODAL APPROACH 2 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Land Use Mix 0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 3 Activity density ⇥ 103 MILAN SAFETY PERCEPTION LIVELINESS
41. = : : + ) ) + ⋯ + > > + The Linear Regression 41 Liveliness metric 1 MULTI- MODAL APPROACH 2
42. The Linear Regression 42 Liveliness metric Perceived safety = : : + ) ) + ⋯ + > > + 1 MULTI- MODAL APPROACH 2
43. The Linear Regression 43 Liveliness metric Perceived safety Population density = : : + ) ) + ⋯ + > > + 1 MULTI- MODAL APPROACH 2
44. Spatial auto-correlation 44 Getis, Arthur, and Daniel A. Griffith. "Comparative spatial filtering in regression analysis." Geographical analysis 34.2 (2002): 130-140. “everything is related to everything else, but near things are more related than distant things.” Tobler's first law of geography • We control for spatial auto-correlation 1 2 MULTI- MODAL APPROACH
45. The multi-modal approach 45 (SPATIAL) OLS REGRESSION SAFETY PERCEPTION LIVELINESS 1 MULTI- MODAL APPROACH 2
46. Safety perception and liveliness 46 1 Presence of people Women, Young people around Visual elements for safety 2 3
47. Presence of people 1st
48. 2 1 Safety perception <-> presence of people 48 3 TESTS Urban metric Beta coefficient P-value Population density Employees density Deprivation Distance from the center Safety appearance adj − R) 0.91
49. 2 1 Safety perception <-> presence of people 49 3 TESTS Urban metric Beta coefficient P-value Population density 0.155 ** Employees density 0.328 ** Deprivation -0.022 Distance from the center -0.257 ** Safety appearance 0.105 ** adj − R) 0.91 ** p-value < 0.001; * p-value < 0.01;
50. Women and young people around 2nd
51. 1 2 51 3 TESTS ** p-value < 0.001; * p-value < 0.01; Safety perception <-> women around Urban metric Beta coefficient P-value Percentage of women (census) Deprivation Distance from the center Safety appearance adj − R) 0.65
52. 1 2 52 3 TESTS ** p-value < 0.001; * p-value < 0.01; Safety perception <-> women around Urban metric Beta coefficient P-value Percentage of women (census) 0.001 Deprivation -0.005 Distance from the center -0.003 Safety appearance 0.020 ** adj − R) 0.65
53. 1 2 53 3 TESTS ** p-value < 0.001; * p-value < 0.01; Safety perception <-> young people around Urban metric Beta coefficient P-value Percentage of people < 30 years (census) Deprivation Distance from the center Safety appearance adj − R) 0.66
54. 1 2 54 3 TESTS ** p-value < 0.001; * p-value < 0.01; Safety perception <-> young people around Urban metric Beta coefficient P-value Percentage of people < 30 years (census) -0.001 Deprivation 0.032 ** Distance from the center -0.150 ** Safety appearance -0.048 ** adj − R) 0.66
55. Visual elements for safety 3rd
56. Deep Neural Networks 56 x1 x2 x3 Layer 1 Layer 2 Layer 4 Layer 3 x4 Score 1 2 3 TESTS
57. 1 2 3 Visual elements for safety perception 57 TESTS HIGH SAFETY PERCEPTION RANDOMLY OBSCURE PART OF THE IMAGE AND PREDICT
58. 1 2 3 Visual elements for safety perception 58 TESTS HIGH SAFETY PERCEPTION RANDOMLY OBSCURE PART OF THE IMAGE AND PREDICT
59. 1 2 3 Visual elements for safety perception 59 TESTS LOW SAFETY PERCEPTION RANDOMLY OBSCURE PART OF THE IMAGE AND PREDICT
60. Street facing windows and greenery contribute positively to the appearance of safety 1 2 3 TESTS
61. Why does it matter?
62. “What this [paper] does is put the facts on the table, and that’s a big step” … “It will bring up a lot of other research, in which, I don’t have any doubt, this will be put up as a seminal step” Luis Valenzuela, Urban Planner Harvard University Source: http://news.mit.edu/2016/quantifying-urban-revitalization-1024
63. Useful! • Evaluate the districts vitality through visual appearance • Plan regulatory interventions • Mobile phone data as valid proxy to census 63 Barlacchi, Gianni, et al. "A multi-source dataset of urban life in the city of Milan and the Province of Trentino." Scientific data 2 (2015).
64. Practical applications • Real estate evaluation (land and buildings) • Know in advance the best places for retails (age/gender) • Analyze economic shocks in a region and respond to them 64
65. In the (next) future • Temporal dimension (aesthetic change, people’s behavior) • How urban perception influences our mobility? 65 A B
66. 66 1 Presence of people Women, Young people around Visual elements for safety 2 3 Safety perception and liveliness a multi-modal approach
67. Thanks @denadai2