Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Are Safer Looking Neighborhoods More Lively? A ...

Are Safer Looking Neighborhoods More Lively? A multimodal Investigation into Urban Life

Presentation for the Responsive Environments: City eMotion course of Harvard University 2017.
Study from Google street view images in order to understand safety perception in urban enviroment with Deep Learning techniques. Original paper: https://arxiv.org/abs/1608.00462

Marco De Nadai

March 30, 2017
Tweet

More Decks by Marco De Nadai

Other Decks in Research

Transcript

  1. Marco De Nadai Are Safer Looking Neighborhoods More Lively? A

    multimodal Investigation into Urban Life
  2. Who am I? 2 • Ph.D. student in Computer science

    • Reality: I’m in Computational Social Science • University of Trento (Italy) - FBK
  3. Outline 3 • Motivation to study cities • How we

    do it • Security perception • Methods • Results • Next
  4. Cities, very difficult to explain 4 COMPLEX Not only agglomeration

    of residents, factories, shops • Millions of individuals • Continuously evolving A small change generates a cascading throughout
  5. The role of data mining and machine learning 5 DATA

    MINING • Inexpensive way to understand mechanisms; • New stimulus to social research; MACHINE LEARNING • New tools to expand the notion of what is predictable; Shmueli, Galit. "To explain or to predict?." Statistical science 25, no. 3 (2010): 289-310.
  6. Predict deprivation and per-capita income solely relying on mobility diversity

    and social diversity 6 Pappalardo, L., Vanhoof, M., Gabrielli, L., Smoreda, Z., Pedreschi, D., & Giannotti, F. (2016). An analytical framework to nowcast well-being using mobile phone data. International Journal of Data Science and Analytics
  7. Predict poverty from satellite imagery (75% variation of economic outcomes)

    7 Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S. (2016). Combining satellite imagery and machine learning to predict poverty. Science
  8. Predict crime rates from POIs, mobility, demographics 8 Wang, H.,

    Kifer, D., Graif, C., & Li, Z. (2016). Crime rate inference with big data. In Proceedings of the 22nd ACM SIGKDD.
  9. Understand underlying mechanisms of a city 9 Arbesman, Samuel. Overcomplicated:

    Technology at the Limits of Comprehension. Penguin, 2016.
  10. Understand underlying mechanisms of a city 10 Arbesman, Samuel. Overcomplicated:

    Technology at the Limits of Comprehension. Penguin, 2016. Describe
  11. Understand underlying mechanisms of a city 11 Arbesman, Samuel. Overcomplicated:

    Technology at the Limits of Comprehension. Penguin, 2016. Describe Predict
  12. Understand underlying mechanisms of a city 12 Arbesman, Samuel. Overcomplicated:

    Technology at the Limits of Comprehension. Penguin, 2016. Describe Predict Generate
  13. Understand underlying mechanisms of a city 13 Arbesman, Samuel. Overcomplicated:

    Technology at the Limits of Comprehension. Penguin, 2016. Describe Predict Generate
  14. A multitude of dimensions and aspects! 15 DESCRIBE & PREDICT

    PREDICT & GENERATE Crime 3 Urban vitality 1 Security perception 2 Structural design 4
  15. Security perception SUBJECTIVE CHARACTERISTICS Q: how can new sources of

    data and deep learning models help to link urban visual perception and the behavior of people?
  16. Broken windows theory • City mismanagement • Dirty places •

    Poor infrastructure Lead to misbehavior => Crime 17 Wilson, James Q., and George L. Kelling. "Broken windows." Critical issues in policing: Contemporary readings (1982): 395- 407. 2 1 3 SECURITY PERCEPTION 4
  17. Jane Jacobs and Newman Two of the most influential books

    in city planning • Lit streets • Street-facing windows • Physical demarcation private-public 18 Klemek, C. (2011) ‘Dead or Alive at Fifty? Reading Jane Jacobs on her Golden Anniversary’ Dissent, Vol. 58, No. 2, 75–79. 2 1 3 URBAN VITALITY 4
  18. Answer by surveys! • High costs • Not scalable •

    # people • # cities • How should we design the survey? 20
  19. Place Pulse 2.0 24 Salesses, P., Schechtner, K., & Hidalgo,

    C. A. (2013). The collaborative image of the city: mapping the inequality of urban perception. PloS one 1 2 MULTI- MODAL APPROACH
  20. Safety perception: MIT Place Pulse 25 … 1 10 1

    2 MULTI- MODAL APPROACH Place Pulse • New York • Boston • Linz • Salzburg Place Pulse 2 • Rome • Milan PROBLEM: • Just thousands of examples • Sparse in space and # votes
  21. Safety perception: fix sparse images • From 6.7 to 100

    images per km^2 • 360 degrees of images • Learning safety perception, predict in Rome and Milan • Aggregation per district 26 1 2 MULTI- MODAL APPROACH
  22. Deep Learning Networks (DNNs) • How do you extract features?

    • Handcraft features • Implicitly (Kernels) • Self-taught learning (learn representation) • Black boxes • Hard to train 27 1 2 MULTI- MODAL APPROACH
  23. DNNs: Representation 28 1 2 MULTI- MODAL APPROACH * From

    Paolo Frasconi slides – Deep Learning course – University of Trento
  24. DNNs: pre-trained Imagenet/Places 31 1 2 MULTI- MODAL APPROACH •

    Trained on 1.2 millions of images Hard to train?
  25. Deep Neural Networks 32 1 2 MULTI- MODAL APPROACH x1

    x2 x3 Layer 1 Layer 2 Layer 4 Layer 3 x4 Score
  26. Safety perception: fix sparse votes • Learning safety perception, predict

    in Rome and Milan • Standard architecture AlexNet CNN 33 * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using Places Database.” NIPS, 2014. 1 2 MULTI- MODAL APPROACH
  27. Safety perception: fix sparse votes • Learning safety perception, predict

    in Rome and Milan • Standard architecture AlexNet CNN 34 * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using Places Database.” NIPS, 2014. Model type NY - NY NY - Boston Boston - Boston Boston - NY 1 2 MULTI- MODAL APPROACH
  28. Safety perception: fix sparse votes • Learning safety perception, predict

    in Rome and Milan • Standard architecture AlexNet CNN 35 * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using Places Database.” NIPS, 2014. ** Ordonez, Vicente, and Tamara L. Berg. "Learning high-level judgments of urban perception.” ECCV, 2014. Model type State of art [**] Our model NY - NY 0.687 0.718 NY - Boston 0.701 0.734 Boston - Boston 0.718 0.744 Boston - NY 0.636 0.693 1 2 MULTI- MODAL APPROACH
  29. Safety perception: aggregation 36 1 2 MULTI- MODAL APPROACH 3.5

    3.7 3.9 4.2 4.4 4.6 4.8 5.0 Safety score DUOMO SAN SIRO QUARTO OGGIARO CITTA' STUDI BICOCCA TRASTEVERE TIBURTINO OSTIENSE PRIMAVALLE LESS SAFE SAFER ROME MILAN
  30. 1 2 Liveliness: mobile phone data • Anonymized mobile phone

    calls activity as a proxy for urban liveliness • Broken down by gender, age • 3 months time span (in 2015) • Rome and Milan 38 MULTI- MODAL APPROACH
  31. Liveliness: metrics • People around in district |',)*+ | |'

    | • Fraction females |',)*+ | |',)*+ | • People below 30 (and above 50) |(< 30)',)*+ | |',)*+ | 39 0.2 0.0 0.2 0.4 0.6 0.8 1.0 Land Use Mix 0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 31.9 Activity density ⇥ 103 ROME MILAN 0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 31.9 Activity density 103 MILAN MULTI- MODAL APPROACH 1 2
  32. 1 Link: regression 40 4.2 4.4 4.6 4.8 5.0 afety

    score DUOMO SAN SIRO QUARTO OGGIARO CITTA' STUDI BICOCCA MULTI- MODAL APPROACH 2 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Land Use Mix 0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 3 Activity density ⇥ 103 MILAN SAFETY PERCEPTION LIVELINESS
  33. = : : + ) ) + ⋯ + >

    > + The Linear Regression 41 Liveliness metric 1 MULTI- MODAL APPROACH 2
  34. The Linear Regression 42 Liveliness metric Perceived safety = :

    : + ) ) + ⋯ + > > + 1 MULTI- MODAL APPROACH 2
  35. The Linear Regression 43 Liveliness metric Perceived safety Population density

    = : : + ) ) + ⋯ + > > + 1 MULTI- MODAL APPROACH 2
  36. Spatial auto-correlation 44 Getis, Arthur, and Daniel A. Griffith. "Comparative

    spatial filtering in regression analysis." Geographical analysis 34.2 (2002): 130-140. “everything is related to everything else, but near things are more related than distant things.” Tobler's first law of geography • We control for spatial auto-correlation 1 2 MULTI- MODAL APPROACH
  37. Safety perception and liveliness 46 1 Presence of people Women,

    Young people around Visual elements for safety 2 3
  38. 2 1 Safety perception <-> presence of people 48 3

    TESTS Urban metric Beta coefficient P-value Population density Employees density Deprivation Distance from the center Safety appearance adj − R) 0.91
  39. 2 1 Safety perception <-> presence of people 49 3

    TESTS Urban metric Beta coefficient P-value Population density 0.155 ** Employees density 0.328 ** Deprivation -0.022 Distance from the center -0.257 ** Safety appearance 0.105 ** adj − R) 0.91 ** p-value < 0.001; * p-value < 0.01;
  40. 1 2 51 3 TESTS ** p-value < 0.001; *

    p-value < 0.01; Safety perception <-> women around Urban metric Beta coefficient P-value Percentage of women (census) Deprivation Distance from the center Safety appearance adj − R) 0.65
  41. 1 2 52 3 TESTS ** p-value < 0.001; *

    p-value < 0.01; Safety perception <-> women around Urban metric Beta coefficient P-value Percentage of women (census) 0.001 Deprivation -0.005 Distance from the center -0.003 Safety appearance 0.020 ** adj − R) 0.65
  42. 1 2 53 3 TESTS ** p-value < 0.001; *

    p-value < 0.01; Safety perception <-> young people around Urban metric Beta coefficient P-value Percentage of people < 30 years (census) Deprivation Distance from the center Safety appearance adj − R) 0.66
  43. 1 2 54 3 TESTS ** p-value < 0.001; *

    p-value < 0.01; Safety perception <-> young people around Urban metric Beta coefficient P-value Percentage of people < 30 years (census) -0.001 Deprivation 0.032 ** Distance from the center -0.150 ** Safety appearance -0.048 ** adj − R) 0.66
  44. Deep Neural Networks 56 x1 x2 x3 Layer 1 Layer

    2 Layer 4 Layer 3 x4 Score 1 2 3 TESTS
  45. 1 2 3 Visual elements for safety perception 57 TESTS

    HIGH SAFETY PERCEPTION RANDOMLY OBSCURE PART OF THE IMAGE AND PREDICT
  46. 1 2 3 Visual elements for safety perception 58 TESTS

    HIGH SAFETY PERCEPTION RANDOMLY OBSCURE PART OF THE IMAGE AND PREDICT
  47. 1 2 3 Visual elements for safety perception 59 TESTS

    LOW SAFETY PERCEPTION RANDOMLY OBSCURE PART OF THE IMAGE AND PREDICT
  48. “What this [paper] does is put the facts on the

    table, and that’s a big step” … “It will bring up a lot of other research, in which, I don’t have any doubt, this will be put up as a seminal step” Luis Valenzuela, Urban Planner Harvard University Source: http://news.mit.edu/2016/quantifying-urban-revitalization-1024
  49. Useful! • Evaluate the districts vitality through visual appearance •

    Plan regulatory interventions • Mobile phone data as valid proxy to census 63 Barlacchi, Gianni, et al. "A multi-source dataset of urban life in the city of Milan and the Province of Trentino." Scientific data 2 (2015).
  50. Practical applications • Real estate evaluation (land and buildings) •

    Know in advance the best places for retails (age/gender) • Analyze economic shocks in a region and respond to them 64
  51. In the (next) future • Temporal dimension (aesthetic change, people’s

    behavior) • How urban perception influences our mobility? 65 A B
  52. 66 1 Presence of people Women, Young people around Visual

    elements for safety 2 3 Safety perception and liveliness a multi-modal approach