Are Safer Looking Neighborhoods More Lively? A multimodal Investigation into Urban Life

Presentation for the Responsive Environments: City eMotion course of Harvard University 2017.
Study from Google street view images in order to understand safety perception in urban enviroment with Deep Learning techniques. Original paper: https://arxiv.org/abs/1608.00462

Transcript

1. Marco De Nadai
   Are Safer Looking
   Neighborhoods More Lively?
   A multimodal Investigation
   into Urban Life
   

   View Slide

2. Who am I?
   2
   • Ph.D. student in Computer science
   • Reality: I’m in Computational Social Science
   • University of Trento (Italy) - FBK
   

   View Slide

3. Outline
   3
   • Motivation to study cities
   • How we do it
   • Security perception
   • Methods
   • Results
   • Next
   

   View Slide

4. Cities, very difficult to explain
   4
   COMPLEX
   Not only agglomeration of
   residents, factories, shops
   • Millions of individuals
   • Continuously evolving
   A small change generates a
   cascading throughout
   

   View Slide

5. The role of data mining and machine learning
   5
   DATA MINING
   • Inexpensive way to understand mechanisms;
   • New stimulus to social research;
   MACHINE LEARNING
   • New tools to expand the notion of what is predictable;
   Shmueli, Galit. "To explain or to predict?." Statistical science 25, no. 3 (2010): 289-310.
   

   View Slide

6. Predict deprivation and per-capita
   income solely relying on mobility
   diversity and social diversity
   6
   Pappalardo, L., Vanhoof, M., Gabrielli, L., Smoreda, Z.,
   Pedreschi, D., & Giannotti, F. (2016). An analytical framework to
   nowcast well-being using mobile phone data. International
   Journal of Data Science and Analytics
   

   View Slide

7. Predict poverty from satellite
   imagery
   (75% variation of economic outcomes)
   7
   Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S.
   (2016). Combining satellite imagery and machine learning to
   predict poverty. Science
   

   View Slide

8. Predict crime rates from POIs,
   mobility, demographics
   8
   Wang, H., Kifer, D., Graif, C., & Li, Z. (2016). Crime rate inference
   with big data. In Proceedings of the 22nd ACM SIGKDD.
   

   View Slide

9. Understand underlying mechanisms of a city
   9
   Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
   

   View Slide

10. Understand underlying mechanisms of a city
    10
    Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
    Describe
    

    View Slide

11. Understand underlying mechanisms of a city
    11
    Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
    Describe Predict
    

    View Slide

12. Understand underlying mechanisms of a city
    12
    Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
    Describe Predict Generate
    

    View Slide

13. Understand underlying mechanisms of a city
    13
    Arbesman, Samuel. Overcomplicated: Technology at the Limits of Comprehension. Penguin, 2016.
    Describe Predict Generate
    

    View Slide

14. Multi-modal understanding
    14
    

    View Slide

15. A multitude of dimensions and aspects!
    15
    DESCRIBE
    &
    PREDICT
    PREDICT
    &
    GENERATE
    Crime
    3
    Urban vitality
    1
    Security perception
    2
    Structural design
    4
    

    View Slide

16. Security perception
    SUBJECTIVE CHARACTERISTICS
    Q: how can new sources of data and deep learning models help to
    link urban visual perception and the behavior of people?
    

    View Slide

17. Broken windows theory
    • City mismanagement
    • Dirty places
    • Poor infrastructure
    Lead to misbehavior => Crime
    17
    Wilson, James Q., and George L. Kelling. "Broken windows."
    Critical issues in policing: Contemporary readings (1982): 395-
    407.
    2
    1 3
    SECURITY PERCEPTION 4
    

    View Slide

18. Jane Jacobs and Newman
    Two of the most influential books in
    city planning
    • Lit streets
    • Street-facing windows
    • Physical demarcation private-public
    18
    Klemek, C. (2011) ‘Dead or Alive at Fifty? Reading Jane Jacobs
    on her Golden Anniversary’ Dissent, Vol. 58, No. 2, 75–79.
    2
    1 3
    URBAN VITALITY 4
    

    View Slide

19. Are people avoiding places
    where they feel unsafe?
    QUESTION
    

    View Slide

20. Answer by surveys!
    • High costs
    • Not scalable
    • # people
    • # cities
    • How should we design the survey?
    20
    

    View Slide

21. Visual perception and liveliness
    AUTOMATICALLY LINKED
    

    View Slide

22. The multi-modal approach
    22
    SAFETY PERCEPTION LIVELINESS
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

23. Safety perception: MIT Place Pulse
    23
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

24. Place Pulse 2.0
    24
    Salesses, P., Schechtner, K., & Hidalgo, C. A. (2013). The collaborative image of the city: mapping the
    inequality of urban perception. PloS one
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

25. Safety perception: MIT Place Pulse
    25
    …
    1
    10
    1 2
    MULTI- MODAL APPROACH
    Place Pulse
    • New York
    • Boston
    • Linz
    • Salzburg
    Place Pulse 2
    • Rome
    • Milan
    PROBLEM:
    • Just thousands of examples
    • Sparse in space and # votes
    

    View Slide

26. Safety perception: fix sparse images
    • From 6.7 to 100 images per km^2
    • 360 degrees of images
    • Learning safety perception,
    predict in Rome and Milan
    • Aggregation per district
    26
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

27. Deep Learning Networks (DNNs)
    • How do you extract features?
    • Handcraft features
    • Implicitly (Kernels)
    • Self-taught learning (learn representation)
    • Black boxes
    • Hard to train
    27
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

28. DNNs: Representation
    28
    1 2
    MULTI- MODAL APPROACH
    * From Paolo Frasconi slides – Deep Learning course – University of Trento
    

    View Slide

29. DNNs: Representation
    29
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

30. DNNs: Representation
    30
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

31. DNNs: pre-trained Imagenet/Places
    31
    1 2
    MULTI- MODAL APPROACH
    • Trained on 1.2 millions of images
    Hard to train?
    

    View Slide

32. Deep Neural Networks
    32
    1 2
    MULTI- MODAL APPROACH
    x1
    x2
    x3
    Layer 1 Layer 2
    Layer 4
    Layer 3
    x4
    Score
    

    View Slide

33. Safety perception: fix sparse votes
    • Learning safety perception,
    predict in Rome and Milan
    • Standard architecture AlexNet CNN
    33
    * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using
    Places Database.” NIPS, 2014.
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

34. Safety perception: fix sparse votes
    • Learning safety perception,
    predict in Rome and Milan
    • Standard architecture AlexNet CNN
    34
    * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using
    Places Database.” NIPS, 2014.
    Model type
    NY - NY
    NY - Boston
    Boston - Boston
    Boston - NY
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

35. Safety perception: fix sparse votes
    • Learning safety perception,
    predict in Rome and Milan
    • Standard architecture AlexNet CNN
    35
    * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using
    Places Database.” NIPS, 2014.
    ** Ordonez, Vicente, and Tamara L. Berg. "Learning high-level judgments of urban perception.” ECCV, 2014.
    Model type State of art [**] Our model
    NY - NY 0.687 0.718
    NY - Boston 0.701 0.734
    Boston - Boston 0.718 0.744
    Boston - NY 0.636 0.693
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

36. Safety perception: aggregation
    36
    1 2
    MULTI- MODAL APPROACH
    3.5 3.7 3.9 4.2 4.4 4.6 4.8 5.0
    Safety score
    DUOMO
    SAN SIRO
    QUARTO
    OGGIARO
    CITTA'
    STUDI
    BICOCCA
    TRASTEVERE
    TIBURTINO
    OSTIENSE
    PRIMAVALLE
    LESS SAFE SAFER
    ROME MILAN
    

    View Slide

37. Liveliness
    

    View Slide

38. 1 2
    Liveliness: mobile phone data
    • Anonymized mobile phone calls
    activity as a proxy for urban
    liveliness
    • Broken down by gender, age
    • 3 months time span (in 2015)
    • Rome and Milan
    38
    MULTI- MODAL APPROACH
    

    View Slide

39. Liveliness: metrics
    • People around in district
    |',)*+
    |
    |'
    |
    • Fraction females
    |',)*+
    |
    |',)*+
    |
    • People below 30 (and above 50)
    |(< 30)',)*+
    |
    |',)*+
    |
    39
    0.2
    0.0
    0.2
    0.4
    0.6
    0.8
    1.0
    Land Use Mix
    0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 31.9
    Activity density ⇥ 103
    ROME
    MILAN
    0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 31.9
    Activity density 103
    MILAN
    MULTI- MODAL APPROACH 1 2
    

    View Slide

40. 1
    Link: regression
    40
    4.2 4.4 4.6 4.8 5.0
    afety score
    DUOMO
    SAN SIRO
    QUARTO
    OGGIARO
    CITTA'
    STUDI
    BICOCCA
    MULTI- MODAL APPROACH 2
    0.2
    0.0
    0.2
    0.4
    0.6
    0.8
    1.0
    1.2
    Land Use Mix
    0.8 1.2 1.8 2.7 4.1 6.1 9.3 14.0 21.1 3
    Activity density ⇥ 103
    MILAN
    SAFETY PERCEPTION LIVELINESS
    

    View Slide

41. = :
    :
    + )
    )
    + ⋯ + >
    >
    +
    The Linear Regression
    41
    Liveliness metric
    1
    MULTI- MODAL APPROACH 2
    

    View Slide

42. The Linear Regression
    42
    Liveliness metric
    Perceived safety
    = :
    :
    + )
    )
    + ⋯ + >
    >
    +
    1
    MULTI- MODAL APPROACH 2
    

    View Slide

43. The Linear Regression
    43
    Liveliness metric
    Perceived safety
    Population density
    = :
    :
    + )
    )
    + ⋯ + >
    >
    +
    1
    MULTI- MODAL APPROACH 2
    

    View Slide

44. Spatial auto-correlation
    44
    Getis, Arthur, and Daniel A. Griffith. "Comparative spatial filtering in
    regression analysis." Geographical analysis 34.2 (2002): 130-140.
    “everything is related to everything else,
    but near things are more related than
    distant things.”
    Tobler's first law of geography
    • We control for spatial auto-correlation
    1 2
    MULTI- MODAL APPROACH
    

    View Slide

45. The multi-modal approach
    45
    (SPATIAL) OLS
    REGRESSION
    SAFETY PERCEPTION LIVELINESS
    1
    MULTI- MODAL APPROACH 2
    

    View Slide

46. Safety perception and liveliness
    46
    1
    Presence
    of people
    Women, Young people
    around
    Visual elements
    for safety
    2 3
    

    View Slide

47. Presence of people
    1st
    

    View Slide

48. 2
    1
    Safety perception <-> presence of people
    48
    3
    TESTS
    Urban metric Beta coefficient P-value
    Population density
    Employees density
    Deprivation
    Distance from the center
    Safety appearance
    adj − R) 0.91
    

    View Slide

49. 2
    1
    Safety perception <-> presence of people
    49
    3
    TESTS
    Urban metric Beta coefficient P-value
    Population density 0.155 **
    Employees density 0.328 **
    Deprivation -0.022
    Distance from the center -0.257 **
    Safety appearance 0.105 **
    adj − R) 0.91
    ** p-value < 0.001; * p-value < 0.01;
    

    View Slide

50. Women and young people around
    2nd
    

    View Slide

51. 1 2
    51
    3
    TESTS
    ** p-value < 0.001; * p-value < 0.01;
    Safety perception <-> women around
    Urban metric Beta coefficient P-value
    Percentage of women (census)
    Deprivation
    Distance from the center
    Safety appearance
    adj − R) 0.65
    

    View Slide

52. 1 2
    52
    3
    TESTS
    ** p-value < 0.001; * p-value < 0.01;
    Safety perception <-> women around
    Urban metric Beta coefficient P-value
    Percentage of women (census) 0.001
    Deprivation -0.005
    Distance from the center -0.003
    Safety appearance 0.020 **
    adj − R) 0.65
    

    View Slide

53. 1 2
    53
    3
    TESTS
    ** p-value < 0.001; * p-value < 0.01;
    Safety perception <-> young people around
    Urban metric Beta coefficient P-value
    Percentage of people < 30 years (census)
    Deprivation
    Distance from the center
    Safety appearance
    adj − R) 0.66
    

    View Slide

54. 1 2
    54
    3
    TESTS
    ** p-value < 0.001; * p-value < 0.01;
    Safety perception <-> young people around
    Urban metric Beta coefficient P-value
    Percentage of people < 30 years (census) -0.001
    Deprivation 0.032 **
    Distance from the center -0.150 **
    Safety appearance -0.048 **
    adj − R) 0.66
    

    View Slide

55. Visual elements for safety
    3rd
    

    View Slide

56. Deep Neural Networks
    56
    x1
    x2
    x3
    Layer 1 Layer 2
    Layer 4
    Layer 3
    x4
    Score
    1 2 3
    TESTS
    

    View Slide

57. 1 2 3
    Visual elements for safety perception
    57
    TESTS
    HIGH SAFETY PERCEPTION
    RANDOMLY OBSCURE
    PART OF THE IMAGE
    AND PREDICT
    

    View Slide

58. 1 2 3
    Visual elements for safety perception
    58
    TESTS
    HIGH SAFETY PERCEPTION
    RANDOMLY OBSCURE
    PART OF THE IMAGE
    AND PREDICT
    

    View Slide

59. 1 2 3
    Visual elements for safety perception
    59
    TESTS
    LOW SAFETY PERCEPTION
    RANDOMLY OBSCURE
    PART OF THE IMAGE
    AND PREDICT
    

    View Slide

60. Street facing windows and greenery
    contribute positively to the appearance
    of safety
    1 2 3
    TESTS
    

    View Slide

61. Why does it matter?
    

    View Slide

62. “What this [paper] does is put the facts on the table,
    and that’s a big step”
    …
    “It will bring up a lot of other research, in which, I
    don’t have any doubt, this will be put up as a
    seminal step”
    Luis Valenzuela, Urban Planner
    Harvard University
    Source: http://news.mit.edu/2016/quantifying-urban-revitalization-1024
    

    View Slide

63. Useful!
    • Evaluate the districts vitality through visual appearance
    • Plan regulatory interventions
    • Mobile phone data as valid proxy to census
    63
    Barlacchi, Gianni, et al. "A multi-source dataset of urban life in the city of Milan and the Province of
    Trentino." Scientific data 2 (2015).
    

    View Slide

64. Practical applications
    • Real estate evaluation (land and buildings)
    • Know in advance the best places for retails (age/gender)
    • Analyze economic shocks in a region and respond to them
    64
    

    View Slide

65. In the (next) future
    • Temporal dimension (aesthetic change, people’s behavior)
    • How urban perception influences our mobility?
    65
    A B
    

    View Slide

66. 66
    1
    Presence
    of people
    Women, Young people
    around
    Visual elements
    for safety
    2 3
    Safety perception and liveliness
    a multi-modal approach
    

    View Slide

67. Thanks
    @denadai2
    

    View Slide